HP3000-L Archives

December 1995, Week 2

HP3000-L@RAVEN.UTC.EDU

Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Charles D MEEKER <[log in to unmask]>
Reply To:
Charles D MEEKER <[log in to unmask]>
Date:
Thu, 7 Dec 1995 09:29:09 CST
Content-Type:
text/plain
Parts/Attachments:
text/plain (45 lines)
Jerry Fochtman tells of a strange problem:
 
>I ran across an interesting situation the other day when using the CI to
>help perform some simple calculations.  Consider the following:
>
>   :SETVAR total 216340.17+61300.00+39800.00
>   :SHOWVAR total
>    TOTAL =     317440.1562500
>
>I can appreciate that CI wouldn't necessarily know what precision to
>display the data.  However, if this is rounded to 2 places, it would be
>317440.16 .... when it should be 317440.17...
 
And Bruce says:
 
>   M5(7):SETVAR total 216340.17+61300.00+39800.00
>   Unexpected character found in number. (CIERR 9810)
>
>I get this on 5.0 Push. I don't know what OS you're running, but it's
>different from mine! The symptom you're seeing can be explained if the CI
>is doing fixed-point math with fractions, using five or six bits for the
>fractional part.
 
I tried it and got the same answer as Bruce.  Then I recalled something:
 
980 (2): mpex
 
 
MPEX/3000  25N41109  (c) VESOFT Inc, 1980  4.0  05:00244  For help type
'HELP'
%SETVAR total 216340.17+61300.00+39800.00
 
%showvar total
 
TOTAL =        317440.156250
 
%
 
 
Maybe you should talk to VESoft...
 
 
Chuck Meeker
Rockwell

ATOM RSS1 RSS2