Subject: | |
From: | |
Reply To: | |
Date: | Thu, 7 Dec 1995 09:29:09 CST |
Content-Type: | text/plain |
Parts/Attachments: |
|
|
Jerry Fochtman tells of a strange problem:
>I ran across an interesting situation the other day when using the CI to
>help perform some simple calculations. Consider the following:
>
> :SETVAR total 216340.17+61300.00+39800.00
> :SHOWVAR total
> TOTAL = 317440.1562500
>
>I can appreciate that CI wouldn't necessarily know what precision to
>display the data. However, if this is rounded to 2 places, it would be
>317440.16 .... when it should be 317440.17...
And Bruce says:
> M5(7):SETVAR total 216340.17+61300.00+39800.00
> Unexpected character found in number. (CIERR 9810)
>
>I get this on 5.0 Push. I don't know what OS you're running, but it's
>different from mine! The symptom you're seeing can be explained if the CI
>is doing fixed-point math with fractions, using five or six bits for the
>fractional part.
I tried it and got the same answer as Bruce. Then I recalled something:
980 (2): mpex
MPEX/3000 25N41109 (c) VESOFT Inc, 1980 4.0 05:00244 For help type
'HELP'
%SETVAR total 216340.17+61300.00+39800.00
%showvar total
TOTAL = 317440.156250
%
Maybe you should talk to VESoft...
Chuck Meeker
Rockwell
|
|
|