Subject: | |
From: | |
Reply To: | |
Date: | Thu, 7 Dec 1995 07:44:15 -0700 |
Content-Type: | text/plain |
Parts/Attachments: |
|
|
Sorry to follow up my own post, but:
> :SETVAR total 216340.17+61300.00+39800.00
> :SHOWVAR total
> TOTAL = 317440.1562500
>
I assumed that the CI could be doing fixed-point math, which is still true
albeit unlikely (unless someone has designs on the CI for a graphics
programming environment!).
The other reason is that the command interpreter (but I still can't
duplicate this!) is using 32-bit floating point. 32-bit floating point
provides only 24 bits of precision, and the number 216340.17 requires more
than that to represent. To a rough approximation, every significant decimal
digit requires a little more than three bits to represent in binary; the
number 216340.17 therefore requires more than 24 bits to represent. The
program you're using appears to calculate in single-precision but display
in double-precision; this isn't a good idea, but it's pretty typical of
quick-and-dirty implementations which use a single routine to format both
single- and double-precision numerics, converting single-precision to
double before doing the display conversion. Essentially, the
single-to-double conversion tacks a bunch of significant zeros onto a
number that didn't have any right to them, so the display formatter doesn't
round off at the right spot.
-- Bruce
---------------------------------------------------------------------------
Bruce Toback Tel: (602) 996-8601| My candle burns at both ends;
OPT, Inc. (800) 858-4507| It will not last the night;
11801 N. Tatum Blvd. Ste. 142 | But ah, my foes, and oh, my friends -
Phoenix AZ 85028 | It gives a lovely light.
[log in to unmask] | -- Edna St. Vincent Millay
|
|
|