HP3000-L Archives

December 1995, Week 2

HP3000-L@RAVEN.UTC.EDU

Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Bruce Toback <[log in to unmask]>
Reply To:
Bruce Toback <[log in to unmask]>
Date:
Thu, 7 Dec 1995 07:44:15 -0700
Content-Type:
text/plain
Parts/Attachments:
text/plain (34 lines)
Sorry to follow up my own post, but:
>   :SETVAR total 216340.17+61300.00+39800.00
>   :SHOWVAR total
>    TOTAL =     317440.1562500
>
 
I assumed that the CI could be doing fixed-point math, which is still true
albeit unlikely (unless someone has designs on the CI for a graphics
programming environment!).
 
The other reason is that the command interpreter (but I still can't
duplicate this!) is using 32-bit floating point. 32-bit floating point
provides only 24 bits of precision, and the number 216340.17 requires more
than that to represent. To a rough approximation, every significant decimal
digit requires a little more than three bits to represent in binary; the
number 216340.17 therefore requires more than 24 bits to represent. The
program you're using appears to calculate in single-precision but display
in double-precision; this isn't a good idea, but it's pretty typical of
quick-and-dirty implementations which use a single routine to format both
single- and double-precision numerics, converting single-precision to
double before doing the display conversion. Essentially, the
single-to-double conversion tacks a bunch of significant zeros onto a
number that didn't have any right to them, so the display formatter doesn't
round off at the right spot.
 
-- Bruce
 
---------------------------------------------------------------------------
Bruce Toback    Tel: (602) 996-8601| My candle burns at both ends;
OPT, Inc.            (800) 858-4507| It will not last the night;
11801 N. Tatum Blvd. Ste. 142      | But ah, my foes, and oh, my friends -
Phoenix AZ 85028                   | It gives a lovely light.
[log in to unmask]                 |     -- Edna St. Vincent Millay

ATOM RSS1 RSS2