When I use Calculator (v3.2.1) to do the following calculation:
9000 - 8387.7 - 557.3

I get the result:
54.999999999999

What gives?

--
Peter McCallum
Mackay Qld AUSTRALIA

Peter McCallum wrote:
> When I use Calculator (v3.2.1) to do the following calculation:
> 9000 - 8387.7 - 557.3
>
> I get the result:
> 54.999999999999
>

Using v. 4.0 I get the same result. But changing the precision from 12
to 16 gives a more pleasing result. And so does changing the precision
to 6. A bit odd.

In article <1gx8fdy.xkc4ho1cgzvfwN%p5m8.REMOVETHIS******.com. au>,
p5m8.REMOVETHIS******.com.au (Peter McCallum) wrote:

> When I use Calculator (v3.2.1) to do the following calculation:
> 9000 - 8387.7 - 557.3
>
> I get the result:
> 54.999999999999
>
> What gives?

Most computers nowadays converts all floating point numbers into binary
form, and then performs operations on those. Thus, round-off errors might
occur.

You can get better precision in Terminal, by searching for calculator programs:
man -k calculator
One such program is "bc". You can learn about it by typing "man bc". Then type:
echo "9000 - 8387.7 - 557.3" | bc
55.0
Here, the answer is merely a round-off of a higher precision result.

--
Hans Aberg

Phil Wheeler wrote:
> Peter McCallum wrote:
>
>> When I use Calculator (v3.2.1) to do the following calculation:
>> 9000 - 8387.7 - 557.3
>>
>> I get the result:
>> 54.999999999999
>>
>
> Using v. 4.0 I get the same result. But changing the precision from 12
> to 16 gives a more pleasing result. And so does changing the precision
> to 6. A bit odd.
I noticed that about a year ago when I first purchased OSX. (Reminded
me of using Fortan in College.)
Anyway, changing the precision to 10 worked for me.
Thanks for the tip!
GW

In article <1gx8fdy.xkc4ho1cgzvfwN%p5m8.REMOVETHIS******.com. au>,
p5m8.REMOVETHIS******.com.au says...
> When I use Calculator (v3.2.1) to do the following calculation:
> 9000 - 8387.7 - 557.3
>
> I get the result:
> 54.999999999999
>
> What gives?

http://docs.sun.com/source/806-3568/ncg_goldberg.html

--
Randy Howard (2reply remove FOOBAR)
"I don't really care about being right you know,
I just care about success." --Steve Jobs