Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996
I'm expecting 0.01 as the result. Do I have to specify the precision? Even with the full precision available on the platform (Windows XP), the correct answer is still 0.01, not what's printed. What's the deal? I like this language so far, but this is confusing...
Brian
Le 9/6/2005, "brian" <brian@le-roy.org> a écrit:
Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...
Floating point math 
You can use BigDecimal to avoid most problems.
Brian
E
--
template<typename duck>
void quack(duck& d) { d.quack(); }
Thanks, I'll give it a read. Thanks to Bill too. I have to say this is one of the friendlier communities I've come across. The BigDecimal is working out. Newb's not getting burned with RTFM's, tends to keep them coming back for more. Thanks again!
Gary Wright wrote:
On Jun 8, 2005, at 8:40 PM, brian wrote:
Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996I'm expecting 0.01 as the result. Do I have to specify the precision? Even with the full precision available on the platform (Windows XP), the correct answer is still 0.01, not what's printed. What's the deal? I like this language so far, but this is confusing...
This is a floating point math issue not a ruby language issue.
Some base 10 floating point numbers can not be represented exactly
when converted to binary (and vice versa). This often leads to small
differences between the arithmetic results expected by you (in base 10
format) and calculated by the computer (in binary).The solution is to look for "close answers" or switch to a different
encoding (i.e. not binary floating point) for your math. In many
cases just using integers with an appropriate scaling factor solves
the problem.If anyone is interested in all the gory details of this topic here is
a good read: "What Every Computer Scientist Should Know About Floating-Point
Arithmetic".
Along with all the other good responses, I thought I'd just take the
moment to (once again) state that this behaviour isn't unique to ruby:
ruby: "%.16f" % (9.0-8.9) ==> 0.0999999999999996
perl: sprintf("%.16f", 9.0-8.9) ==> 0.0999999999999996
python: "%.16f" % (9.0-8.9) ==> 0.0999999999999996
php: sprintf("%.16f", 9.0-8.9) ==> 0.0999999999999996
C: sprintf(dest, "%.16f", 9.0-8.9) ==> 0.0999999999999996
In all of the cases, specifying a lower precision allows the
float->string conversion to round to the "expected" result:
ruby: "%.10f" % (9.0-8.9) ==> 0.1000000000
perl: sprintf("%.10f", 9.0-8.9) ==> 0.1000000000
python: "%.10f" % (9.0-8.9) ==> 0.1000000000
php: sprintf("%.10f", 9.0-8.9) ==> 0.1000000000
C: sprintf(dest, "%.10f", 9.0-8.9) ==> 0.1000000000
(The source for round_error_c_2 is identical to round_error_c but with
only 10 instead of 16 digits of precision in the printf.)
The major difference is the default precision used by languages in the
float->string conversion. From experimentation (not actually looking
at sources), I came to the following conclusions on the default
precision:
* ruby and perl both use up to 15 significant digits
* python uses up to 12 significant digits
* php uses up to 14 significant digits
* C uses 6 places after the decimal, always
(with %f instead of %.10f in the printf)
For all of ruby, perl, python and php I say "up to X significant
digits" because significant 0's at the end are truncated (e.g. (1.0 -
0.5).to_s returns '0.5', not '0.500000000000000') and significant
digit rules are obeyed (significant digits before the decimal point
reduce the number of digits after the decimal point, leading zeros
after the decimal point are not counted).
Jacob Fugal
On 6/8/05, brian <brian@le-roy.org> wrote:
Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...Brian
Hi Brian,
Le 9/6/2005, "brian" <brian@le-roy.org> a écrit:
>Can someone tell me why this code:
> puts (9.0-8.9).to_s
>results in:
> 0.0999999999999996
>
>I'm expecting 0.01 as the result. Do I have to specify the precision?
>Even with the full precision available on the platform (Windows XP), the
>correct answer is still 0.01, not what's printed. What's the deal? I
>like this language so far, but this is confusing...Floating point math
If you'd like more info about the problem, the first
paragraphs of the section entitled Rounding Error in
this document give the gist of it:
http://docs.sun.com/source/806-3568/ncg_goldberg.html
Regards,
Bill
From: "ES" <ruby-ml@magical-cat.org>