This
('40.12'.to_f*100).to_i
results in this integer :
4011
This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.
Greg
What I find interesting is that the #inspect result of both f1 and f2
below are the same, when they internally are clearly not.
irb(main):001:0> f1 = '40.12'.to_f
=> 40.12
irb(main):002:0> f1*100
=> 4012.0
irb(main):003:0> (f1*100).to_i
=> 4011
irb(main):004:0> (f1*100).round.to_i
=> 4012
irb(main):005:0> 4012.0.to_i
=> 4012
irb(main):006:0> f2 = '40.12000000000001'.to_f
=> 40.12
irb(main):007:0> (f2*100).to_i
=> 4012
···
On Mar 19, 1:37 pm, "Greg Lorriman" <t...@lorriman.com> wrote:
This
('40.12'.to_f*100).to_i
results in this integer :
4011
This
('40.12'.to_f*100).to_i
results in this integer :
4011
This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Only standard numeric rounding error. The problem is that decimal 0.1 cannot
be represented exactly in binary floating point - it's a recurring fraction,
like 1/3 in decimal - so you get only an approximation.
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.
irb(main):002:0> require 'bigdecimal'
=> true
irb(main):003:0> f = BigDecimal.new('40.12')
=> #<BigDecimal:b7cb7548,'0.4012E2',8(8)>
irb(main):004:0> (f*100).to_i
=> 4012
BigDecimal stores decimal numbers as decimal. It's not exactly well
documented though (I only came across it because Ruby on Rails uses it).
Another solution is to multiply all your numbers by 100 at source, i.e.
works in cents rather than dollars or whatever.
Regards,
Brian.
···
On Tue, Mar 20, 2007 at 04:40:08AM +0900, Greg Lorriman wrote:
No, that's a normal rounding error that comes with floats:
irb(main):001:0> f='40.12'.to_f
=> 40.12
irb(main):002:0> "%20.10f" % f
=> " 40.1200000000"
irb(main):003:0> "%40.30f" % f
=> " 40.119999999999997442046151263639"
irb(main):004:0> "%40.30f" % (f*100)
=> " 4011.999999999999545252649113535881"
If you look at the last line it becomes clear why you get 4011 as result.
Kind regards
robert
···
On 19.03.2007 20:37, Greg Lorriman wrote:
This
('40.12'.to_f*100).to_i
results in this integer :
4011
This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.