I'm sure this has been hashed out somewhere on the list before, but
my searches have turned up fruitless. I had a strange issue come up
while diagnosing a unit test failure.
Notice the first dialog compared with the rest of this IRB session.
What causes -3930.0 to be converted to a Fixnum as -3929 only after
it's been multiplied by 100 or 100.0? Even weirder is that it seems
to only be *this* number. Can I workaround this?
I'm sure this has been hashed out somewhere on the list before, but
my searches have turned up fruitless. I had a strange issue come up
while diagnosing a unit test failure.
Notice the first dialog compared with the rest of this IRB session.
What causes -3930.0 to be converted to a Fixnum as -3929 only after
it's been multiplied by 100 or 100.0? Even weirder is that it seems
to only be *this* number. Can I workaround this?
Rounding errors are typical for floating point numbers. In this case the result of -39.30*100 ist a little bit smaller than 3930, so to_i works correct.
Rounding errors are typical for floating point numbers. In this case
the result of -39.30*100 ist a little bit smaller than 3930, so to_i
works correct.
When I think of rounding error, I think of lost precision due to
rounding too early in a series of floating point calculations. When
multiplying a non-repeating Float point by 100, shouldn't the decimal
move over two places without introducing extra precision?
And how can 39.30*100 really be 3930.000000000000454747350886464...,
but 39.31*100 be 3931.0?
BigDecimal seems to be unaffected, so I guess I'll use that as a
workaround:
Rounding errors are typical for floating point numbers. In this case
the result of -39.30*100 ist a little bit smaller than 3930, so to_i
works correct.
When I think of rounding error, I think of lost precision due to
rounding too early in a series of floating point calculations. When
multiplying a non-repeating Float point by 100, shouldn't the decimal
move over two places without introducing extra precision?
--
we can deny everything, except that we have the possibility of being better.
simply reflect on that.
- the dalai lama
Because 39.30 is a base 10 representation of a number and there
is no exact representation of that number in base 2. The Ruby
parser has to do a conversion to base 2 for 39.30 long before
the multiplication occurs so you aren't really multiplying
39.30*100 but instead you are multiplying
> irb(main):001:0> 3930-39.30*100
> => 4.54747350886464e-013
>
> Rounding errors are typical for floating point numbers. In this case
> the result of -39.30*100 ist a little bit smaller than 3930, so to_i
> works correct.
When I think of rounding error, I think of lost precision due to
rounding too early in a series of floating point calculations. When
multiplying a non-repeating Float point by 100, shouldn't the decimal
move over two places without introducing extra precision?
And how can 39.30*100 really be 3930.000000000000454747350886464...,
Almost all programming languages use binary floating point, so 0.30
can't be
represented exactly. A very few languages use binary-coded decimal.
Examples: Decimal BASIC and Business BASIC (I think). The BASIC
that came with the old 8-bit Atari used binary-coded decimal, so it
would
have had no problem with 3930-39.30*100.
Since computers of today are so much faster than 8-bit computers,
one would think that they could afford to incur the speed penalty
associated with binary-coded decimal.