Integer/Float oddity

ruby-talk
1

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

2

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

Two quick notes - I think you meant

(100*(1.15-1)).round.to_int
=> 15

See if this clears anything up:
printf("%.15f", (100*(1.15-1)))
14.999999999999991=> nil

The result is not really 15, but just really really close, due to the
imprecision of floats - always round moving to ints - to_i is a
"floor" function. Similarly be very careful when comparing floats.

Patrick

···

On Thu, 31 Mar 2005 23:09:46 +0900, horati0@linuxmail.org <horati0@linuxmail.org> wrote:

3

<horati0@linuxmail.org> schrieb im Newsbeitrag
news:1112277873.331258.267880@f14g2000cwb.googlegroups.com...

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

15.0.to_int
=> 15

what am i missing here? i never would have thought to use .round to
get the 'correct' result unless someone told me. by correct i mean if
i was doing this with a paper and pencil, i would expect to get 15, not
14.

any insight would be appreciated, especially a way (if possible) to
override this behaviour so i dont run into this in the future.

thanks,
horati0

This is a typical limitation of numeric calculations. The reason for what
you see is that 1.15-1 is smaller than 0.15:

(1.15-1) - 0.15

=> -8.32667268468867e-17

consequently

(100 * (1.15-1))-15

=> -8.88178419700125e-15

this is negative => (100 * (1.15-1)) is smaller than 15 and thus it's
correctly converted to 14 by to_int.

Kind regards

    robert

4

horati0@linuxmail.org wrote:

not really an oddity, more a lack of understanding on my part regarding
ruby's internal type conversion...

(100*(1.15-1))
=> 15.0

(100*(1.15-1)).to_int
=> 14

(100*(1.15-1)).to_int.round
=> 15

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

Float#to_int simply truncates, while Float#round goes to the nearest integer.

Try this: "%.20f" % (100*(1.15-1))

···

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/&gt;

5

Actually newLISP produces the correct answer:

(mul 100 (sub 1.15 1)) => 15

You probably used * and - for integer arithmetik only, instead of mul
and sub for floats.

Lutz

www.newlisp.org

when contacting me via email put the word newlisp in the subject line.

6

Glenn Parker wrote:

> not really an oddity, more a lack of understanding on my part

regarding

> ruby's internal type conversion...
>
> (100*(1.15-1))
> => 15.0
>
> (100*(1.15-1)).to_int
> => 14
>
> (100*(1.15-1)).to_int.round
> => 15

FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?

oops, yep thats what i meant. thanks for all the replies to my query;
i figured it had something to do with the internal representation of fp
numbers.

this whole thing started as a conversation i was having with my dad
about the relative merits of various programming languages. he is a
die-hard xbasic user, and pointed me to the following post on an xb
list:

···

horati0@linuxmail.org wrote:

---------------------------------------------------------------------

[snip]

Message: 7
Date: Wed, 30 Mar 2005 12:59:42 -0000
From: "Bruno Schaefer" <bup.schaefer@...>
Subject: XBasic is one of the best!

I'm back now to work with XBasic, because there are strange things
with other languages.

If you try to calculate the following simple expression

INT(100*(1.15-1))

XBasic produces the correct result: 15.

This is a calculation, which is solved by children easy.

But try this with other languages
(e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
you will get the result

14 !!!!?

What a nonsense ! Does somebody understand this ?

This is one the reason for me to use XBasic!

---------------------------------------------------------------------
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

7

michael.b.masi wrote:

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

Probably nothing about the internal representation is different. The difference would have to be in xbasic's "INT()" operator, which apparently performs rounding.

Dealing with floating point numbers can often produce counter-intuitive results, mostly because simple analogies based on fixed-point math fail. For example, the concept of "equality" in floating point math often clashes with common sense. You might assume that the following expression is true:

   (100*(1.15-1)) - 15 == 0

The computer would disagree. What you really want to test is:

   abs( (100*(1.15-1)) - 15 ) < TOLERANCE

Where the choice of TOLERANCE depends very much on the application.

···

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/&gt;

8

Nothing. It's just that Xbasic (apparently) defines INT() to mean "closest integer" and some other languages define it to mean "largest integer not exceeding". Both give results consistent with their definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

Steve

···

michael.b.masi@gmail.com wrote:

[snip]
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

9

While the behavior of xbasic in this circumstance would seem to be
"correct", and is probably useful in simple calculations, it would
make it unsuitable for serious calculations. The "correct" answer is
actually not. This is due to the way floating point operations are
calculated by computers.

Without going into too much depth, lets just say that floats are
stored as fractions in the computer's memory. Specifically, as
fractions with denominators that are powers of 2; (x/2^n). Just as
some fractions don't translate precisely to decimal numbers (1/2
becomes 0.5, but 1/7 becomes 0.142857142857142...), many decimal
numbers don't translate precisely to binary numbers (0.5 becomes
0.1b2, but 0.3 becomes 0.00100110011b2...). This causes rounding
errors.

To avoid this, use rational numbers, which are precise fractions:
require 'mathn'
    ==> true
num = 115/100
    ==>23/20
num -= 1
    ==>3/20
num * 100
    ==>15

cheers,
Mark

···

On Fri, 1 Apr 2005 00:59:45 +0900, michael.b.masi@gmail.com <michael.b.masi@gmail.com> wrote:

i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to this
seemingly simple number manipulation?

horati0

10

Glenn Parker wrote:
> > not really an oddity, more a lack of understanding on my part
regarding
> > ruby's internal type conversion...
> >
> > (100*(1.15-1))
> > => 15.0
> >
> > (100*(1.15-1)).to_int
> > => 14
> >
> > (100*(1.15-1)).to_int.round
> > => 15
>
> FWIW, I get 14. Perhaps you meant (100*(1.15-1)).round.to_int ?
>

oops, yep thats what i meant. thanks for all the replies to my

query;

i figured it had something to do with the internal representation of

fp

numbers.

this whole thing started as a conversation i was having with my dad
about the relative merits of various programming languages. he is a
die-hard xbasic user, and pointed me to the following post on an xb
list:

---------------------------------------------------------------------

[snip]

Message: 7
Date: Wed, 30 Mar 2005 12:59:42 -0000
From: "Bruno Schaefer" <bup.schaefer@...>
Subject: XBasic is one of the best!

I'm back now to work with XBasic, because there are strange things
with other languages.

If you try to calculate the following simple expression

INT(100*(1.15-1))

XBasic produces the correct result: 15.

This is a calculation, which is solved by children easy.

But try this with other languages
(e.g. newLISP, Visual Basic, Liberty Basic, IBasic, Yabasic, Python)
you will get the result

14 !!!!?

What a nonsense ! Does somebody understand this ?

This is one the reason for me to use XBasic!

---------------------------------------------------------------------
i'm assuming that since this behaviour is exhibited in ruby and the
rest of the above "other languages" and NOT xbasic that it is really
due to an idiosyncracy of xbasic that the "right" answer comes out.

so the question becomes, what is different about xbasic's internal
numeric representation that allows it to give a correct answer to

this

···

michael.b.masi@gmail.com wrote:

> horati0@linuxmail.org wrote:
seemingly simple number manipulation?

horati0

11

Still, one is considered the standard (which can be translated as
"correct"), and the other isn't. The standardized behavior is to
truncate, if you count on that and it rounds, your computations are
not going to come out the same. IIRC, the IEEE standard for floating
point numbers specifies how they should be converted to integers.

cheers,
Mark

···

On Fri, 1 Apr 2005 04:23:00 +0900, Steven Jenkins <steven.jenkins@ieee.org> wrote:

michael.b.masi@gmail.com wrote:
> [snip]
> i'm assuming that since this behaviour is exhibited in ruby and the
> rest of the above "other languages" and NOT xbasic that it is really
> due to an idiosyncracy of xbasic that the "right" answer comes out.
>
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to this
> seemingly simple number manipulation?

Nothing. It's just that Xbasic (apparently) defines INT() to mean
"closest integer" and some other languages define it to mean "largest
integer not exceeding". Both give results consistent with their
definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

12

Mark Hubbart wrote:

Nothing. It's just that Xbasic (apparently) defines INT() to mean
"closest integer" and some other languages define it to mean "largest
integer not exceeding". Both give results consistent with their
definitions. Either one gives you what you need to round *or* truncate.

round(x) is equivalent to truncate(x + .5)

Still, one is considered the standard (which can be translated as
"correct"), and the other isn't. The standardized behavior is to
truncate, if you count on that and it rounds, your computations are
not going to come out the same. IIRC, the IEEE standard for floating
point numbers specifies how they should be converted to integers.

I don't think it does. IEEE 754 specifies how rounding is applied in floating-point calculations, but does not define an API.

No standard requires *every* language to have an operator called INT() that truncates. (C doesn't, for example.) Absent that, Xbasic can define INT() however it wants. You might wish they'd chosen differently, but that doesn't make it wrong.

Also note that ISO C defines various math library functions (e.g., floor()), but does not mandate IEEE 754 compliance.

Steve