Float precision

Sorry if this has been answered before.
I’ll demonstrate my question with irb.

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

I tried using Bignum, but I could use some syntax help.

note on documentation: I love the programmers guide, but I really
cherish some exampels for class defenitions etc on how to use a class.

db

···


You don’t exist, go away!

It’s not missing; it’s just not printed:

irb(main):001:0> class Float; def inspect; “%.10f” % self; end; end
nil
irb(main):002:0> a = 20020912.194220
20020912.1942199990

···

On Sat, Sep 14, 2002 at 02:54:56AM +0900, Daniel Bretoi wrote:

Sorry if this has been answered before.
I’ll demonstrate my question with irb.

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

Daniel Bretoi wrote:

Sorry if this has been answered before.
I’ll demonstrate my question with irb.

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

irb(main):001:0> a = 20020912.194220
20020912.19
irb(main):002:0> printf “%.6f”, a
20020912.194220nil

The precision is still there. It must just be how irb chooses to display
numbers.

Those were omitted by Float#inspect method.
irb calls #inspect for each result and reports that without quote.

a = 20020912.194220
a == 20020912.19 #=> false
a == 20020912.194220 #=> true

Give format and you will get all digits:

irb(main):001:0> a = 20020912.194220
20020912.19
irb(main):002:0> sprintf("%.6f", a)
“20020912.194220”

ieee754hack.rb explains more details of internal representation:
http://www.notwork.org/~gotoken/ruby/p/as-is/ieee754hack.rb

– Gotoken

···

At Sat, 14 Sep 2002 02:54:56 +0900, Daniel Bretoi wrote:

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

Hi,

No, you don’t miss any precision. Internally, the Ruby Float type is
always represented by the C double type, with precision of at least
2.220446049e-16 (depends on how you define it).

You can just try this

printf "%.6f\n", a

and it will give you your last 6 digits (on display).

Regards,

Bill

···

===========================================================================
Daniel Bretoi lists@debonair.net wrote:

Sorry if this has been answered before.
I’ll demonstrate my question with irb.

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

I tried using Bignum, but I could use some syntax help.

note on documentation: I love the programmers guide, but I really
cherish some exampels for class defenitions etc on how to use a class.

db


You don’t exist, go away!

Isn’t that a dangerous pitfall?

irb(main):001:0> a = 20020912.194220
20020912.19
irb(main):002:0> a = a.to_s
"20020912.19"

if it’s all storered, shouldn’t the whole number end up in the string?

···

On Sat, Sep 14, 2002 at 04:03:19AM +0900, Paul Brannan wrote:

On Sat, Sep 14, 2002 at 02:54:56AM +0900, Daniel Bretoi wrote:

Sorry if this has been answered before.
I’ll demonstrate my question with irb.

irb(main):001:0> a = 20020912.194220
20020912.19

Why do I miss the last 5 digits when I try to assign a?

It’s not missing; it’s just not printed:

irb(main):001:0> class Float; def inspect; “%.10f” % self; end; end
nil
irb(main):002:0> a = 20020912.194220
20020912.1942199990


You don’t exist, go away!

There’s an RCR for this:

http://www.rubygarden.org/article.php?sid=231

Paul

···

On Sat, Sep 14, 2002 at 04:41:52AM +0900, Daniel Bretoi wrote:

Isn’t that a dangerous pitfall?

irb(main):001:0> a = 20020912.194220
20020912.19
irb(main):002:0> a = a.to_s
"20020912.19"

if it’s all storered, shouldn’t the whole number end up in the string?