Greetings all.
I'm having a little problem with numeric precision and being a NOOB to
Ruby, I'm not quite sure where I am going wrong.
When I perform the following calculation:
n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_f
res = (n / d).to_f
Ruby is returning a value of 249,419,480,327.0
However, the more precise number I am expecting is 249,419,480,328.784
How do I do this to get the more precise value?
Thanks.
Doug
--
Posted via http://www.ruby-forum.com/.
Greetings all.
I'm having a little problem with numeric precision and being a NOOB to
Ruby, I'm not quite sure where I am going wrong.When I perform the following calculation:
n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_fres = (n / d).to_f
Ruby is returning a value of 249,419,480,327.0
However, the more precise number I am expecting is 249,419,480,328.784
How do I do this to get the more precise value?
You can use rational:
require 'rational'
=> true
n = (9703 * 16346204482307500) + 1
=> 158607222091829672501
d = 65537
=> 65537
res = (n / d)
=> 2420117217630188
res.to_f
=> 2.42011721763019e+15
Floating point arithmetic always has some numeric issues.
robert
2006/5/10, doug meharry <doug@dmeharry.com>:
--
Have a look: Robert K. | Flickr
doug meharry wrote:
Greetings all.
I'm having a little problem with numeric precision and being a NOOB to
Ruby, I'm not quite sure where I am going wrong.When I perform the following calculation:
n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_fres = (n / d).to_f
Ruby is returning a value of 249,419,480,327.0
However, the more precise number I am expecting is 249,419,480,328.784
How do I do this to get the more precise value?
Thanks.
Doug
Oops! Never mind. I some of the numbers in my preceeding message were
slightly off. This produced the expected results.
n = ((9704.0 * 1684654692600.0) + 1).to_f
d = (65537.0).to_f
res = (n / d).to_f
Ruby is returning a value of 249,445,185,727.0 which is what I also
think it should be.
--
Posted via http://www.ruby-forum.com/\.