Ruby 1.6.8 vs Ruby 1.8.0 preview 2 - benchmarks

Hi all,

One of the concerns I’m having about Ruby 1.8 is that it appears to be
getting slower. Not so much that it will be a serious issue for me,
but Ruby’s speed has been something some of the Perl and Python folks
have been using against Ruby. I visited the Great Computer Language
Shootout (http://www.bagley.org/~doug/shootout/), put together 9 of
the functions he used and ran them using both 1.6.8 and 1.8 p2.

In 7 of the 9 tests, 1.8 was slower. Should I be worried?

Here is a sampling of the results. Complete code at the bottom. I
ran the benchmarks and almost always came up with the same results -
1.6.8 was faster in 7 of 9 tests (and sometimes 8). The one place
where I noticed a massive improvement, however, was in the 'list’
test.

Dual Pentium II 400
512 MB RAM
Mandrake Linux 9.0

1.6.8

                   user     system      total        real

Ackermann function: 1.430000 0.000000 1.430000 ( 1.450032)
Array access: 9.010000 0.020000 9.030000 ( 9.139022)
Fibonacci numbers: 8.940000 0.010000 8.950000 ( 9.036173)
Hash access I: 7.260000 0.000000 7.260000 ( 7.241685)
Hash access II: 9.960000 0.060000 10.020000 ( 10.047116)
Lists: 19.250000 0.050000 19.300000 ( 19.398597)
Nested loop: 10.460000 0.010000 10.470000 ( 10.538426)
Sieve of Eratosthenes: 13.880000 0.040000 13.920000 ( 13.926563)
Word Frequency: 2.700000 0.000000 2.700000 ( 2.669599)

1.8.0 preview 2

                   user     system      total        real

Ackermann function: 1.560000 0.000000 1.560000 ( 1.545286)
Array access: 10.020000 0.010000 10.030000 ( 10.017142)
Fibonacci numbers: 9.130000 0.010000 9.140000 ( 9.201564)
Hash access I: 7.960000 0.010000 7.970000 ( 8.046752)
Hash access II: 9.780000 0.030000 9.810000 ( 9.939460)
Lists: 2.320000 0.020000 2.340000 ( 2.337373)
Nested loop: 11.850000 0.000000 11.850000 ( 11.849551)
Sieve of Eratosthenes: 15.030000 0.040000 15.070000 ( 15.065020)
Word Frequency: 2.990000 0.020000 3.010000 ( 3.003743)

benchmark.rb

require "gcls"
require "benchmark"
include Benchmark

max = 300000
bm do |x|
x.report("Ackermann function: "){
max.times{ ack }
}
x.report("Array access: "){
1000.times{ array_access }
}
x.report("Fibonacci numbers: "){
100.times{ fib }
}
x.report("Hash access I: "){
10000.times{ hash_access_I }
}
x.report("Hash access II: "){
5.times{ hash_access_II }
}
x.report("Lists: "){
3.times{
for iter in 1…10
result = lists
end
}
}
x.report("Nested loop: "){
5.times{ nested_loop }
}
x.report("Sieve of Eratosthenes: "){
10.times{ sieve_of_eratosthenes }
}
x.report("Word Frequency: "){
1000.times{ word_frequency }
}
end

gcls.rb

Ackermann function

def ack(m=0, n=0)
if m == 0 then
n + 1
elsif n == 0 then
ack(m - 1, 1)
else
ack(m - 1, ack(m, n - 1))
end
end

Array access

def array_access(n=1)
x = Array.new(n)
y = Array.new(n, 0)

for i in 0…n
x[i] = i + 1
end

for k in 0…999
(n-1).step(0,-1) do |i|
y[i] = y.at(i) + x.at(i)
end
end
end

Fibonacci numbers

def fib(n=20)
if n < 2 then
1
else
fib(n-2) + fib(n-1)
end
end

Hash Access I

def hash_access_I(n=20)
hash = {}
for i in 1…n
hash[’%x’ % i] = 1
end

c = 0
n.downto 1 do |i|
c += 1 if hash.has_key? i.to_s
end
end

Hash Access II

def hash_access_II(n=20)
hash1 = {}
for i in 0 … 9999
hash1[“foo_” << i.to_s] = i
end

hash2 = Hash.new(0)
n.times do
for k in hash1.keys
hash2[k] += hash1[k]
end
end
end

lists

SIZE = 10000
def lists
li1 = (1…SIZE).to_a
li2 = li1.dup
li3 = Array.new

while (not li2.empty?)
li3.push(li2.shift)
end

while (not li3.empty?)
li2.push(li3.pop)
end

li1.reverse!

if li1[0] != SIZE then
p "not SIZE"
return(0)
end

if li1 != li2 then
return(0)
end

return(li1.length)
end

def nested_loop(n = 10)
x = 0
n.times do
n.times do
n.times do
n.times do
n.times do
n.times do
x += 1
end
end
end
end
end
end
end

def sieve_of_eratosthenes(n=20)
count = i = j = 0
flags0 = Array.new(8192,1)

n.times do
count = 0
flags = flags0.dup
for i in 2 … 8192
next unless flags[i]
(i+i).step(8192, i) do |j|
flags[j] = nil
end
count = count + 1
end
end
end

def statistical_moments
sum = 0.0
nums = []
num = nil

for line in STDIN.readlines()
num = Float(line)
nums << num
sum += num
end

n = nums.length()
mean = sum/n;
deviation = 0.0
average_deviation = 0.0
standard_deviation = 0.0
variance = 0.0
skew = 0.0
kurtosis = 0.0

for num in nums
deviation = num - mean
average_deviation += deviation.abs()
variance += deviation2;
skew += deviation
3;
kurtosis += deviation**4
end

average_deviation /= n
variance /= (n - 1)
standard_deviation = Math.sqrt(variance)

if (variance > 0.0)
skew /= (n * variance * standard_deviation)
kurtosis = kurtosis/(n * variance * variance) - 3.0
end

nums.sort()
mid = n / 2

if (n % 2) == 0
median = (nums.at(mid) + nums.at(mid-1))/2
else
median = nums.at(mid)
end
end

def word_frequency
data = “While the word Machiavellian suggests cunning, duplicity,
or bad faith, it would be unfair to equate the word with the man. Old
Nicoló was actually a devout and principled man, who had profound
insight into human nature and the politics of his time. Far more
worthy of the pejorative implication is Cesare Borgia, the incestuous
and multi-homicidal pope who was the inspiration for The Prince. You
too may ponder the question that preoccupied Machiavelli: can a
government stay in power if it practices the morality that it preaches
to its people?“
freq = Hash.new(0)
for word in data.downcase.tr_s(’^A-Za-z’,’ ‘).split(’ ')
freq[word] += 1
end
freq.delete(””)
lines = Array.new
freq.each{|w,c| lines << sprintf("%7d\t%s\n", c, w) }
end

Regards,

Dan

In article 6e613a32.0303282059.d94f726@posting.google.com,

···

Daniel Berger djberg96@hotmail.com wrote:

Hi all,

One of the concerns I’m having about Ruby 1.8 is that it appears to be
getting slower. Not so much that it will be a serious issue for me,
but Ruby’s speed has been something some of the Perl and Python folks
have been using against Ruby. I visited the Great Computer Language
Shootout (http://www.bagley.org/~doug/shootout/), put together 9 of
the functions he used and ran them using both 1.6.8 and 1.8 p2.

In 7 of the 9 tests, 1.8 was slower. Should I be worried?

Here is a sampling of the results. Complete code at the bottom. I
ran the benchmarks and almost always came up with the same results -
1.6.8 was faster in 7 of 9 tests (and sometimes 8). The one place
where I noticed a massive improvement, however, was in the ‘list’
test.

Dual Pentium II 400
512 MB RAM
Mandrake Linux 9.0

1.6.8

                  user     system      total        real

Ackermann function: 1.430000 0.000000 1.430000 ( 1.450032)
Array access: 9.010000 0.020000 9.030000 ( 9.139022)
Fibonacci numbers: 8.940000 0.010000 8.950000 ( 9.036173)
Hash access I: 7.260000 0.000000 7.260000 ( 7.241685)
Hash access II: 9.960000 0.060000 10.020000 ( 10.047116)
Lists: 19.250000 0.050000 19.300000 ( 19.398597)
Nested loop: 10.460000 0.010000 10.470000 ( 10.538426)
Sieve of Eratosthenes: 13.880000 0.040000 13.920000 ( 13.926563)
Word Frequency: 2.700000 0.000000 2.700000 ( 2.669599)

1.8.0 preview 2

                  user     system      total        real

Ackermann function: 1.560000 0.000000 1.560000 ( 1.545286)
Array access: 10.020000 0.010000 10.030000 ( 10.017142)
Fibonacci numbers: 9.130000 0.010000 9.140000 ( 9.201564)
Hash access I: 7.960000 0.010000 7.970000 ( 8.046752)
Hash access II: 9.780000 0.030000 9.810000 ( 9.939460)
Lists: 2.320000 0.020000 2.340000 ( 2.337373)
Nested loop: 11.850000 0.000000 11.850000 ( 11.849551)
Sieve of Eratosthenes: 15.030000 0.040000 15.070000 ( 15.065020)
Word Frequency: 2.990000 0.020000 3.010000 ( 3.003743)

That is somewhat worrisome… I was hoping we were moving the other
direction.

Phil

In 7 of the 9 tests, 1.8 was *slower*. Should I be worried?

One of the reason is the suppression of some #=== methods : in this case
ruby call Kernel#=== which make a rb_funcall2("=="), this give 2 funcall
rather than one if I'm right

Another reason is the rewrite of fix_step, int_step in num_step which give
a highter number of call to rb_equal

Probably it exist other reasons : I've just looked why there is a big
difference in the call to rb_equal()

Guy Decoux

Yikes. Somehow, somewhere that needs to change. I have be reading on Python
2.3 and it is supposedly 10-15% faster than 2.2 because they are focusing on
speeding things up.

Just out of interest, I ran your tests on my system (FreeBSD-4.7, P266MMX,
128MB RAM) and the results are below. I installed the 16->18 shim to get the
benchmark module.

Looking at the “total time” column they are both pretty similar, with 1.8
faster in five of the tests, very much so in the case of ‘Lists’, and only
significantly slower in two (Hash Access I and Word Frequency)

For some reason the “real” time column is slower. e.g.
Array access: total user+cpu time: 17.22 —> 16.95
real time: 17.63 —> 17.97

There must be something funny going on - the program does the tiniest amount
of I/O so it’s hard to see what it’s blocking on. I think it may be because
of the following warning which is generated twice for each test:

/usr/local/lib/ruby/site_ruby/1.8/benchmark.rb:435: warning: obsolete method Time::times; use Process::times

It probably makes more sense to compare the CPU used rather than the actual
absolute execution time anyway.

Regards,

Brian.

[brian@vaio ruby]$ ruby -v benchmark.rb
ruby 1.6.8 (2003-01-27) [i386-freebsd4.7]
user system total real
Ackermann function: 2.500000 0.000000 2.500000 ( 2.656021)
Array access: 17.203125 0.015625 17.218750 ( 17.631144)
Fibonacci numbers: 16.625000 0.007812 16.632812 ( 16.901553)
Hash access I: 14.890625 0.007812 14.898438 ( 15.144399)
Hash access II: 19.945312 0.031250 19.976562 ( 20.313897)
Lists: 55.296875 0.039062 55.335938 ( 56.143935)
Nested loop: 18.265625 0.000000 18.265625 ( 18.525846)
Sieve of Eratosthenes: 27.195312 0.054688 27.250000 ( 27.721882)
Word Frequency: 5.726562 0.007812 5.734375 ( 5.821429)

[brian@vaio ruby]$ ruby18 -v benchmark.rb
ruby 1.8.0 (2003-03-03) [i386-freebsd4.7]
user system total real
Ackermann function: 2.578125 0.000000 2.578125 ( 2.972841)
Array access: 16.921875 0.023438 16.945312 ( 17.969264)
Fibonacci numbers: 15.703125 0.015625 15.718750 ( 16.443847)
Hash access I: 17.484375 0.015625 17.500000 ( 18.225830)
Hash access II: 19.109375 0.054688 19.164062 ( 19.871894)
Lists: 4.304688 0.015625 4.320312 ( 4.715083)
Nested loop: 18.023438 0.007812 18.031250 ( 18.852277)
Sieve of Eratosthenes: 23.718750 0.093750 23.812500 ( 24.495048)
Word Frequency: 6.773438 0.023438 6.796875 ( 7.650401)

···

On Sat, Mar 29, 2003 at 02:04:16PM +0900, Daniel Berger wrote:

One of the concerns I’m having about Ruby 1.8 is that it appears to be
getting slower. Not so much that it will be a serious issue for me,
but Ruby’s speed has been something some of the Perl and Python folks
have been using against Ruby. I visited the Great Computer Language
Shootout (http://www.bagley.org/~doug/shootout/), put together 9 of
the functions he used and ran them using both 1.6.8 and 1.8 p2.

In 7 of the 9 tests, 1.8 was slower. Should I be worried?

Any idea what in particular they have focused on?

Gavin

···

On Sunday, March 30, 2003, 1:05:57 AM, Bob wrote:

Yikes. Somehow, somewhere that needs to change. I have be reading on Python
2.3 and it is supposedly 10-15% faster than 2.2 because they are focusing on
speeding things up.

In 7 of the 9 tests, 1.8 was slower. Should I be worried?

One of the reason is the suppression of some #=== methods : in this case
ruby call Kernel#=== which make a rb_funcall2(“==”), this give 2 funcall
rather than one if I’m right

Another reason is the rewrite of fix_step, int_step in num_step which give
a highter number of call to rb_equal

why this changes had been applied?

···

il Sat, 29 Mar 2003 22:43:57 +0900, ts decoux@moulon.inra.fr ha scritto::

Guy Decoux

For some reason the “real” time column is slower. e.g.
Array access: total user+cpu time: 17.22 —> 16.95
real time: 17.63 —> 17.97

There must be something funny going on - the program does the tiniest amount
of I/O so it’s hard to see what it’s blocking on. I think it may be because
of the following warning which is generated twice for each test:

/usr/local/lib/ruby/site_ruby/1.8/benchmark.rb:435: warning: obsolete method Time::times; use Process::times

That was my broken ruby16->18 shim installation, which I’ve now fixed. But I
also realised that I was running under X, with KDE and Mozilla eating up
chunks of memory, and hence likely to mess things up especially with regards
to garbage collection.

So, exiting out of all that, and running from a text console, I get the
following results:

[1.6.8]
user system total real
Ackermann function: 2.523438 0.000000 2.523438 ( 2.519697)
Array access: 17.882812 0.015625 17.898438 ( 17.928351)
Fibonacci numbers: 16.078125 0.007812 16.085938 ( 16.114064)
Hash access I: 15.031250 0.007812 15.039062 ( 15.086221)
Hash access II: 20.273438 0.046875 20.320312 ( 20.425834)
Lists: 57.328125 0.007812 57.335938 ( 57.639749)
Nested loop: 18.125000 0.000000 18.125000 ( 18.162831)
Sieve of Eratosthenes: 27.914062 0.007812 27.921875 ( 27.968728)
Word Frequency: 5.820312 0.023438 5.843750 ( 5.856217)

[1.8.0p2]
user system total real
Ackermann function: 2.609375 0.000000 2.609375 ( 2.611862)
Array access: 16.226562 0.000000 16.226562 ( 16.235060)
Fibonacci numbers: 16.132812 0.000000 16.132812 ( 16.153930)
Hash access I: 17.500000 0.000000 17.500000 ( 17.587397)
Hash access II: 19.101562 0.031250 19.132812 ( 19.182812)
Lists: 4.281250 0.015625 4.296875 ( 4.535421)
Nested loop: 17.906250 0.000000 17.906250 ( 17.917728)
Sieve of Eratosthenes: 23.765625 0.023438 23.789062 ( 23.820149)
Word Frequency: 6.742188 0.007812 6.750000 ( 6.861685)

Now the “total CPU” and “real” (elapsed) times are very similar, which is
what I’d expect. I still see 1.8 faster in 5 out of 9 tests, so on this
evidence I’m not hugely concerned about moving to 1.8

I compiled both 1.6.8 and 1.8.0p2 from source, on the same machine, with the
same version of GCC (2.95.4) and libraries. This is worth thinking about -
you don’t want to end up comparing the optimisation performance of two
different C compilers, rather than the code itself!

Regards,

Brian.

“Gavin Sinclair” gsinclair@soyabean.com.au wrote in message
news:34454318134.20030330012449@soyabean.com.au…

Yikes. Somehow, somewhere that needs to change. I have be reading on
Python
2.3 and it is supposedly 10-15% faster than 2.2 because they are
focusing on
speeding things up.

Any idea what in particular they have focused on?

Gavin

The link: Radar – O’Reilly

···

On Sunday, March 30, 2003, 1:05:57 AM, Bob wrote:

why this changes had been applied?

Well, if I'm right (don't forget that I'm stupid) and if one reason for
the slowdown is the call to rb_equal() then it's easy to understand the
change

Imagine that in your source you have different classes and for these
classes by default #== and #=== are the same method you can write

    rb_define_method(rb_cKernel, "===", rb_equal, 1);

    rb_define_method(rb_cA, "==", a_equal, 1);
    rb_define_method(rb_cA, "===", a_equal, 1);
    /* ... */
    rb_define_method(rb_cB, "==", b_equal, 1);
    rb_define_method(rb_cB, "===", b_equal, 1);
    /* ... */

Then you think that it's just stupid and you write

    rb_define_method(rb_cKernel, "===", rb_equal, 1);

    rb_define_method(rb_cA, "==", a_equal, 1);
    /* ... */
    rb_define_method(rb_cB, "==", b_equal, 1);
    /* ... */

and you rely on the fact that rb_equal() will call #==

but now when you call A#=== ruby first call Kernel#=== which call A#==,
rather than calling directly A#===

If this method is frequently called you can see the difference.

Probably another reason is that Kernel#=== was modified, in 1.6 Kernel#===
is the same than Kernel#== (rb_obj_equal()) and don't call rb_equal() and
the classes must define #=== if they want a different behavior that
Kernel#===

This is why I don't trust benchmark, they show artificial things which
don't reflect fatally what you use in a script

Guy Decoux

thanks!

···

il Sun, 30 Mar 2003 02:27:22 +0900, ts decoux@moulon.inra.fr ha scritto::

why this changes had been applied?