Parrot SMOP benchmark

I was just reading Dan Sugalski’s slides from RubyConf 2002 and noticed the benchmarks relating to something called “SMOP” at the end of the article.

It has Ruby 1.6.6 being the slowest at 206s versus Perl 5.6.0 at 165s and Python 2.2 at 136s. Has anyone tried running it against 1.8.0 … yes, I know it’s only been out for a few hours :-). Might give us a feel for how well some of the performance improvements work in practice.

Does anyone know where to get the Ruby code for this SMOP thingy?

The thing that really gives me hope is the final figure … Parrot(fastest) 4.9s !! It would be interesting to see a figure for Java, assuming it can actually do everything in SMOP.

Harry O.

I was just reading Dan Sugalski’s slides from RubyConf 2002 and
noticed the benchmarks relating to something called “SMOP” at the
end of the article.

It has Ruby 1.6.6 being the slowest at 206s versus Perl 5.6.0 at
165s and Python 2.2 at 136s. Has anyone tried running it against
1.8.0 … yes, I know it’s only been out for a few hours :-). Might
give us a feel for how well some of the performance improvements
work in practice.

Does anyone know where to get the Ruby code for this SMOP thingy?

It’s in the parrot CVS repository. A couple of dozen lines of code, a
bit too big in its current state to cut’n’paste in.

The thing that really gives me hope is the final figure …
Parrot(fastest) 4.9s !! It would be interesting to see a figure for
Java, assuming it can actually do everything in SMOP.

It only counts from 10 million to 0–I think Java can manage that. :slight_smile:

···

At 8:22 AM +0900 8/5/03, Harry Ohlsen wrote:

                                     Dan

--------------------------------------“it’s like this”-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk

Harry Ohlsen harryo@qiqsolutions.com writes:

I was just reading Dan Sugalski’s slides from RubyConf 2002 and
noticed the benchmarks relating to something called “SMOP” at the
end of the article.

It has Ruby 1.6.6 being the slowest at 206s versus Perl 5.6.0 at
165s and Python 2.2 at 136s. Has anyone tried running it against
1.8.0 … yes, I know it’s only been out for a few hours :-). Might
give us a feel for how well some of the performance improvements
work in practice.

Does anyone know where to get the Ruby code for this SMOP thingy?

Here are the results on my machine (P4 @ 1.6 GHz, 512MB, Debian
unstable with kernel 2.4.21). Ruby 1.8.0 is freshly compiled using gcc
3.3.1 20030728 (Debian prerelease), the other interpreters are the
current Debian unstable versions:

ruby 1.6.8 (2003-07-09) [i386-linux]
Iterations: 10000000
Estimated ops: 20000000
Elapsed time: 10.052047
M op/s: 1.989644497

ruby 1.8.0 (2003-08-04) [i686-linux]
Iterations: 10000000
Estimated ops: 20000000
Elapsed time: 7.29644
M op/s: 2.74106276485519

perl 5.8.0 built for i386-linux-thread-multi
Iterations: 100000000
Estimated ops: 200000000
Elapsed time: 66
M op/s: 3.03030303030303

Python 2.2.3+ (#1, Jul 5 2003, 11:04:18)
Iterations: 10000000
Estimated ops: 20000000
Elapsed time: 6.38665604591
M op/s: 3.13152921595

···


Pierre-Charles David (pcdavid emn fr)
Computer Science PhD Student, École des Mines de Nantes, France
Homepage: http://pcdavid.net/

Harry Ohlsen harryo@qiqsolutions.com writes:

I was just reading Dan Sugalski’s slides from RubyConf 2002 and noticed the
benchmarks relating to something called “SMOP” at the end of the article.

Remember that MOPS only test the speed of the dispatch loop. If you have
one op called do_entire_program, it doesn’t really matter how slowly you
dispatch it.

···


“If that makes any sense to you, you have a big problem.”
– C. Durance, Computer Science 234

mops for ruby (cvsweb):

http://cvs.perl.org/cvsweb/parrot/examples/mops/
mops.rb?rev=1.2&content-type=text/x-cvsweb-markup

Machine: PowerPC 1GHz, 1GB RAM, Ruby 1.8.0

Result:
Iterations: 10000000
Estimated ops: 20000000
Elapsed time: 23.52137
M op/s: 0.8502906080725741

-rich

···

On Monday, August 4, 2003, at 07:29 PM, Dan Sugalski wrote:

It’s in the parrot CVS repository. A couple of dozen lines of code, a
bit too big in its current state to cut’n’paste in.

Thanks Pierre-Charles.

So, it looks like Ruby’s speed (for this trivial example) has improved by
about 30% or a bit more. That’s pretty good, I reckon.

ruby 1.6.8 (2003-07-09) [i386-linux]
M op/s: 1.989644497

ruby 1.8.0 (2003-08-04) [i686-linux]
M op/s: 2.74106276485519

The comparisons against perl and python are less interesting, apart from the
fact that we can see we’re still slightly behind them, but getting pretty
close!

···

On Tue, 5 Aug 2003 17:18, Pierre-Charles David wrote:

perl 5.8.0 built for i386-linux-thread-multi
M op/s: 3.03030303030303

Python 2.2.3+ (#1, Jul 5 2003, 11:04:18)
M op/s: 3.13152921595

Elapsed time: 23.52137

And from earlier:

It has Ruby 1.6.6 being the slowest at 206s versus Perl 5.6.0 at 165s
and Python 2.2 at 136s. Has anyone tried running it against 1.8.0 …
yes, I know it’s only been out for a few hours :-). Might give us a
feel for how well some of the performance improvements work in
practice.

Does anyone know where to get the Ruby code for this SMOP thingy?

To be clear here…

Is this saying that Ruby has increased in speed by an order of
magnitude? That it is now 5 times faster than Python 2.2 and almost 7
times faster than Perl?

If so, wow.

Ben

···

On Tuesday, August 5, 2003, at 12:06 AM, Richard Kilmer wrote:

Harry Ohlsen wrote:

Thanks Pierre-Charles.

So, it looks like Ruby’s speed (for this trivial example) has improved by
about 30% or a bit more. That’s pretty good, I reckon.

ruby 1.6.8 (2003-07-09) [i386-linux]
M op/s: 1.989644497

ruby 1.8.0 (2003-08-04) [i686-linux]
M op/s: 2.74106276485519

The comparisons against perl and python are less interesting, apart from
the fact that we can see we’re still slightly behind them, but getting
pretty close!

to be fair Python 2.3 should be used. according to python.org
Python 2.3 should be about 20-30% faster than Python 2.2

Markus

···

On Tue, 5 Aug 2003 17:18, Pierre-Charles David wrote:

perl 5.8.0 built for i386-linux-thread-multi
M op/s: 3.03030303030303

Python 2.2.3+ (#1, Jul 5 2003, 11:04:18)
M op/s: 3.13152921595

So, it looks like Ruby’s speed (for this trivial example) has improved by
about 30% or a bit more. That’s pretty good, I reckon.

ruby 1.6.8 (2003-07-09) [i386-linux]
M op/s: 1.989644497

ruby 1.8.0 (2003-08-04) [i686-linux]
M op/s: 2.74106276485519

I’m not so sure. See that ruby 1.6.8 from Debian is compiled withouth any
i686 optimizations.

[…]

Is this saying that Ruby has increased in speed by an
order of magnitude? That it is now 5 times faster than
Python 2.2 and almost 7 times faster than Perl?

I think it means you should run it for each language not
just one. I would presume the time differences would change
according to the machine on which you run.

cheers,

···


Iain.

No, of course not. It means that the machine this test was run on was
significantly faster than the one I ran the test on. (I used the
stock ruby build shipped with Mac OS X on my iBook, which is rather
significantly slower than the machine used to run this test)

···

At 1:39 PM +0900 8/5/03, Ben Giddings wrote:

On Tuesday, August 5, 2003, at 12:06 AM, Richard Kilmer wrote:

Elapsed time: 23.52137

And from earlier:

It has Ruby 1.6.6 being the slowest at 206s versus Perl 5.6.0 at
165s and Python 2.2 at 136s. Has anyone tried running it against
1.8.0 … yes, I know it’s only been out for a few hours :-).
Might give us a feel for how well some of the performance
improvements work in practice.

Does anyone know where to get the Ruby code for this SMOP thingy?

To be clear here…

Is this saying that Ruby has increased in speed by an order of magnitude?


Dan

--------------------------------------“it’s like this”-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk

Carlos angus@quovadis.com.ar writes:

So, it looks like Ruby’s speed (for this trivial example) has improved by
about 30% or a bit more. That’s pretty good, I reckon.

ruby 1.6.8 (2003-07-09) [i386-linux]
M op/s: 1.989644497

ruby 1.8.0 (2003-08-04) [i686-linux]
M op/s: 2.74106276485519

I’m not so sure. See that ruby 1.6.8 from Debian is compiled withouth any
i686 optimizations.

I had not used any special compilation flag either, but just to be
sure I’ve recompiled a stock Ruby 1.6.8 in the same environment, and
it seems you’re right, the speedup is less impressive:

ruby 1.6.8 (2002-12-24) [i686-linux]
M op/s: 2.254698905

ruby 1.8.0 (2003-08-04) [i686-linux]
M op/s: 2.64777391717631

That’s a 17.5% speedup, instead of about 37% for the previous measure.
Still quite nice. The date tag of the Debian version is different
though, so maybe Debian uses a patched version, which could explain
the difference.

···


Pierre-Charles David (pcdavid emn fr)
Computer Science PhD Student, École des Mines de Nantes, France
Homepage: http://pcdavid.net/