I just ran the benchmark suite that comes with Ruby 1.9 on my 32-bit
machine (1.3 GHz Athlon Thunderbird). The two versions of Ruby are
1.8.6-p110 and Ruby 1.9.0 revision 13500 from the SVN repository. Both
were compiled "-march=athlon-tbird -O3" with gcc 4.2.0. Here are the
numbers -- if you've got OpenOffice, I've attached the spreadsheet:
Notes:
Column 1 is the benchmark name
Column 2 is the time for 1.8
Column 3 is the time for 1.9
Column 4 is the ratio of 1.8 time / 1.9 time
Column 5 is the log to the base e (natural log) of the ratio.
The statistics below are derived from these natural logs. Specifically,
Min is the worst case ratio from the test suite, Max is the best case
ratio from the test suite, and Pxx are the percentiles. What these mean
is that xx percent of the time, you can expect the ratio to be *below*
that value. So, since P50 is 5.1, half of the time you can expect the
ratio to be below 5.1 and half of the time you can expect it to be above
5.1. Only 25 percent of the time is the ratio below 2.8 percent, etc.
So ... in a nutshell ... 1.9.0 is five times as fast as 1.8.6!
I can do fancier stuff in R, but I probably won't -- 5x is easy to remember.
So ... in a nutshell ... 1.9.0 is five times as fast as 1.8.6!
I'd be careful about making that assertion. The benchmarks in Ruby 1.9 are specific areas that Koichi has optimized to make 1.9 run them faster. There are other areas, especially as regards core classes and evaluated code, where 1.9 is no faster or even slower.
Not trying to rain on the parade, but even Koichi has recommended not taking the 1.9 benchmark numbers as indicative of overall performance.
I can do fancier stuff in R, but I probably won't -- 5x is easy to remember.
Some warning:
1. Results of vm[12]_* benchmarks should be subtracted
loop_whileloop[12] results.
Does anyone hack benchmark program?
2. These benchmarks are only "micro" benchmarks. The purpose of
these benchmarks are only to measure VM basic performance.
Especially benchmarks focuses features which YARV achives
performance improvement on.
So practical application (such as using String manipulation) will
not be fast.
So ... in a nutshell ... 1.9.0 is five times as fast as 1.8.6!
I'd be careful about making that assertion. The benchmarks in Ruby 1.9
are specific areas that Koichi has optimized to make 1.9 run them
faster. There are other areas, especially as regards core classes and
evaluated code, where 1.9 is no faster or even slower.
Not trying to rain on the parade, but even Koichi has recommended not
taking the 1.9 benchmark numbers as indicative of overall performance.
- Charlie
Well ... I'll agree with that after looking at the codes. They're
definitely micro-benchmarks and probably won't stress the cache/RAM
interface on modern chips. I'm hoping to get a Rails benchmark running
in the next few days -- it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don't suppose you have a
Rails benchmark laying around in your repositories.
Meanwhile, I ran them on the Athlon64 X2 and got slightly better results
-- which scale by cycle time as well. Unfortunately CodeAnalyst won't
tell me much on the T-Bird -- the chip is too ancient to interact
correctly with the profiler. But I should be able to get some pretty
good profiles on the Athlon64 X2 with it.
For what its worth, the RubyGems test suite runs about 20-25% faster on a recent 1.9.0, as reported by test/unit.
···
On Sep 24, 2007, at 01:00 , SASADA Koichi wrote:
M. Edward (Ed) Borasky wrote:
I can do fancier stuff in R, but I probably won't -- 5x is easy to remember.
2. These benchmarks are only "micro" benchmarks. The purpose of
these benchmarks are only to measure VM basic performance.
Especially benchmarks focuses features which YARV achives
performance improvement on.
So practical application (such as using String manipulation) will
not be fast.
Well ... I'll agree with that after looking at the codes. They're
definitely micro-benchmarks and probably won't stress the cache/RAM
interface on modern chips. I'm hoping to get a Rails benchmark running
in the next few days -- it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don't suppose you have a
Rails benchmark laying around in your repositories.
Alexey Verhovsky from ThoughtWorks does have a "Pet Store" app in Rails that he's been using to benchmark. I don't know how complete it is or how similar to a "real world" application, but it might be a good place to start (and I suspect he'd be interested in having others try it and improve it).
Probably easiest to search JRuby mailing list archive for it...I don't have a link handy.
Well ... I'll agree with that after looking at the codes. They're
definitely micro-benchmarks and probably won't stress the cache/RAM
interface on modern chips. I'm hoping to get a Rails benchmark running
in the next few days -- it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don't suppose you have a
Rails benchmark laying around in your repositories.
Alexey Verhovsky from ThoughtWorks does have a "Pet Store" app in Rails that he's been using to benchmark. I don't know how complete it is or how similar to a "real world" application, but it might be a good place to start (and I suspect he'd be interested in having others try it and improve it).
Probably easiest to search JRuby mailing list archive for it...I don't have a link handy.
- Charlie
Take a look at the project tw-commons in RubyForge.
Well ... I'll agree with that after looking at the codes. They're
definitely micro-benchmarks and probably won't stress the cache/RAM
interface on modern chips. I'm hoping to get a Rails benchmark running
in the next few days -- it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don't suppose you have a
Rails benchmark laying around in your repositories.
Alexey Verhovsky from ThoughtWorks does have a "Pet Store" app in Rails
that he's been using to benchmark. I don't know how complete it is or
how similar to a "real world" application, but it might be a good place
to start (and I suspect he'd be interested in having others try it and
improve it).
Probably easiest to search JRuby mailing list archive for it...I don't
have a link handy.
- Charlie
Yeah ... he told me about that on the Rails list. I downloaded it but
there aren't any installation instructions. It came up but none of the
links worked, presumably because I didn't do some necessary step.
On 9/24/07, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:
Yeah ... he told me about that on the Rails list. I downloaded it but
there aren't any installation instructions. It came up but none of the
links worked, presumably because I didn't do some necessary step.
On 9/24/07, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:
Yeah ... he told me about that on the Rails list. I downloaded it but
there aren't any installation instructions. It came up but none of the
links worked, presumably because I didn't do some necessary step.
