JRuby performance questions answered

ruby-talk
1

Many folks still seem to be in the dark about JRuby performance relative to Ruby 1.8.x. I figured a short post with some basic info could start a discussion for anyone wanting to know more.

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and for most execution benchmarks it is the fastest 1.8-compatible Ruby implementation available. This is based on the YARV benchmarks, the Alioth benchmarks, a number of benchmarks we created for JRuby, and a few other external benchmarks we run from time to time. Many (most?) of these are microbenchmarks, but a number are nontrivial code.

For example, M. Ed. Borasky's MatrixBenchmark:

JRuby:
Hilbert matrix of dimension 64 times its inverse = identity? true
  18.784000 0.000000 18.784000 ( 18.784000)

Ruby 1.8.6:
Hilbert matrix of dimension 64 times its inverse = identity? true
  32.560000 0.110000 32.670000 ( 32.776915)

Or the YARV "pentomino" benchmark:

~/NetBeansProjects/jruby $ time jruby -J-server test/bench/yarv/bm_app_pentomino.rb

real 1m31.770s
user 1m32.016s
sys 0m1.278s
~/NetBeansProjects/jruby $ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m48.100s
user 1m47.489s
sys 0m0.287s

Ola Bini of the JRuby team recently ran all the YARV benchmarks and reported the results here:

And I periodically post about performance progress on my blog here:

I expect we'll see Alioth updated soon after the 1.1 or 1.1 beta 1 releases.

What about Rails?

One of the biggest uses for JRuby recently has been Rails. Rails performance is a more complicated thing to measure; there's a lot of non-execution bottlenecks that get in the way. But both Ola and Nick Sieger (also of JRuby team) have found that JRuby on Rails performance is either very near or exceeding Ruby 1.8.x on Rails:


(toward the bottom of the post are updated Rails numbers)

http://blog.nicksieger.com/articles/2007/10/25/jruby-on-rails-fast-enough

Rails is probably the ultimate real-world benchmark, and we're excited to see that JRuby's starting to pass Ruby 1.8.x performance here as well.

I won't drag this on any longer, but I'd encourage you to try out your own Ruby code and let us know how it performs. We've spent a lot of time on performance issues for the 1.1 release, and have many future plans to continue increasing performance. We want your input.

Many people believed we'd never be faster than the C implementation, and many still think we're slower. Now that I've set that record straight, any questions?

- Charlie

2

Will JRuby support continuations (via callcc)?

I've been reading about Actors in Scala recently and the technical
documentation mentioned problems with trying to implement
continuations on the JVM due to lack of stack manipulation.

3

Quoting Charles Oliver Nutter <charles.nutter@sun.com>:

Many people believed we'd never be faster than the C implementation,
and many still think we're slower. Now that I've set that record
straight, any questions?

1. How long will it be before Alioth has some *reasonable* numbers for jRuby? As of yesterday, they still have you significantly slower than MRI. So I need to take jRuby out of my slides for RubyConf :slight_smile: ... I

2. I haven't actually been benchmarking jRuby recently ... too busy profiling MRI. :slight_smile:

3. I'm using the "Pet Store" benchmark for Rails. It's as close to a "real Rails benchmark" as anything I can find.

4. How's your performance on a SPARC relative to MRI?

···

- Charlie

4

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
for most execution benchmarks it is the fastest 1.8-compatible Ruby
implementation available.

Congrats on such an important benchmark! What made the difference 1.0
to 1.1?
-Roger

···

--
Posted via http://www.ruby-forum.com/\.

5

Charles Oliver Nutter wrote:

Many people believed we'd never be faster than the C implementation, and
many still think we're slower.

Isn't that still a correct impression and statement? Currently the C
implementation to benchmark against is 1.9, which is soon to become
the official release (early next year, if all goes to plan). 1.8 is
mostly 3 years-old code now or so.
Or is JRuby now faster than 1.9?

Now that I've set that record straight, any questions?

I do have one, yes. How much work does the JRuby team place in
actually improving Sun's JVM?

Recently there was a question of actually turning off ObjectSpace for
JRuby as a default, instead of actually taking the JVM and enhancing
it to support some sort of ObjectSpace, which seemed more like the
right answer to me.

6

Paul Stickney wrote:

Will JRuby support continuations (via callcc)?

I've been reading about Actors in Scala recently and the technical
documentation mentioned problems with trying to implement
continuations on the JVM due to lack of stack manipulation.

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support continuations in the near future. However, there is research happening now to support continuations on the JVM...if that takes, we'll piggy back off that support.

- Charlie

7

znmeb@cesmail.net wrote:

Quoting Charles Oliver Nutter <charles.nutter@sun.com>:

Many people believed we'd never be faster than the C implementation,
and many still think we're slower. Now that I've set that record
straight, any questions?

1. How long will it be before Alioth has some *reasonable* numbers for jRuby? As of yesterday, they still have you significantly slower than MRI. So I need to take jRuby out of my slides for RubyConf :slight_smile: ... I

The current published Alioth numbers are based on JRuby 1.0(ish), which was generally 2-3x slower than MRI. I'm hoping the numbers will be updated soon after the 1.1 releases...but it probably won't happen until 1.1 final comes out in December. If someone else wants to re-run them for us, it would make us very happy :slight_smile:

2. I haven't actually been benchmarking jRuby recently ... too busy profiling MRI. :slight_smile:

Well, we know there's still a lot we can do for JRuby performance...including not only the 1.9 benchmarks, but a number of additional ideas we have that won't affect 1.8 compatibility.

3. I'm using the "Pet Store" benchmark for Rails. It's as close to a "real Rails benchmark" as anything I can find.

Yeah, that's what Ola's been using, and we're *real close* to matching MRI on that one right now. Still hunting for bottlenecks.

4. How's your performance on a SPARC relative to MRI?

Haven't given it a try...I don't have access to a good sparc box at the moment (and yes, I know...I work at Sun, so I can have a sparc by snapping my fingers...it's a matter of having time to monkey with it).

- Charlie

8

Roger Pack wrote:

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
for most execution benchmarks it is the fastest 1.8-compatible Ruby
implementation available.

Congrats on such an important benchmark! What made the difference 1.0 to 1.1?

Mostly the compiler, but there's been a number of optimizations across the whole runtime and most of the core classes.

- Charlie

9

gga wrote:

Charles Oliver Nutter wrote:

Many people believed we'd never be faster than the C implementation, and
many still think we're slower.

Isn't that still a correct impression and statement? Currently the C
implementation to benchmark against is 1.9, which is soon to become
the official release (early next year, if all goes to plan). 1.8 is
mostly 3 years-old code now or so.
Or is JRuby now faster than 1.9?

Ruby 1.9 is not likely to be the standard, widely-used version of Ruby for some time. Matz estimated it probably wouldn't be the case until late 2008. He also mentioned at the conference that while 1.9.1 might be feature-stable, it's still a bit experimental and will probably not be a production-ready VM for a while.

In my opinion, no, it's not really a direct comparison to put JRuby up against Ruby 1.9 because 1.9 contains several compatibility-breaking changes. Many of those changes enable faster performance at the cost of backward compatibility. JRuby is an implementation of Ruby 1.8, and under that definition, it's appropriate to compare it to other implementations of Ruby 1.8. Some example of compatibility-breaking changes in Ruby 1.9:

- case/when with literals will be constant time, but not invoke ===
- Fixnum math will be performed as direct operations, rather than dispatch through Fixnum#+, Fixnum#-, and so on.
- block arguments can only be local variables

And so on. We plan to make all the same optimizations, but also make them individually configurable, so that if you want to have fast integer math but everything else 1.8 semantics, you'll be able to. Or if you want to turn on 1.9 features as a whole, you can.

Since you ask, yes, there's a few benchmarks where we're already faster than 1.9:

JRuby:
1m loops yielding three fixnums 10 times to block splatting and accessing them
   3.225000 0.000000 3.225000 ( 3.225000)
   3.189000 0.000000 3.189000 ( 3.189000)
   3.170000 0.000000 3.170000 ( 3.170000)
   3.152000 0.000000 3.152000 ( 3.151000)
   3.322000 0.000000 3.322000 ( 3.322000)

1.9:
1m loops yielding three fixnums 10 times to block splatting and accessing them
   4.940000 0.010000 4.950000 ( 4.973445)
   4.950000 0.020000 4.970000 ( 4.971358)
   4.950000 0.010000 4.960000 ( 4.984771)
   4.950000 0.010000 4.960000 ( 4.980683)
   4.950000 0.010000 4.960000 ( 4.973008)

~/NetBeansProjects/jruby $ jruby -J-server test/bench/bench_fib_iterative.rb
  11.610000 0.000000 11.610000 ( 11.611000)
  11.481000 0.000000 11.481000 ( 11.481000)
~/NetBeansProjects/jruby $ ../ruby1.9/ruby -I ../ruby1.9/lib test/bench/bench_fib_iterative.rb
  13.110000 4.700000 17.810000 ( 17.928527)
  12.930000 4.770000 17.700000 ( 17.818631)

1.9's eval performance is also considerably degraded due to the cost of compilation...in some cases even slower than Ruby 1.8.

Now that I've set that record straight, any questions?

I do have one, yes. How much work does the JRuby team place in
actually improving Sun's JVM?

Recently there was a question of actually turning off ObjectSpace for
JRuby as a default, instead of actually taking the JVM and enhancing
it to support some sort of ObjectSpace, which seemed more like the
right answer to me.

The problems with ObjectSpace are not just problems you can flip a switch and make go away; they're systemic issues that will affect any modern GC. Look at it this way:

In Ruby, where Objects do not move around in memory, it's easy to just walk through the entire set of objects in memory and return them in turn. There's no parallel threads to create more garbage, and there's no concurrent garbage collector moving objects around and compacting the heap.

On the JVM, however, it's an entirely different situation. Not only are threads actually running in parallel, creating their own garbage...the GC itself runs in parallel on many versions of Java; so you'd not only have to stop all other threads from running, you'd have to stop the garbage collector itself.

The other answer here is that...you already can do these things; they're used for profiling and debugging modes of the JVM. And oddly enough, the most common applications of ObjectSpace.each_object are for runtime profiling and debugging. The difference in the JVM (and in JRuby 1.1) is that we're not going to saddle users with the performance hit of what's typically a profiling/debugging feature that they'll never use on a production server.

(and of course we're very keen on improving the JVM...but there's only so many hours in a day)

- Charlie

10

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

···

On Oct 31, 11:24 pm, Charles Oliver Nutter <charles.nut...@sun.com> wrote:

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support

11

Quoting Charles Oliver Nutter <charles.nutter@sun.com>:

···

znmeb@cesmail.net wrote:

Quoting Charles Oliver Nutter <charles.nutter@sun.com>:

Many people believed we'd never be faster than the C implementation,
and many still think we're slower. Now that I've set that record
straight, any questions?

1. How long will it be before Alioth has some *reasonable* numbers for jRuby? As of yesterday, they still have you significantly slower than MRI. So I need to take jRuby out of my slides for RubyConf :slight_smile: ... I

The current published Alioth numbers are based on JRuby 1.0(ish), which
was generally 2-3x slower than MRI. I'm hoping the numbers will be
updated soon after the 1.1 releases...but it probably won't happen
until 1.1 final comes out in December. If someone else wants to re-run
them for us, it would make us very happy :slight_smile:

I can't help you with that, but if you'll give me the numbers you do have (time in seconds for each benchmark on the latest jRuby, MRI and KRI on a common platform) I'll run the boxplots off and post them to the list.

12

William James wrote:

···

On Oct 31, 11:24 pm, Charles Oliver Nutter <charles.nut...@sun.com> > wrote:

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

None of snow or rain or heat or gloom of night stays these couriers from the swift completion of their appointed rounds.

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

13

William James wrote:

···

On Oct 31, 11:24 pm, Charles Oliver Nutter <charles.nut...@sun.com> > wrote:

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

Perhaps

JRuby, XRuby, IronRuby, Ruby.NET; none of them will support

Too poetic?

- Charlie

14

The original polysyndeton conveyed the intended meaning effectively IMO.
How thin is the line between a deliberate figure of speech and a grammatical
accident!

This doesn't sound too bad to me, but I'm no native speaker:
"Neither JRuby nor XRuby, IronRuby or Ruby.NET will ..."

"Or" must bind more tightly than "nor" there in order for the corresponding
logical proposition to be correct:

  AND(NOT(JRuby), NOT(OR(XRuby, IronRuby, Ruby.NET)))

"None of JRuby, XRuby, IronRuby or Ruby.NET will ..." and
"As for JRuby, XRuby, IronRuby and Ruby.NET, none [of them] will ..."
seem OK too.

···

On Fri, Nov 02, 2007 at 09:35:44AM +0900, Charles Oliver Nutter wrote:

William James wrote:
>On Oct 31, 11:24 pm, Charles Oliver Nutter <charles.nut...@sun.com> > >wrote:
>
>>Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support
>
>"Neither ... nor ..." is for two alternatives. Perhaps this will
>suffice.
>
>None of JRuby or XRuby or IronRuby or Ruby.NET will support

Perhaps

JRuby, XRuby, IronRuby, Ruby.NET; none of them will support

Too poetic?

--
Mauricio Fernandez - http://eigenclass.org

15

Mauricio Fernandez wrote:

William James wrote:

Neither JRuby nor XRuby nor IronRuby nor Ruby.NET will support

"Neither ... nor ..." is for two alternatives. Perhaps this will
suffice.

None of JRuby or XRuby or IronRuby or Ruby.NET will support

Perhaps

JRuby, XRuby, IronRuby, Ruby.NET; none of them will support

Too poetic?

The original polysyndeton conveyed the intended meaning effectively IMO. How thin is the line between a deliberate figure of speech and a grammatical
accident!

I agree, and thanks for teaching me a new word.

This doesn't sound too bad to me, but I'm no native speaker:
"Neither JRuby nor XRuby, IronRuby or Ruby.NET will ..."

That could sound like the "Neither ... nor ..." clause modifies IronRuby. As in:

Neither fish nor fowl, greens or libertarians will be the first choice of voters who are repelled by two-party politics.

It might not even be grammatical, but it is clear that we are not talking about electing ducks and sturgeons. I'd rewrite it anyway.

"Or" must bind more tightly than "nor" there in order for the corresponding
logical proposition to be correct:

I don't think it works that way, at least as I hear it.

···

On Fri, Nov 02, 2007 at 09:35:44AM +0900, Charles Oliver Nutter wrote:

On Oct 31, 11:24 pm, Charles Oliver Nutter <charles.nut...@sun.com> >>> wrote:

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

16

Mauricio Fernandez wrote:
>This doesn't sound too bad to me, but I'm no native speaker:
>"Neither JRuby nor XRuby, IronRuby or Ruby.NET will ..."

That could sound like the "Neither ... nor ..." clause modifies
IronRuby. As in:

Neither fish nor fowl, greens or libertarians will be the first choice
of voters who are repelled by two-party politics.

It might not even be grammatical, but it is clear that we are not
talking about electing ducks and sturgeons. I'd rewrite it anyway.

Yet another effect is achieved by reordering the phrase this way:
Neither fish nor greens, fowl or libertarians
This being a play on the standard idiom [1], the two parts are connected,
yielding a new meaning.

Is any sufficiently complex figure of speech undistinguishable from nonsense? :slight_smile:

>"Or" must bind more tightly than "nor" there in order for the corresponding
>logical proposition to be correct:

I don't think it works that way, at least as I hear it.

I had to repeat it a few times before I convinced myself that this is how
things should work, but the interpretation as a juxtaposition is very
attractive. Maybe I'm drawn to it because my mother tongue is closer to Latin?

A skilled writer could use the ambiguity for very effective double-entendres.

[1] I had to look it up. In Spanish, French and German the idiom is "neither
meat nor fish" (actually, "neither fish nor meat" in German; I wonder why).

···

On Sat, Nov 03, 2007 at 04:40:41AM +0900, Joel VanderWerf wrote:

--
Mauricio Fernandez - http://eigenclass.org

17

The switching of nouns may have more to do with culture than language roots.

I couldn't resist an attempt at lolcat...

Ther be JRuby, XRuby, IronRuby, Ruby.NET, and no s'port continue stufs now.

Latin rooted languages allow a lot of latitude, certainly, and I think
if they lost that latitude, they would make a lot of poets bored, as
you suggested.

I could have used "but" instead of "and" in that last sentence and
still convey the same meaning, but it might have changed the intent if
your vernacular idioms differ from mine.

Todd

···

On 11/3/07, Mauricio Fernandez <mfp@acm.org> wrote:

I had to repeat it a few times before I convinced myself that this is how
things should work, but the interpretation as a juxtaposition is very
attractive. Maybe I'm drawn to it because my mother tongue is closer to Latin?

A skilled writer could use the ambiguity for very effective double-entendres.

[1] I had to look it up. In Spanish, French and German the idiom is "neither
meat nor fish" (actually, "neither fish nor meat" in German; I wonder why).