Ruby vs JRuby Performance

ruby-talk
1

I knew that there was a penalty to be paid when running JRuby, but I did not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard" problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

BTW, there is no GUI in my program. I hope to learn the GUI part with JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

2

Victor Reyes schrieb:
[...]

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does not mean its overall performance is bad. I suggest you modify the program so it runs for example 10 times and then run again. I am sure the JRuby program will be very much below 10s

bye Jochen

3

Victor Reyes wrote:

I knew that there was a penalty to be paid when running JRuby, but I did not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard" problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

BTW, there is no GUI in my program. I hope to learn the GUI part with JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

Send the source to Charlie Nutter ... he is always looking for jRuby performance test cases.

4

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Try to run it with jruby -J-server ...

5

Victor Reyes wrote:

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard"
problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:
*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Am I in the ball-park?

According to the metrics posted here, JRuby runs about twice as fast,
takes longer to load and uses a lot more memory. Here is a little
something on the metrics:

···

--
Posted via http://www.ruby-forum.com/\.

6

*Ruby vs JRuby Performance <Ruby vs JRuby Performance - Ruby - Ruby-Forum*
Posted by Victor Reyes (Guest)
on 09.06.2008 15:41
[image: (Received via mailing list)]

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard"

problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

BTW, there is no GUI in my program. I hope to learn the GUI part with

JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE,
Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

Reply with quote<http://www.ruby-forum.com/topic/155783?reply_to=686221#postform&gt;
  *Re: Ruby vs JRuby Performance<Ruby vs JRuby Performance - Ruby - Ruby-Forum;

···

*
Posted by Jochen Theodorou (Guest)
on 09.06.2008 16:15
[image: (Received via mailing list)]

Victor Reyes schrieb:
[...]

jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does

not mean its overall performance is bad. I suggest you modify the
program so it runs for example 10 times and then run again. I am sure
the JRuby program will be very much below 10s

bye Jochen

Reply with quote<http://www.ruby-forum.com/topic/155783?reply_to=686235#postform&gt;
  *Re: Ruby vs JRuby Performance<Ruby vs JRuby Performance - Ruby - Ruby-Forum;
*
Posted by M. Edward (Ed) Borasky (Guest)
on 09.06.2008 16:28 [image: (Received via mailing list)] Victor Reyes wrote:

ruby -v
Elapsed Time: 1.058 Secs*

Send the source to Charlie Nutter ... he is always looking for jRuby
performance test cases.

Reply with quote<http://www.ruby-forum.com/topic/155783?reply_to=686241#postform&gt;
  *Re: Ruby vs JRuby Performance<Ruby vs JRuby Performance - Ruby - Ruby-Forum;
*
Posted by ThoML (Guest)
on 09.06.2008 16:31

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Try to run it with jruby -J-server ...

Reply with quote<http://www.ruby-forum.com/topic/155783?reply_to=686245#postform&gt;
*Re: Ruby vs JRuby Performance<Ruby vs JRuby Performance - Ruby - Ruby-Forum;
*
Posted by Lloyd Linklater (lloyd <http://www.ruby-forum.com/user/show/6864&gt;\)

on 09.06.2008 16:44 Victor Reyes wrote:

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard"
problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:
*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Am I in the ball-park?

According to the metrics posted here, JRuby runs about twice as fast,
takes longer to load and uses a lot more memory. Here is a little
something on the metrics:

======================================================================================

Team,

First, thank you for all your comments.

As suggested by JoChen, I executed the program under JRuby 14 times.

To be fair to Ruby, I also executed it 14 times under Ruby.
You can see the results below.

As suggested by Ed Borasky, I will send the code to Charlie Nutter, if
I can get his email. Otherwise I can post it here. It is not great

code at all, but it is doing the work for now.

I executed the program using *jruby -J-server* as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.
I also can't believe that jruby has been clocked at twice the speed of
ruby since after all, jruby has more overhead than ruby. Then again,

I am not expert on the subject (or any subject).

*JRuby executions:
*Elapsed Time: 0.888
Elapsed Time: 0.881
Elapsed Time: 0.977
Elapsed Time: 0.888
Elapsed Time: 0.872

Elapsed Time: 0.884
Elapsed Time: 0.869
Elapsed Time: 0.892
Elapsed Time: 0.876
Elapsed Time: 0.872
Elapsed Time: 0.882
Elapsed Time: 0.882
Elapsed Time: 0.867
Elapsed Time: 0.881
Elapsed Time: 0.906

Elapsed Time: 0.881

*Ruby executions:*
Elapsed Time: 0.422
Elapsed Time: 0.391
Elapsed Time: 0.407
Elapsed Time: 0.406
Elapsed Time: 0.407

Elapsed Time: 0.422
Elapsed Time: 0.406
Elapsed Time: 0.438
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.39
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.422

Elapsed Time: 0.39

*jruby -J-server sudoku01final.rb*
Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

Thank you

Victor

7

Victor Reyes wrote:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Far too short.

~/amsterdam ➔ time ruby -e "puts 1"
1

real 0m0.006s
user 0m0.003s
sys 0m0.004s

~/amsterdam ➔ time jruby -e "puts 1"
1

real 0m0.594s
user 0m0.460s
sys 0m0.101s

Is JRuby 100 times slower than Ruby? No. It is too short a test. Run something longer and you'll get more realistic results.

- Charlie

8

Victor Reyes wrote:

BTW, there is no GUI in my program. I hope to learn the GUI part with JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, Etc.
I guess there is trade-off which one must be willing to accept.

For what it's worth, 1.1.3 should include an update to the interpreter that will help short-running apps a bit. But even then runs shorter than 5-10s aren't going to really hit their stride as far as JRuby goes.

- Charlie

9

Victor Reyes wrote:

I knew that there was a penalty to be paid when running JRuby, but I did not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard" problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

Ugh, and I keep forgetting...yes, we'd love to see the code.

- Charlie

10

I executed the program using *jruby -J-server* as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.

I didn't realize how small the numbers already are. Have you checked
the result of

time ruby -v
time jruby -v

just to get a feeling for how much time is spent in startup alone?

11

Victor Reyes schrieb:
[...]

As suggested by JoChen, I executed the program under JRuby 14 times.

To be fair to Ruby, I also executed it 14 times under Ruby.
You can see the results below.

As suggested by Ed Borasky, I will send the code to Charlie Nutter, if
I can get his email. Otherwise I can post it here. It is not great

code at all, but it is doing the work for now.

I executed the program using *jruby -J-server* as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.
I also can't believe that jruby has been clocked at twice the speed of
ruby since after all, jruby has more overhead than ruby. Then again,

I am not expert on the subject (or any subject).

*JRuby executions:
*Elapsed Time: 0.888
Elapsed Time: 0.881

[...]

Elapsed Time: 0.881

*Ruby executions:*
Elapsed Time: 0.422
Elapsed Time: 0.391

[...]

Elapsed Time: 0.39

*jruby -J-server sudoku01final.rb*
Elapsed Time: 2.474

[...]

Elapsed Time: 2.414

well, looking at these numbers I would suggest you subscribe to the jruby mailing list and ask why jruby is 100% slower than ruby :wink: The server mode really looks awful... They need to know about this!

bye Jochen

12

Jochen Theodorou wrote:

well, looking at these numbers I would suggest you subscribe to the jruby mailing list and ask why jruby is 100% slower than ruby :wink: The server mode really looks awful... They need to know about this!

You are here trolling on the Ruby lists Jochen? Don't you have some Groovy bugs to be fixing? :slight_smile:

The server VM starts up a lot slower on most systems, which is the reason for this test to be invalid. And in general, any benchmark under 5-10s is going to be a pretty poor test of JRuby performance, since the code might not even get compiled or optimized on such a short run.

- Charlie

13

Charles Oliver Nutter schrieb:

Jochen Theodorou wrote:

well, looking at these numbers I would suggest you subscribe to the jruby mailing list and ask why jruby is 100% slower than ruby :wink: The server mode really looks awful... They need to know about this!

You are here trolling on the Ruby lists Jochen? Don't you have some Groovy bugs to be fixing? :slight_smile:

ah, I didn't try to troll, I put so many :wink: in.

The server VM starts up a lot slower on most systems, which is the reason for this test to be invalid. And in general, any benchmark under 5-10s is going to be a pretty poor test of JRuby performance, since the code might not even get compiled or optimized on such a short run.

well, ok, but it was:

*jruby -J-server sudoku01final.rb*
Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

that's >12s... unless he did start over again and again, but I assumed that is not the case because of:

*JRuby executions:* Elapsed Time: 0.888 [...]

vs.

*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

but maybe I was wrong.

bye Jochen

14

Team,

I'll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* multiple
times. Then I did the same for ruby: *ruby sudoku01final.rb.*
Although the code is not great code, that does not really matter since the
same "bad" or "good" code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

Thank you

Victor

···

On Tue, Jun 10, 2008 at 8:19 AM, Jochen Theodorou <blackdrag@uni.de> wrote:

Charles Oliver Nutter schrieb:

Jochen Theodorou wrote:

well, looking at these numbers I would suggest you subscribe to the jruby
mailing list and ask why jruby is 100% slower than ruby :wink: The server mode
really looks awful... They need to know about this!

You are here trolling on the Ruby lists Jochen? Don't you have some Groovy
bugs to be fixing? :slight_smile:

ah, I didn't try to troll, I put so many :wink: in.

The server VM starts up a lot slower on most systems, which is the reason

for this test to be invalid. And in general, any benchmark under 5-10s is
going to be a pretty poor test of JRuby performance, since the code might
not even get compiled or optimized on such a short run.

well, ok, but it was:

*jruby -J-server sudoku01final.rb*

Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

that's >12s... unless he did start over again and again, but I assumed
that is not the case because of:

*JRuby executions:* Elapsed Time: 0.888 [...]

vs.

*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

but maybe I was wrong.

bye Jochen

15

Just put a loop in your Ruby program:

12.times do

...main code

end

by restarting the JVM each time, you are not seeing any of JRuby's benefits.

Les

···

On Tue, Jun 10, 2008 at 3:15 PM, Victor Reyes <victor.reyes@gmail.com> wrote:

On Tue, Jun 10, 2008 at 8:19 AM, Jochen Theodorou <blackdrag@uni.de> wrote:

Charles Oliver Nutter schrieb:

Jochen Theodorou wrote:

well, looking at these numbers I would suggest you subscribe to the jruby
mailing list and ask why jruby is 100% slower than ruby :wink: The server mode
really looks awful... They need to know about this!

You are here trolling on the Ruby lists Jochen? Don't you have some Groovy
bugs to be fixing? :slight_smile:

ah, I didn't try to troll, I put so many :wink: in.

The server VM starts up a lot slower on most systems, which is the reason

for this test to be invalid. And in general, any benchmark under 5-10s is
going to be a pretty poor test of JRuby performance, since the code might
not even get compiled or optimized on such a short run.

well, ok, but it was:

*jruby -J-server sudoku01final.rb*

Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

that's >12s... unless he did start over again and again, but I assumed
that is not the case because of:

*JRuby executions:* Elapsed Time: 0.888 [...]

vs.

*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

but maybe I was wrong.

bye Jochen

Team,

I'll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* multiple
times. Then I did the same for ruby: *ruby sudoku01final.rb.*
Although the code is not great code, that does not really matter since the
same "bad" or "good" code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

16

Victor Reyes schrieb:
[...]

I'll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* multiple
times.

ok, that was not what I intended. I did mean, what others also said, a loop inside the program.

bye Jochen

17

Victor Reyes wrote:

I'll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* multiple
times. Then I did the same for ruby: *ruby sudoku01final.rb.*
Although the code is not great code, that does not really matter since the
same "bad" or "good" code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

I'd love to give it a try. If it does turn out to be slower, it will be fun to investigate and figure out why.

- Charlie

18

The JVM is built for long running processes, optimizing things over time
(jruby takes advantage of this), MRI is going to remain the same speed over
time. (for the most part, see below)

here is a quick example... first the somewhat silly example code:

···

#######################################
  require 'benchmark'
  num = ARGV[0] || 10

  puts "Running #{num} times"

  puts Benchmark.measure {
    num.to_i.times do
      234 + 89239423 + 23482903 + 34892389423 + 234823
    end
  }.total
#######################################
now run it 10 times each: ( ruby/jruby speed.rb 10)
  ruby: 0.0
  jruby: 0.0519

10,000 times:
  ruby:0.03
  jruby:0.105

100,000 times:
  ruby: 0.26
  jruby: 0.217

1,000,000 times:
  ruby: 2.61
  jruby: 1.303

now with some additional optimization for the jvm: jruby -J-server speed.rb
1000000
  jruby: 0.845

This is obviously a silly little test, but can possibly shed light on the
start-up cost diffs between JRuby and MRI... and where the benefit lies. So
who is faster? MRI? (yes) JRuby? (yes) just depends on what problem you
are trying to solve.

On Tue, Jun 10, 2008 at 9:24 AM, Leslie Viljoen <leslieviljoen@gmail.com> wrote:

Just put a loop in your Ruby program:

12.times do

...main code

end

by restarting the JVM each time, you are not seeing any of JRuby's
benefits.

Les

--
Aloha!

Mike McKinney
(http://blog.huikau.com)

19

Charlie,

The code is attached to this thread.
However, I will send it to your email.
Also, remember that this is code written by a Rubist wannabe and not
necessary by one who knows the ins and outs of the language.
I'll be interested in knowing what you find!

Thank you

Victor

···

On Fri, Jun 20, 2008 at 8:52 AM, Charles Oliver Nutter < charles.nutter@sun.com> wrote:

Victor Reyes wrote:

I'll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* multiple
times. Then I did the same for ruby: *ruby sudoku01final.rb.*
Although the code is not great code, that does not really matter since the
same "bad" or "good" code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

I'd love to give it a try. If it does turn out to be slower, it will be fun
to investigate and figure out why.

- Charlie

20

Victor Reyes wrote:

The code is attached to this thread.
However, I will send it to your email.
Also, remember that this is code written by a Rubist wannabe and not
necessary by one who knows the ins and outs of the language.
I'll be interested in knowing what you find!

Ok, I managed to get it to run a few times in a cycle. First here's MRI's time for me:

   Elapsed time: around 0.191s

Now the time running "jruby sudoku01final.rb"

(all using JRuby trunk, "soylatte" Java 6 on OS X)

   Elapsed time: anywhere from 0.491s to 0.525s

If I specify --server, it degrades on this short run:

   Elapsed time: 1.6s to 1.65s

However, trying both runs ten times produces a very different result:

   Ten times without --server:
   Elapsed time (final result): 0.105s

   Ten times with --server:
   Elapsed time (final result): 0.212s

So there's some interesting slowdown for this short benchmark running under server. Let's try 100 iterations:

   100 times without --server:
   Elapsed time (final result): 0.103

   100 times with --server:
   Elapsed time (final result): 0.065s

It seems like the run is just too short for the server VM to really do much optimization, which is why we have JRuby run with the client VM by default.

I've attached the modified source.

- Charlie

sudoku01Final.rb (21.3 KB)