Does anyone have benchmark programs for YARV?

ruby-talk
1

Hi,

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

Current YARV has some limitations (especially, can't require
any files). But I can fix YARV or your program to work and
measure execution time.

If you have a ruby program which is slow for you, feel free to
send me that :slight_smile: Perhaps, YARV will be optimized for your
application.

Thanks,

···

--
// SASADA Koichi at atdot dot net
//

2

SASADA Koichi wrote:

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

Maybe you can have a look at the Ruby code for the Debian language
shootout (available at http://shootout.alioth.debian.org/\) -- that way
you could also easily compare performance to that of other programming
languages which might be a good motivation.

3

SASADA Koichi wrote:

Hi,

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

Current YARV has some limitations (especially, can't require
any files). But I can fix YARV or your program to work and
measure execution time.

If you have a ruby program which is slow for you, feel free to
send me that :slight_smile: Perhaps, YARV will be optimized for your
application.

Thanks,

I don't know if this is what you are looking for or not, but Needle
(http://rubyforge.org/projects/needle\) has a 'benchmarks' subdirectory
with a handful of different benchmarks.

If you manage to get them to run under YARV, I'd love to see the results. :slight_smile:

- Jamis

···

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

4

SASADA Koichi ha scritto:

Hi,

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

I don't have a benchmark.. but I think the OO Richards Benchmark as
found here:
http://pws.prserv.net/dlissett//ben/bench1-ruby-28.htm
may be interesting.

Current YARV has some limitations (especially, can't require
any files). But I can fix YARV or your program to work and
measure execution time.

If you have a ruby program which is slow for you, feel free to
send me that :slight_smile: Perhaps, YARV will be optimized for your
application.

Not that it is a valuable application... but I have this dumb example of
using backtracking to solve the map coloring problem as found here:
http://www.ccs.neu.edu/home/dorai/t-y-scheme/t-y-scheme-Z-H-16.html#node_sec_14.4.2

It stresses callcc a lot and takes ~15 seconds to terminate on my old
athlon 2000 box.
If you want take a look at it at:
http://rafb.net/paste/results/x4gckY22.html

(sorry for italian naming I guess it is still understandable :slight_smile:

5

I have made a ruby compatible regexp engine.. written entirely in Ruby.
It builds some rather big AST's and does some heavy computations.

On my 700 MHz box it takes 37.8 seconds to run the default tests.
And it takes 19.6 seconds to run all the ~1650 rubicon regexps.
There is 580 tests and 677 assertions (and 0 failures, 0 errors).

http://raa.ruby-lang.org/project/regexp/

···

On Thu, 2 Dec 2004 20:33:59 +0900, SASADA Koichi <ko1@atdot.net> wrote:

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

--
Simon Strandgaard

6

Hi,

I already use that:)
http://www.atdot.net/svn/yarv/trunk/benchmark/

Thanks anyway.

Florian Gross wrote:

···

SASADA Koichi wrote:

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

Maybe you can have a look at the Ruby code for the Debian language
shootout (available at http://shootout.alioth.debian.org/\) -- that way
you could also easily compare performance to that of other programming
languages which might be a good motivation.

7

gabriele renzi ha scritto:

SASADA Koichi ha scritto:

Hi,

I'm developing YARV - Yet Another RubyVM. I'm looking for
benchmark programs to evaluate YARV's performance. So, Does
anyone have suitable benchmarks for this purpose?

I don't have a benchmark.. but I think the OO Richards Benchmark as
found here:
http://pws.prserv.net/dlissett//ben/bench1-ruby-28.htm
may be interesting.

Current YARV has some limitations (especially, can't require
any files). But I can fix YARV or your program to work and
measure execution time.

If you have a ruby program which is slow for you, feel free to
send me that :slight_smile: Perhaps, YARV will be optimized for your
application.

Not that it is a valuable application... but I have this dumb example of
using backtracking to solve the map coloring problem as found here:
http://www.ccs.neu.edu/home/dorai/t-y-scheme/t-y-scheme-Z-H-16.html#node_sec_14.4.2

It stresses callcc a lot and takes ~15 seconds to terminate on my old
athlon 2000 box.
If you want take a look at it at:
http://rafb.net/paste/results/x4gckY22.html

btw, today I should study butwhen one is forced to do stuff, he revertes
to something else. So I get yarv and tried it on my winxp setup

[gcc version 3.2.3 (mingw special 20030504-1), athlon 2000+ today
snapshot from 1.9 and yarv at revision 77].

Did make test-all and got it run fine (even if there were some warnings
in the compilation)
Then I stuffed my code as bm_4col.rb in the benchmark/ dir and launched
make bench. I got a crash on my code[1], but, well, I didn't expect it
to work :).
Results for other benchmarks are really impressive tough.
(i.e. the ackermann function timing is :
ruby 2.493000 0.010000 2.503000 ( 2.624000)
yarv 0.250000 0.000000 0.250000 ( 0.261000)
)

If someone's curious I've put them at:
ada2.unipv.it/~riffraff/bench.log.gz

[1]
../benchmark/bm_4col.rb:93: [BUG] BUG: unknown node: NODE_MASGN
ruby 1.9.0 (2004-12-03) [i386-mingw32]

8

Hi,

I don't know if this is what you are looking for or not, but Needle
(http://rubyforge.org/projects/needle\) has a 'benchmarks' subdirectory
with a handful of different benchmarks.

If you manage to get them to run under YARV, I'd love to see the results. :slight_smile:

- Jamis

I'll try it if 'require' is implemented.

Thanks,
SASADA Koichi

9

SASADA Koichi wrote:

I already use that:)
http://www.atdot.net/svn/yarv/trunk/benchmark/

Interesting, is the speedup in percent available somewhere?

10

gabriele renzi ha scritto:

further note:
make test-all from time to time gives:

c:/ruby/lib/ruby/1.9/test/unit/util/backtracefilter.rb:16: [BUG]
Segmentation fault
ruby 1.9.0 (2004-12-03) [i386-mingw32]

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.
make: *** [test-all] Error 3

11

Hi,

btw, today I should study butwhen one is forced to do stuff, he revertes
to something else. So I get yarv and tried it on my winxp setup

[gcc version 3.2.3 (mingw special 20030504-1), athlon 2000+ today
snapshot from 1.9 and yarv at revision 77].

Did make test-all and got it run fine (even if there were some warnings
in the compilation)
Then I stuffed my code as bm_4col.rb in the benchmark/ dir and launched
make bench. I got a crash on my code[1], but, well, I didn't expect it
to work :).
Results for other benchmarks are really impressive tough.
(i.e. the ackermann function timing is :
ruby 2.493000 0.010000 2.503000 ( 2.624000)
yarv 0.250000 0.000000 0.250000 ( 0.261000)
)

If someone's curious I've put them at:
ada2.unipv.it/~riffraff/bench.log.gz

I'll try to run your program.

Thanks,
SASADA Koichi

12

Hi,

Florian Gross wrote:

SASADA Koichi wrote:

I already use that:)
http://www.atdot.net/svn/yarv/trunk/benchmark/

Interesting, is the speedup in percent available somewhere?

today's result:
http://www.atdot.net/yarv/bench_20041206.txt

Regards,
SASADA Koichi

13

Hi,

gabriele renzi wrote:

gabriele renzi ha scritto:

further note:
make test-all from time to time gives:

c:/ruby/lib/ruby/1.9/test/unit/util/backtracefilter.rb:16: [BUG]
Segmentation fault
ruby 1.9.0 (2004-12-03) [i386-mingw32]

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.
make: *** [test-all] Error 3

Could you run following command on today's revision?

$ make test-all TESTOPT=-v

Thanks,
SASADA Koichi

14

SASADA Koichi wrote:

Hi,

Florian Gross wrote:

SASADA Koichi wrote:

I already use that:)
http://www.atdot.net/svn/yarv/trunk/benchmark/

Interesting, is the speedup in percent available somewhere?

today's result:
http://www.atdot.net/yarv/bench_20041206.txt

wow! very impressive results... looking forward to be able to use yarv...

Regards,

   Michael

15

SASADA Koichi ha scritto:

Hi,

gabriele renzi wrote:

gabriele renzi ha scritto:

further note:
make test-all from time to time gives:

c:/ruby/lib/ruby/1.9/test/unit/util/backtracefilter.rb:16: [BUG]
Segmentation fault
ruby 1.9.0 (2004-12-03) [i386-mingw32]

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.
make: *** [test-all] Error 3

Could you run following command on today's revision?

I tried it yesterday and it worked (I tried many times and always worked)

So revision 81 seem to work fine, revisions 82/83 cause the compiler to
fail:

C:\yarv\trunk>make
gcc -g -O2 -I. -Ic:/ruby/lib/ruby/1.9/i386-mingw32
-Ic:/ruby/lib/ruby/1.9/i386
-mingw32 -I. -DDISPATCH_THREADED_CODE -DDISPATCH_DIRECT_THREADED_CODE
-DOPT_BASI
C_OPERATIONS -DOPT_REGEXP_MATCH -c compile.c
gcc -g -O2 -fno-crossjumping -I. -Ic:/ruby/lib/ruby/1.9/i386-mingw32
-Ic:/ruby
/lib/ruby/1.9/i386-mingw32 -I. -DDISPATCH_THREADED_CODE
-DDISPATCH_DIRECT_THREAD
ED_CODE -DOPT_BASIC_OPERATIONS -DOPT_REGEXP_MATCH -c vm.c
cc1.exe: unrecognized option `-fno-crossjumping'
make: *** [vm.o] Error 1

16

gabriele renzi ha scritto:

gcc -g -O2 -fno-crossjumping -I. -Ic:/ruby/lib/ruby/1.9/i386-mingw32
-Ic:/ruby
/lib/ruby/1.9/i386-mingw32 -I. -DDISPATCH_THREADED_CODE
-DDISPATCH_DIRECT_THREAD
ED_CODE -DOPT_BASIC_OPERATIONS -DOPT_REGEXP_MATCH -c vm.c
cc1.exe: unrecognized option `-fno-crossjumping'
make: *** [vm.o] Error 1

fwiw, this was, again a winxp box on a athlon 2000+,
gcc version 3.2.3 (mingw special 20030504-1)

Using gcc version 3.4.2 (mingw-special) it compiles and runs fine on
that windows box.
BTW, in case I find some problems should I report them via the tracker
interface at rubyforge, or via yarv-devel, or just ignore them because
it's a 0.0.0x release?

17

Hi,

fwiw, this was, again a winxp box on a athlon 2000+,
gcc version 3.2.3 (mingw special 20030504-1)

Using gcc version 3.4.2 (mingw-special) it compiles and runs fine on
that windows box.

I added check that gcc is support this option or not.

BTW, in case I find some problems should I report them via the tracker
interface at rubyforge, or via yarv-devel, or just ignore them because
it's a 0.0.0x release?

Plsase post yarv-devel :slight_smile:

Regards,
ko1

18

I have a nice integer benchmark solving Connect-4,
currently available in Java and C (under 500 lines each):

http://www.cwi.nl/~tromp/c4/fhour.html

I'm currently busy porting it to Haskell,
but a Ruby version would also be a welcome addition.

Does anybody here have the time and energy to port it to Ruby?

regards,
-John