Benchmarking whole programs

ruby-talk
1

I don't know. Does the benchmark package take startup time into
account? You can always reverse the tests to ensure that they aren't
being tainted by startup time couldn't you?

Another thing you can do is to run your implementation through the
profiler. I also see ruby-prof on the RAA which you could try.

I also see YABM, but there are no links so I can't comment on that
package.

Dan

···

-----Original Message-----
From: gabriele renzi [mailto:surrender_it@remove-yahoo.it]
Sent: Tuesday, April 05, 2005 10:35 AM
To: ruby-talk ML
Subject: Benchmarking whole programs

Hi gurus and nubys,

I was playing with the ruby programs in the alioth language
shootout and I realized I have no idea on how to benchmark
them all. What I mean is: If I want to benchmark a single
function I can use the
usual benchmark code also shown in ri, but what if I want to
check two
slightly different implementations of a a single program?

I can't just stick them all in a #meth1 and #meth2 thing, since there
may be constants defined (actually they almost surely will, at least
classes and modules), I suppose I could do something like:
        Benchmark.bm do |x|
          x.report {require 'first_impl'; main()}
          x.report {require 'second_impl'; main()}
        end

but this seems kludgy and loading the thing at each run could mess up
the results, I think.

2

* Berger, Daniel <Daniel.Berger@qwest.com> [0442 17:42]:

···

> -----Original Message-----
> From: gabriele renzi [mailto:surrender_it@remove-yahoo.it]
> Sent: Tuesday, April 05, 2005 10:35 AM
> To: ruby-talk ML
> Subject: Benchmarking whole programs
>
>
> Hi gurus and nubys,
>
> I was playing with the ruby programs in the alioth language
> shootout and I realized I have no idea on how to benchmark
> them all. What I mean is: If I want to benchmark a single
> function I can use the
> usual benchmark code also shown in ri, but what if I want to
> check two
> slightly different implementations of a a single program?
>
> I can't just stick them all in a #meth1 and #meth2 thing, since there
> may be constants defined (actually they almost surely will, at least
> classes and modules), I suppose I could do something like:
> Benchmark.bm do |x|
> x.report {require 'first_impl'; main()}
> x.report {require 'second_impl'; main()}
> end
>
> but this seems kludgy and loading the thing at each run could mess up
> the results, I think.

I don't know. Does the benchmark package take startup time into
account? You can always reverse the tests to ensure that they aren't
being tainted by startup time couldn't you?

Another thing you can do is to run your implementation through the
profiler. I also see ruby-prof on the RAA which you could try.

For those who don't know -
Don't use that for a walltime, it slows things right down.

For a full app, what's wrong with

time first_impl.rb
time second_impl.rb

--
'Tempers are wearing thin. Let's hope some robot doesn't kill everybody.'
    -- Bender
Rasputin :: Jack of All Trades - Master of Nuns

3

Dick Davies wrote:

* Berger, Daniel <Daniel.Berger@qwest.com> [0442 17:42]:

Another thing you can do is to run your implementation through the
profiler. I also see ruby-prof on the RAA which you could try.

For those who don't know -
Don't use that for a walltime, it slows things right down.

The recently-posted ruby-prof extension by Shugo Maeda is absolutely
wonderful. Such a simple idea; why did it take so long? It times and
counts method calls with very low overhead using clock, gettimeofday, or
the CPU timer (on Pentium or PowerPC only). Unfortunely, it requires
Ruby 1.8 newer than 2005-03-22, which means CVS, or Ruby 1.9 newer than
2005-03-17.

I've seen a couple of other fast Ruby profilers, but they're either
closed-source or Windows-only. This contribution deserves some hearty
pimping action: go forth and profile with impunity, for it is no longer
intolerably slow.

Thanks Shugo!
jeremy

4

Dick Davies ha scritto:

I don't know. Does the benchmark package take startup time into
account? You can always reverse the tests to ensure that they aren't
being tainted by startup time couldn't you?

yup, I was thinking of doing the same, but see later

Another thing you can do is to run your implementation through the
profiler. I also see ruby-prof on the RAA which you could try.

I recall rbprof as part of the AspectR distribution, not sure if it still works. Anyhow, again, see later..

For those who don't know - Don't use that for a walltime, it slows things right down.

For a full app, what's wrong with

time first_impl.rb
time second_impl.rb

nothing, actually, even if I think that should be something like

time first_impl.rb
time second_impl.rb

to warm up the cache :slight_smile:

And nothing is wrong with the ideas that Daniel Berger pointed out, I was just wondering if there was something builtin in ruby's benchmark.rb or if people had wrote something like this, since it seem something quite common to think about.
Some changes like "use an array instead of a whole class" or "use two structs and a mixin instead of two classes", I mean, stuff that changes larger things that simple methods, are imho quite common, and even if probably the timing differences could be unvaluable it would be interesting to look at them :slight_smile:

Thanks both.