Ruby Performance

ruby-talk
81

Hi Isaac,

Before we go any further, I want to be clear about a couple of things. I have no problems with what goes on around the shootout site. I don't think that you'll find many people here that do, since a lot of them probably found their own path to ruby. There are quite a few adventurous and curious souls on the list and they spend a lot of time pulling ruby apart trying to understand it more clearly and find ways to make it better. (Putting Matz in the unenviable position of having to reconsider some of the basic features of the language on a regular basis. If I were he, I'd have an ulcer by now...)

By all means, carry on with experimentation with all of the languages that you can get your hands on. But let's not pretend that we're measuring anything.

I was trying to answer "I don't think that I've learned much from the
shootout".

Let's try again - "What is it that we are learning about programming
languages?"

I don't know what you learn about programming languages from shootout,
the website states "Our goals are to learn about programming
languages..." - the goals of the folk administering the shootout are to
learn about programming languages...

I understand you here, but I don't think that you are very clear about this. More later...

Let's try again - "What do we gain from comparing performance in
"various (possibly meaningless) ways?"

Some perspective on how performance varies between programming language
implementations and tasks.

This is a part of what I take issue with. The site offers a great deal of disclaimers regarding the worthlessness of the "benchmarks" there. But most of the space on the main page is devoted to reporting and comparing these benchmarks.

I guess I'm not sure why the site is called "The Computer Language
Shootout". To me, a shootout feels like a thing that pits two entities
against one another in a competition. The result of a shootout is
definitely a winner and a loser, depending on the definition of victory.
   
History - Doug Bagley's Shootout begat Aldo Calpini's Win32 Shootout
begat Brent Fulgham's Shootout.

With regard to your justification of the name, I'm not sure that history is a good reason to keep it.

When you discuss the site, you talk about it as an educational tool. I'm
all for that kind of thing, and your response to me is reasonable and if I
were looking for help it would be helpful.

The problem is the use of the word benchmark. By claiming to offer
"benchmarks", the site is purporting to offer measured and standard
methods of comparisons between programming languages. The many
disclaimers show that these numbers should be taken with a grain of salt,
but the site's keywords, "benchmarking fast programming language benchmark
performance benchmarks shootout program" show otherwise.

And so I think that the site, or more specifically its design, is very
disingenuous. The site offers itself as a home for objective comparison
of the performance of programming languages. There is no such thing, and
so the site should not claim that there is.
   
The site provides multiple comparison programs, which show various
different language implementations "winning".

The site shows different ways to "win" - by CPU time, by memory use, by
LOCs.

The site provides a synthetic overall score and invites you to
"manipulate the multipliers and weights to make your favourite language
the fastest programming language in the Shootout" for "a solution that
is simple, neat, and wrong".

Does the site proclaim A is faster than B, or subvert that simplistic
notion?

Isaac, you entry into this discussion thread was the following post:

> That's even more simplistic than the benchmark programs on the Computer
> Language Shootout :slight_smile:
>
> Here's Ruby vs Perl
> http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&sort=fullcpu <http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&sort=fullcpu&gt;
>
> and here are the old Doug Bagley programs
> http://shootout.alioth.debian.org/old/benchmark.php?test=all&lang=ruby&sort=fullcpu <http://shootout.alioth.debian.org/old/benchmark.php?test=all&lang=ruby&sort=fullcpu&gt;

When you follow the Ruby vs Perl link, you are treated to a nice picture and some numbers, all of which (you would assume from reading the page) offer some insight into the comparative merits of the languages.

In this case, the site _does_ proclaim a winner in any number of categories.

The site talks out of both sides of its mouth. Even with all of the language condemning the "simplistic notion" of winners and losers, the site still offers results. It is these results that are dishonest and misleading.

I think that the site would work much better _without_ the results page. Why not let the curious discover their own results? Hands on experimentation is the best way to explore the concepts that you are interested in. Maybe you can find a way to offer free shell accounts that give access to the tools required to experiment with these "benchmarks".

Or at the very least, take the focus off of those confounded numbers.

82

[in reply to no single post in particular]

Well I really had no idea of how much emotional ettatchment people had
to their languages of choice.

When I posted my original question, I had no idea that the later
conversation would
contain such passion, humor, sarcasim and personal insult.

Any way, my orginal post stemmed from my curiosity with regards
to what Ruby was trying to achieve:

I'm a relatively new Ruby programmer, I am curious as to what Ruby is
trying to achieve that other scripting languages do not already offer (Apart
from the syntactic differences of yet another scripting language, that is).

The highly non-real-world and simplistic benchmarks were not meant to
provoke the zeolots, but rather well displayed my lack of understanding
for what Ruby was trying to achieve. Of which I still dont quite understand.

Based on my experience of Ruby (which no doubt has been partly
determined by my experiences with other languages) I can honestly say
that it offers me little that is new/unachievable in other languages.

That said, RAILS as a frame-work is very well thought out and has
well-understood and communicated goals and I'm sure that many more
quality software products will be built using Ruby.

Good day and fare-well to you all.

···

--
Brad.

83

Hi Matthew

-snip-

By all means, carry on with experimentation with all of the languages
that you can get your hands on. But let's not pretend that we're
measuring anything.

On the contrary, we are measuring 'something'.

imo The issue is interpretation - what relevance (if any) do those
measurements have, what worth (if any) do those measurements have, what
(if anything) do they mean?

-snip-

>Some perspective on how performance varies between programming language
>implementations and tasks.
>
This is a part of what I take issue with. The site offers a great deal
of disclaimers regarding the worthlessness of the "benchmarks" there.
But most of the space on the main page is devoted to reporting and
comparing these benchmarks.

Flawed is not a synonym for worthless :wink:

-snip-

>History - Doug Bagley's Shootout begat Aldo Calpini's Win32 Shootout
>begat Brent Fulgham's Shootout.
>
With regard to your justification of the name, I'm not sure that history
is a good reason to keep it.

You may be right about that.

-snip-

When you follow the Ruby vs Perl link, you are treated to a nice picture
and some numbers, all of which (you would assume from reading the page)
offer some insight into the comparative merits of the languages.

In this case, the site _does_ proclaim a winner in any number of categories.

Yes - under said conditions, said program X was measured to be A,B,C
and said program Y was measured to be D,E,F.

imo that's not The issue.

The site talks out of both sides of its mouth. Even with all of the
language condemning the "simplistic notion" of winners and losers, the
site still offers results. It is these results that are dishonest and
misleading.

Let me clarify what I meant by "simplistic notion".

I meant it's simplistic to measure one program using language
implementation A, and one program using language implementation B, and
then *generalize* to say that language A is faster than language B.

I meant it's simplistic to measure cpu time without measuring memory
use, and then claim that program A is faster than program B.

It's simply true that under said conditions, said program X was
measured to be A,B,C and said program Y was measured to be D,E,F -
there's nothing dishonest about those results.

I think that the site would work much better _without_ the results
page.

Which page is "the results page"?

···

Why not let the curious discover their own results? Hands on
experimentation is the best way to explore the concepts that you are
interested in. Maybe you can find a way to offer free shell accounts
that give access to the tools required to experiment with these
"benchmarks".

Or at the very least, take the focus off of those confounded numbers.

84

Bradley Kite wrote:

[in reply to no single post in particular]

Well I really had no idea of how much emotional ettatchment people had
to their languages of choice.

When I posted my original question, I had no idea that the later
conversation would
contain such passion, humor, sarcasim and personal insult.

There has always been heated debate about which programming language was
best (or better than another). "better" is not easily defined (and it
also changes with context) so there's plenty room for personal taste and
preference. These in turn increase the likelyhood of heated debate...

Any way, my orginal post stemmed from my curiosity with regards
to what Ruby was trying to achieve:

I'm a relatively new Ruby programmer, I am curious as to what Ruby
is trying to achieve that other scripting languages do not already
offer (Apart from the syntactic differences of yet another
scripting language, that is).

The highly non-real-world and simplistic benchmarks were not meant to
provoke the zeolots, but rather well displayed my lack of
understanding
for what Ruby was trying to achieve. Of which I still dont quite
understand

Hm... Maybe you should just use it for a while and see for yourself?

IMHO Ruby tries to be a pure OO dynamic language with clean syntax that
doesn't make simple tasks more complicated than necessary. And it does
this just great. My 0.02 EUR.

Kind regards

    robert

85

When I posted my original question, I had no idea that the later
conversation would contain such passion, humor, sarcasim and
personal insult.

It's Usenet. There's one in every crowd.

Based on my experience of Ruby (which no doubt has been partly
determined by my experiences with other languages) I can honestly say
that it offers me little that is new/unachievable in other languages.

<smile> Well for any Turing-complete language, there's nothing
technically achievable in one that isn't also achievable in all
the others.

The one thing Turing-completeness doesn't cover is the number
and/or degree of headaches the programmer is subjected to before
the desired achievement is accomplished.

From what I've read from other Rubyists, and from my own
experience, many of us use Ruby because we find it more fun and
more productive than other languages we've learned so far.

(For me, that's Forth, C, C++, Java, Perl, Python, and Smalltalk.)

Best wishes to you in whatever language YOU find most fun and
most productive. It's not going to be Ruby for everyone, it just
is for a lot of us.

Regards,

Bill

···

From: "Bradley Kite" <bradley.kite@gmail.com>

86

When you follow the Ruby vs Perl link, you are treated to a nice picture
and some numbers, all of which (you would assume from reading the page)
offer some insight into the comparative merits of the languages.

In this case, the site _does_ proclaim a winner in any number of
categories.

Yes - under said conditions, said program X was measured to be A,B,C
and said program Y was measured to be D,E,F.

imo that's not The issue.

Aha! It appears that this is where we'll have to agree to disagree. For
me this is The issue. There were no conditions mentioned on the page
that you linked to.

There are so many factors that contribute to any measure of performance
for a "program" that any comparison must be qualified to the point of
irrelevance. Some such factors might be:

- Compiled vs Interpreted
-- Implementation of the Compiler/Interpreter
--- Version of the implementation
-- Compilation flags

- Implementation of the algorithm
-- Suitability of the implementation to the algorithm

- System Environment and its affects on the execution environment.

I could spend all day listing considerations like these. To attempt to
boil all of this down to one number for any measure of performance is
naive. These numbers aren't useful to the casual observer who will
misunderstand them, nor are they useful to the person who spends the time
to investigate the execution environment and draw their own conclusions.

Good luck with the site Isaac. I hope that it continues to go well for
you and that you continue to learn from it.

87

Bradley Kite wrote:

[in reply to no single post in particular]

Well I really had no idea of how much emotional ettatchment people had
to their languages of choice.

When I posted my original question, I had no idea that the later
conversation would
contain such passion, humor, sarcasim and personal insult.

Any way, my orginal post stemmed from my curiosity with regards
to what Ruby was trying to achieve:

>> I'm a relatively new Ruby programmer, I am curious as to what Ruby is
>> trying to achieve that other scripting languages do not already offer (Apart
>> from the syntactic differences of yet another scripting language, that is).

The highly non-real-world and simplistic benchmarks were not meant to
provoke the zeolots, but rather well displayed my lack of understanding
for what Ruby was trying to achieve. Of which I still dont quite understand

"a genuine object-oriented, easy-to-use scripting language"

88

ogilthorpe@davie.textdrive.com wrote:

>> When you follow the Ruby vs Perl link, you are treated to a nice picture
>> and some numbers, all of which (you would assume from reading the page)
>> offer some insight into the comparative merits of the languages.
>>
>> In this case, the site _does_ proclaim a winner in any number of
>> categories.
>
> Yes - under said conditions, said program X was measured to be A,B,C
> and said program Y was measured to be D,E,F.
>
> imo that's not The issue.

Aha! It appears that this is where we'll have to agree to disagree. For
me this is The issue. There were no conditions mentioned on the page
that you linked to.

There are so many factors that contribute to any measure of performance
for a "program" that any comparison must be qualified to the point of
irrelevance. Some such factors might be:

- Compiled vs Interpreted
-- Implementation of the Compiler/Interpreter
--- Version of the implementation

The specific implementation and version is given at the bottom of the
page I linked to
http://shootout.alioth.debian.org/benchmark.php?test=all&lang=ruby&sort=fullcpu#about

-- Compilation flags

Are shown for each program, they may vary program to program, for
example
http://shootout.alioth.debian.org/benchmark.php?test=ackermann&lang=gcc&id=3&sort=fullcpu#log

- Implementation of the algorithm
-- Suitability of the implementation to the algorithm

Not sure exactly what you mean, perhaps this
http://shootout.alioth.debian.org/benchmark.php?test=ackermann&lang=all&sort=fullcpu#about

- System Environment and its affects on the execution environment.

Is given in the FAQ
http://shootout.alioth.debian.org/faq.php?sort=fullcpu#measure

I could spend all day listing considerations like these. To attempt to
boil all of this down to one number for any measure of performance is
naive. These numbers aren't useful to the casual observer who will
misunderstand them, nor are they useful to the person who spends the time
to investigate the execution environment and draw their own conclusions.

Good luck with the site Isaac. I hope that it continues to go well for
you and that ou continue to learn from it.

Thank you.