Bottom line: Ruby needs help in the performance area. Let's admit that
and work on improving in that area instead of shooting the messenger -
it'll make us look like whiners if we keep doing the latter. I'd really
rather not have the Ruby community perceived that way from the outside.
[...]
Im slowly getting fed up with this whining, "Ruby is so slow".
Especially if it's compared to Python.
Ever directly compared Ruby vs. Python solutions with regards to
performance?
Python is damn slow at startup, despite loading byte- instead of
sourcecode. On my machine, there is (approximately) a linear
ratio between LOPC and startup time: About 0.1 second per 200
LOPC. I've a Ruby script and a Python script doing the same thing
with a roughly equal model: The Ruby version takes ~0.06s,
the Python script ~0.26s for startup. Just to emphasize that a
bit: that's a factor of 4.3. (And the Ruby script is even
longer because it has some additional features.)
For longer program runs, the Python version comes closer
(but never surpasses) the Ruby one.
In conclusion, IMO Ruby will perform better for batch processing
and I don't know if the opposite is true for bigger/longer running
applications. I don't believe that Python is faster for serious
apps (not just 5 line benchmarks) unless someone shows me a program
written in Ruby and Python were the Python one is significantly faster.
Which is one of the many reasons that I have a problem with the
alioth shootout. Ultimately, it tries to treat compiled, semi-
compiled, and interpreted languages equally. They're not. Compiled
languages will be faster than semi-compiled (bytecode), which will
be faster than interpreted. The level of speed difference is a
matter of scale, and that scale also matters on development time and
clarity.
I have to say that I tend to agree with the astonishment.
Here's what I mean:
Someone comes along and asks about python vs. Ruby performance and
references the alioth benchmarks. Several folks jump on the alioth
benchmarks and call them bogus. I see this as blaming the messanger.
That's putting it politely. The argument against, which has merit, is
stated with so much hyperbole that it damages its own credibility.
First, there are the ad hominem attacks. Words like "brainless", "inane"
and "bullshit". Don't tell me these aren't personal; they are.
Then there's the rhetorical exaggeration: If this benchmark has flaws,
then it has no value. If this benchmark has no value, then no benchmarks
have value. Culminating, of course, in "Benchmarks, like statistics, are
lies." I suppose I could be charitable and interpret that as a
hyperbolic way to say "Benchmarks, like statistics, tell only part of
the story. Use caution in their construction and interpretation." If
that's what was meant, then the original statement is itself,
ironically, a "lie" in precisely the same way. If it was meant
literally, then it betrays a deep ignorance of the practice of science
and engineering, among other things.
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
If you don't have time to see why Ruby did so poorly in the alioth
Ackermann function, why did you say anything? Sorry, but the alioth
benchmarks are crap -- and always will be crap. Trying to improve
Ruby's score in them is a loser's game.
If we keep telling ourselves that Ruby is 'fast
enough' for our application (and it may well be) are we going to be
sitting still while other languages improve performance?
luckily there is work on this.
what i don't get it why everyone
talks about it rather than actually working on it
If i can't use the language because it is to slow to solve the problem
then the development time doesn't matter. There are just too many
problem domains left where you can't ignore it.
I would like to hear some more specifics on these problem domains. I'm not
trying to start an argument, I'm just wondering if the problems you are
thinking of are just not appropriate for any interpreted language.
Ruby or Python or Java will never be as fast as C, and C in most cases is
not as fast as good hand-coded assembly. Any problem domain that requires
major number crunching will need to be coded in the most efficient
language, sometimes even requiring special hardware (like 3D graphics.)
Of course with modern computers and compilers, I doubt anyone would code
an entire application in assembly, but rather entirely in C or with just
the hot spots coded in assembly. The same philosophy applies when moving
to the higher level of Ruby, with C for the hot spots. I certainly
wouldn't try to write a web application with C these days.
In fact I would say that in most cases, for many problem domains, the huge
increase in productivity that you get with higher level languages totally
outweighs performance issues.
I agree, It would be nice if ruby was faster. Luckily,
people are working on this.
Don't go away thinking that X is faster than Ruby. Do
your own tests. We tested Perl against Ruby in a real
world problem (netlist parsing) and Ruby was faster
than Perl by 30%. When we did a C implementation
(which could now be done automatically with optimize)
it was 100% faster (ie, took half the time.)
Are you saying that the Ruby netlist parser was 100% faster than the C
implementation or the other way around?
BTW: I recall someone at a PDX.rb meeting who said he had benchmarked
Ruby vs. OO-style Perl and Ruby was quite a bit faster (don't recall the
percentage). As I recall the explanation was that doing OO in perl
requires more levels of indirection and thus more function calls.
I'm sorry you do. However, yours should be one of the ones that should
be able to modify the depth of the stack with ulimit.
If I cared enough to get more than annoyed at this damned shootout
with its inane implementation rules, I'd do more. But as it is, the
best I can tell people is that the Alioth shootout is bullshit.
-austin
···
On 6/10/05, Isaac Gouy <igouy@yahoo.com> wrote:
Austin Ziegler wrote:
> The definition of Ackermann numbers is, I presume, well known. Because
> some platforms can't modify their stack level (Windows), and because
> the stack level isn't modified in the runs for the benchmark, there's
> going to be an issue with the deep recursion.
We use Debian Linux, so tell us how to modify the stack level for Ruby.
On Jun 10, 2005, at 1:02 PM, Austin Ziegler wrote:
>
> I personally think that most of the shootout items are bogus in just
> about every way. Don't get me wrong -- I'm not at all suggesting that
> Ruby couldn't be faster. I'm just saying that benchmarks, like
> statistics, are lies.
>
> -austin
> --
> Austin Ziegler * halostatue@gmail.com
> * Alternate: austin@halostatue.ca
>
>
The most succinct way that I've heard it put is that there are "lies,
damn lies, and benchmarks".
Trey Campbell
treycampbell@mac.com
There isn't with benchmarks per-se; there can be a problem with how one
chooses to interpret benchmarks.
And there's the real tragedy. People who buy into the crap that is benchmarks.
There is only one benchmark that truly matters. Are your users
complaining about the speed of your program? If so, then you need to
optimise it. If not, then it's good enough.
Everything else is lies and marketing. Things like the alioth
benchmarks don't actually shed any light on anything; they are great
at obscuring. Nothing more.
-austin
···
On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
So you're saying that when the performance requirements crosses a
certain threshold, interpreted languages should not be used.
The idea here is that if Ruby is faster, that threshold where should a
switch needs to be made is shifted.
···
On 6/10/05, Ryan Leavengood <mrcode@netrox.net> wrote:
I would like to hear some more specifics on these problem domains. I'm not
trying to start an argument, I'm just wondering if the problems you are
thinking of are just not appropriate for any interpreted language.
Actually, at a macro level there is no reason that Ruby or Python or Java
can not be as fast as C.
An example here is IronPython. Some of the python benchmarks run FASTER
under IronPython compared to the C Python.
Virtual Machine's are pretty darn smart now, and with the abstraction can do
smart things, like changing out the native code on the fly to end up with
smarter versions (vs. a one pass compilation step).
As an industry we often move up a chain, especially for app dev work.
I have never run into a bottleneck in a web application that occurred due to
the programming language. As soon as you are dealing with networks, disks,
and the like, architecture is always the key.
Just my 2p.
Cheers,
Dion
···
-----Original Message-----
From: Ryan Leavengood [mailto:mrcode@netrox.net]
Sent: Thursday, June 09, 2005 1:38 PM
To: ruby-talk ML
Subject: Re: python/ruby benchmark.
Lothar Scholz said:
>
> If i can't use the language because it is to slow to solve
the problem
> then the development time doesn't matter. There are just too many
> problem domains left where you can't ignore it.
I would like to hear some more specifics on these problem
domains. I'm not trying to start an argument, I'm just
wondering if the problems you are thinking of are just not
appropriate for any interpreted language.
Ruby or Python or Java will never be as fast as C, and C in
most cases is not as fast as good hand-coded assembly. Any
problem domain that requires major number crunching will need
to be coded in the most efficient language, sometimes even
requiring special hardware (like 3D graphics.)
Of course with modern computers and compilers, I doubt anyone
would code an entire application in assembly, but rather
entirely in C or with just the hot spots coded in assembly.
The same philosophy applies when moving to the higher level
of Ruby, with C for the hot spots. I certainly wouldn't try
to write a web application with C these days.
In fact I would say that in most cases, for many problem
domains, the huge increase in productivity that you get with
higher level languages totally outweighs performance issues.
* Phil Tomson <ptkwt@aracnet.com> [2005-06-10 13:55:28 +0900]:
In article <20050610024316.GA21914@freeze.org>,
>* gabriele renzi <surrender_it@remove-yahoo.it> [2005-06-10 09:35:28 +0900]:
>
>> I agree, It would be nice if ruby was faster. Luckily,
>> people are working on this.
>
>Don't go away thinking that X is faster than Ruby. Do
>your own tests. We tested Perl against Ruby in a real
>world problem (netlist parsing) and Ruby was faster
>than Perl by 30%. When we did a C implementation
>(which could now be done automatically with optimize)
>it was 100% faster (ie, took half the time.)
Are you saying that the Ruby netlist parser was 100% faster than the C
implementation or the other way around?
The C implementation was Ruby C (my bad). So, the Ruby C
implementation ended up twice as fast as the Perl
implementation. Bad comparison you say. Well, in this
environment, we found over four netlist parsers written
in Perl. These parsers were intermingled with the netlist
munging code, so it was difficult to do abstraction and
pull out the netlist parser in Perl and optimize it.
But you say, well all that could and should have been done in
Perl. My response is, well, you had 10 years, what were you
waiting for?
The pragmatic nature of Ruby lent itself to writing better code,
more modular code, and thus the ability to optimize a particular
module muuuchh more than Perl.
···
Jim Freeze <jim@freeze.org> wrote:
--
Jim Freeze
Theory and practice are the same, in theory. -- Ryan Davis
No, it's the benchmarks themselves that are the problem. Remember
that NVidia got caught detecting a benchmark in their drivers and
optimizing output for the benchmark.
Similarly, this thread was started because someone came across your
language shootout benchmark page and assumed that Ruby is
dramatically slower in Real Life Examples because it's slower in the
benchmarks. Which, I can assure you, it most assuredly NOT.
You get it correct when you indicate that "the best benchmark is
your program." That's the only one that matters. It also lets you
measure benchmarks that can't be categorised in CPU seconds or
memory use or FLOPS or other nonsense like that.
It's hype, but the idea that Rails makes one 10x more productive is
part of the value of Ruby. Unless your program is of a special case
-- and a lot of these special cases are handled already -- then Ruby
programs will be developed faster and more easily maintained than
their non-Ruby counterparts. Enough so that execution speed is
something of a wash.
At any rate, I consider the Alioth shootout to be harmful to all
languages involved. There is no useful value provided by it,
especially as it does not permit language-appropriate modifications
to the algorithms in use.
-austin
···
On 6/10/05, Isaac Gouy <igouy@yahoo.com> wrote:
Trey Campbell wrote:
On Jun 10, 2005, at 1:02 PM, Austin Ziegler wrote:
I personally think that most of the shootout items are bogus in
just about every way. Don't get me wrong -- I'm not at all
suggesting that Ruby couldn't be faster. I'm just saying that
benchmarks, like statistics, are lies.
The most succinct way that I've heard it put is that there are
"lies, damn lies, and benchmarks".
There isn't with benchmarks per-se; there can be a problem with
how one chooses to interpret benchmarks.
#: the mind was *winged* after Austin Ziegler said on 6/12/2005 5:53 PM :#
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
And there's the real tragedy. People who buy into the crap that is benchmarks.
There is only one benchmark that truly matters. Are your users
complaining about the speed of your program? If so, then you need to
optimise it. If not, then it's good enough.
Everything else is lies and marketing. Things like the alioth
benchmarks don't actually shed any light on anything; they are great
at obscuring. Nothing more.
-austin
Austin, I am still wondering: you develop f.e. 3 months to find out if your client will accept the
software you have invested into? All the companies I have been employed into were not working like
this. We have always built prototypes and decided from that point. And a prototype is a benchmark in
fact.
Also, I agree with the fact the optimization comes later in the dev cycles. But we need to have the
doors open to optimize. Moreover, I read that in Ruby you can always go and code the hot spots in C.
What if you don't have these resources? Which are the costs to bring such a resource and make him
productive?
Maybe I am completely wrong, but this is the way the companies I have worked with are proceeding.
:alex |.::the_mindstorm::.|
···
On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
And there's the real tragedy. People who buy into the crap that is benchmarks.
What's more amazing is that these fools why buy into this "crap" have
made fundamental contributions to the theory and practice of digital
communications, image processing, and numerical analysis, practically
invented systems engineering, found volcanoes on Io and landslides on
Venus, communicated with spacecraft beyond the edge of the solar system,
put three rovers on the surface of Mars, and a whole bunch of other
stuff that will go into history books.
But of course, you know better. You have A Blog and have written Some
Software.
Steve
···
On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:
So you're saying that when the performance requirements crosses a
certain threshold, interpreted languages should not be used.
The idea here is that if Ruby is faster, that threshold where should a
switch needs to be made is shifted.
Of course, and I'm all for that. I want Ruby to be as fast as possible. In
fact I have some application domains in mind I'd like to use Ruby for that
may run into this exact problem (sound processing.)
But still, at this point in the state of computing, I would not use Ruby
in certain applications:
- operating system level code.
- heavy duty 3D rendering.
- device drivers.
- any major number crunching (math, video processing, low-level image
manipulation.)
But hey, maybe with some special hardware (a Ruby Chip?), all the above
would be possible and fast with Ruby. That may be the next level of
computing: hardware accelerated high level languages.
> There isn't with benchmarks per-se; there can be a problem with
> how one chooses to interpret benchmarks.
No, it's the benchmarks themselves that are the problem. Remember
that NVidia got caught detecting a benchmark in their drivers and
optimizing output for the benchmark.
Similarly, this thread was started because someone came across your
language shootout benchmark page and assumed that Ruby is
dramatically slower in Real Life Examples because it's slower in the
benchmarks.
As I said - There isn't a problem with benchmarks per-se; there can be
a problem with how one chooses to interpret benchmarks.
Which, I can assure you, it most assuredly NOT.
Not "dramatically slower" than what other language, for what specific
"Real Life Examples"?
It's hype, but the idea that Rails makes one 10x more productive is
part of the value of Ruby. Unless your program is of a special case
-- and a lot of these special cases are handled already -- then Ruby
programs will be developed faster and more easily maintained than
their non-Ruby counterparts. Enough so that execution speed is
something of a wash.
Faster to develop-in than what other languages - Smalltalk? Lisp?
Mathematica?
At any rate, I consider the Alioth shootout to be harmful to all
languages involved. There is no useful value provided by it,
especially as it does not permit language-appropriate modifications
to the algorithms in use.
What's a language-appropriate modification?
Have you ever asked for one to be accepted in the Alioth shootout?
How does this issue relate to *ruby*? The same charge could be levied
at python, even J2EE. "What happens if you don't have the resources
to speed up something slow?"
I tend to agree with Austin on this; theoretical, academic arguments
don't mean much when a customer wants his software.
It surprised me coming up through the application development
(financial applications) world, but some customers even *WANT* early
releases to be slow, so they can feel good that they tell the
developers to make it faster, and they do.
···
On 6/12/05, Alexandru Popescu <the_mindstorm@evolva.ro> wrote:
Also, I agree with the fact the optimization comes later in the dev cycles. But we need to have the
doors open to optimize. Moreover, I read that in Ruby you can always go and code the hot spots in C.
What if you don't have these resources? Which are the costs to bring such a resource and make him
productive?
#: the mind was *winged* after Alexandru Popescu said on 6/12/2005 6:15 PM :#
#: the mind was *winged* after Austin Ziegler said on 6/12/2005 5:53 PM :#
In my line of work, we use benchmarks and statistics extensively. They
may be useful for lying; so what? They are far more powerful for telling
the truth.
And there's the real tragedy. People who buy into the crap that is benchmarks.
There is only one benchmark that truly matters. Are your users
complaining about the speed of your program? If so, then you need to
optimise it. If not, then it's good enough.
Everything else is lies and marketing. Things like the alioth
benchmarks don't actually shed any light on anything; they are great
at obscuring. Nothing more.
-austin
Austin, I am still wondering: you develop f.e. 3 months to find out if your client will accept the
software you have invested into? All the companies I have been employed into were not working like
this. We have always built prototypes and decided from that point. And a prototype is a benchmark in
fact.
Also, I agree with the fact the optimization comes later in the dev cycles. But we need to have the
doors open to optimize. Moreover, I read that in Ruby you can always go and code the hot spots in C.
What if you don't have these resources? Which are the costs to bring such a resource and make him
productive?
Maybe I am completely wrong, but this is the way the companies I have worked with are proceeding.
:alex |.::the_mindstorm::.|
You may wonder why I keep saying this kind of things.
I want to understand how you ruby working guys are managing this. Usually only the success stories
go public, but I know behind these there are lots of failures.
:alex |.::the_mindstorm::.|
···
On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:
