Python/ruby benchmark

ruby-talk
41

Austin Ziegler wrote:
What's more amazing is that these fools why buy into this "crap" have
made fundamental contributions to the theory and practice of digital
communications, image processing, and numerical analysis, practically
invented systems engineering, found volcanoes on Io and landslides on
Venus, communicated with spacecraft beyond the edge of the solar system,
put three rovers on the surface of Mars, and a whole bunch of other
stuff that will go into history books.

Huh. They didn't do it with benchmarks. (And I could very easily point
out that those same people have screwed up massively -- forgetting to
convert between English units and Metric units?) Look closely at the
people who espouse benchmarks. They're mostly marketers or fools who
can't tell the difference.

There are *real* measures to deal with; they're not benchmarks. They
aren't and never have been.

For a first person shooter, the real measure is "is the game fun?" The
answer will be different for everyone, but there are some objective
things that will break the "fun" factor for just about everyone.
Frames Per Second. Load Time. These things should be as fast as they
possibly can. Gigaflops never enters the question here. Nor does
specmark or anything else like that.

For image manipulations, they need to be quick. But not once does the
Ackermann function ever enter the question.

But of course, you know better. You have A Blog and have written Some
Software.

I do know better. It's not because I have a blog (I do, but I haven't
updated it in months, because I've been busy writing Real Software for
Real People). And I've written a LOT of software. No, I've never
worked for NASA. But I have real world experience, and I know what I
speak of.

Never *once* have I needed to implement an Ackermann function. Not
once. In my entire career. I look at the crap that is on alioth and
there's very little that represents common use. There's some neat
things -- the new DNA transformation ones -- but exactly how many
people will actually be using that in their work?

I won't. Wouldn't have my entire career. Measures that mattered to me
at my last job were "how many bills can I generate in an hour?" At my
current job "what's the average backup speed throughput for this?"

If we're not getting the performance we need, we fix the damn problem.
We don't rely on "benchmarks" -- we rely on real world measurements of
our real problems. Not on pseudo-crap like gigaflops or specmark or
the speed of an airborne swallow. Actually, strike that. The last is
useful.

-austin

···

On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:
--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

42

check out some of the things people are doing

   http://sciruby.codeforpeople.com/wiki.cgi/InterestingProjects

this is for realtime video processing

   http://gridflow.ca/

cheers.

-a

···

On Fri, 10 Jun 2005, Ryan Leavengood wrote:

Gavri Fernandez said:

So you're saying that when the performance requirements crosses a
certain threshold, interpreted languages should not be used.
The idea here is that if Ruby is faster, that threshold where should a
switch needs to be made is shifted.

Of course, and I'm all for that. I want Ruby to be as fast as possible. In
fact I have some application domains in mind I'd like to use Ruby for that
may run into this exact problem (sound processing.)

But still, at this point in the state of computing, I would not use Ruby
in certain applications:

- operating system level code.
- heavy duty 3D rendering.
- device drivers.
- any major number crunching (math, video processing, low-level image
manipulation.)

--

email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
My religion is very simple. My religion is kindness.
--Tenzin Gyatso

===============================================================================

43

In article <10625.206.157.248.34.1118348101.squirrel@www.netrox.net>,

Gavri Fernandez said:

So you're saying that when the performance requirements crosses a
certain threshold, interpreted languages should not be used.
The idea here is that if Ruby is faster, that threshold where should a
switch needs to be made is shifted.

Of course, and I'm all for that. I want Ruby to be as fast as possible. In
fact I have some application domains in mind I'd like to use Ruby for that
may run into this exact problem (sound processing.)

But still, at this point in the state of computing, I would not use Ruby
in certain applications:

- operating system level code.
- heavy duty 3D rendering.
- device drivers.
- any major number crunching (math, video processing, low-level image
manipulation.)

But hey, maybe with some special hardware (a Ruby Chip?),

RubyChip: Introduced in 2010. Apple adopts it in 2012, moves away from
Intel. :wink:

all the above
would be possible and fast with Ruby. That may be the next level of
computing: hardware accelerated high level languages.

Well, maybe not so new. There was a Lisp machine back in the 80's as I
recall. There was also a Forth chip made by Chuck Moore back then too.
Of course there are also HW implementations of the JVM (PicoJava).

Now that we've got fairly inexpensive FPGAs (like the Spartan 3 family
from Xilinx) it's possible that you can do this sort of thing in FPGAs.
It's certainly an intriguing idea.

Phil

···

Ryan Leavengood <mrcode@netrox.net> wrote:

44

Hello Ryan,

But still, at this point in the state of computing, I would not use Ruby
in certain applications:

- operating system level code.
- heavy duty 3D rendering.
- device drivers.
- any major number crunching (math, video processing, low-level image
manipulation.)

Don't forget

- code that must do a lot of parsing
- code that works on large datasets (a GC problem)
- any algorithmic problem

···

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

45

Isaac Gouy wrote:

As I said - There isn't a problem with benchmarks per-se; there can be
a problem with how one chooses to interpret benchmarks.

That's a meaningless distinction. You are suggesting that people who read your benchmarks will somehow interpret them and find information that is simply not there. The benchmarks paint an overly simplified picture, nothing more. The only way to interpret the numbers correctly is to ignore them and look for more pragmatic examples.

I do think there is some small value in the comparitive coding samples, but the rigidity of the rules means that there is very little ingenuity allowed in the code, and thus all the code ends up looking extremely similar, regardless of the language employed. If you enjoy probing variations in whitespace and the placement of semi-colons, you will have lots of fun.

Faster to develop-in than what other languages - Smalltalk? Lisp?
Mathematica?

Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed, Lisp. God help you if you decide to implement a web framework or a compiler in Mathematica.

The real problem with the alioth benchmarks is that they are run by amateurs that allow themselves to be bound by rigid pseudo-academic dogma, but they can't research and fix trivial problems on their own, and then they whine when nobody does it for them. If you guys aren't highly motivated to fix things, why do you think the rest of us will be?

I gave up on them a while ago.

···

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/&gt;

46

Austin Ziegler wrote:

If we're not getting the performance we need, we fix the damn problem.
We don't rely on "benchmarks" -- we rely on real world measurements of
our real problems. Not on pseudo-crap like gigaflops or specmark or
the speed of an airborne swallow. Actually, strike that. The last is
useful.

Yeah, I rely on the speed of airborne swallows at least three times as often as Ackermann functions. :wink:

Curt

47

Austin Ziegler wrote:
-snip-

Never *once* have I needed to implement an Ackermann function. Not
once. In my entire career. I look at the crap that is on alioth and
there's very little that represents common use. There's some neat
things -- the new DNA transformation ones -- but exactly how many
people will actually be using that in their work?

Never once needed to implement a recursive function?

Some details are always specific to a problem domain and application,
but the same representations and approaches are used across problem
domains and across applications - yes the details are from DNA sequence
analysis, but the programs process ascii strings and hashtables.

48

Austin Ziegler wrote:

Huh. They didn't do it with benchmarks.

They did. I was there; you weren't. But don't let facts get in the way.

Steve

49

Hello Michael,

Also, I agree with the fact the optimization comes later in the
dev cycles. But we need to have the
doors open to optimize. Moreover, I read that in Ruby you can
always go and code the hot spots in C.
What if you don't have these resources? Which are the costs to
bring such a resource and make him
productive?

How does this issue relate to *ruby*? The same charge could be levied
at python, even J2EE. "What happens if you don't have the resources
to speed up something slow?"

If a language is a few times faster then the risk factor that this
happens is just lower. This is a management descision.

There are many things to keep in balance when you start a project.
Development time is just one of them and often not that important.

I can just speak for the application domain where i worked for years:
This is products for the mass market. Here you don't have something
like "your customer". In this market you must look a lot about "your
competitor" and this makes it different. Spending a few month more on
development is just okay if the (peak !!) speed is well and
competitive.

This is a domain where Ruby is unfortunately not useable at the
moment (missing a good GUI toolkit is the second reason). And i would
like to see it possible to use it there too.

···

On 6/12/05, Alexandru Popescu <the_mindstorm@evolva.ro> wrote:

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

50

In article <9e7db911050612113833d7fd7a@mail.gmail.com>,

Austin Ziegler wrote:
What's more amazing is that these fools why buy into this "crap" have
made fundamental contributions to the theory and practice of digital
communications, image processing, and numerical analysis, practically
invented systems engineering, found volcanoes on Io and landslides on
Venus, communicated with spacecraft beyond the edge of the solar system,
put three rovers on the surface of Mars, and a whole bunch of other
stuff that will go into history books.

Huh. They didn't do it with benchmarks. (And I could very easily point
out that those same people have screwed up massively -- forgetting to
convert between English units and Metric units?) Look closely at the
people who espouse benchmarks. They're mostly marketers or fools who
can't tell the difference.

There are *real* measures to deal with; they're not benchmarks. They
aren't and never have been.

For a first person shooter, the real measure is "is the game fun?" The
answer will be different for everyone, but there are some objective
things that will break the "fun" factor for just about everyone.
Frames Per Second. Load Time. These things should be as fast as they
possibly can. Gigaflops never enters the question here. Nor does
specmark or anything else like that.

For image manipulations, they need to be quick. But not once does the
Ackermann function ever enter the question.

No, but as you've said, you need quick response from a game in order for
it to be engaging. It comes down to frames per second (as you note).

Never *once* have I needed to implement an Ackermann function. Not
once. In my entire career.

No one is saying that you would need to. The Ackermann benchmark can be
a measure of how well your language handles recursion (of course it can
be written iteratively as well). Sure it may not be something that you'd
ever use, but if someone is perusing the benchmark results and sees that
Ruby falls way behind gawk in this particular benchmark, they're likely
to draw some conclusions. Now you and I might think that the conclusions
they draw are ot fair or accurate, but that doesn't matter. If I've
never encountered Ruby before and all I know of it is from the alioth
benchmarks, the conclusions I come to will not be positive.

I look at the crap that is on alioth and
there's very little that represents common use. There's some neat
things -- the new DNA transformation ones -- but exactly how many
people will actually be using that in their work?
I won't. Wouldn't have my entire career.

Not everyone is doing web development. Some people work in interesting
areas like Bioinformatics. It's great that Ruby is finding a niche in web
development with Rails, but I really hope that doesn't translate into
Ruby only aspiring to be another PHP (ie. being perceived as a language
that is only useful for web development - I'm afraid that's already
starting to happen).

The fact is that it's in various scientific areas where you'll find the
most compute-intensive applications (the other area would be gaming), so
including DNA transformation benchmarks can be very informative.

Measures that mattered to me
at my last job were "how many bills can I generate in an hour?" At my
current job "what's the average backup speed throughput for this?"

If we're not getting the performance we need, we fix the damn problem.

Good; we can all agree on this :wink:

We don't rely on "benchmarks" -- we rely on real world measurements of
our real problems. Not on pseudo-crap like gigaflops or specmark or
the speed of an airborne swallow. Actually, strike that. The last is
useful.

Again, benchmarks don't tell the whole story, but unfortuneately many
will judge Ruby based on these benchmarks - that's just reality. What
are you going to do, buy banner ads on the alioth site which read "Don't
trust these benchmarks!". I don't think that'll work.

Since benchmarks tend to be a uniform measure (we can't all trot out
our 'real world' code to be translated into all the popular languages and
then tested and timed in each one - and if we did that would just become
the next benchmark, wouldn't it?). A good set of benchmarks needs to be
easily implemented in all of the languages being compared and they also
need to exercise a lot of different features that would show up in real
world programs. I'm going to give the alioth folks the benefit of the
doubt here and assume that they want to develop a 'good' set of
benchmarks that can be used to measure relative performance of various
languages. We can either dismiss their efforts with various
pejoratives (which will make people outside of our community wonder
what we're trying to hide) or help out.

Phil

···

Austin Ziegler <halostatue@gmail.com> wrote:

On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:

51

Using 'set' is somewhat faster than Hash in 1.9.
I've rewritten that code using 'set' as following:

  require 'set'

  d = Set.new

  File.open('/usr/share/dict/words') do |f|
    d << $_.chop! while f.gets
  end

  STDIN.each do |l|
    puts l unless d.include? l.chop!
  end

Here are timed results in my machine.

  $ time echo zuul | ruby18 spellcheck-hash.rb
  zuul

  real 0m1.688s
  user 0m1.635s
  sys 0m0.045s

  $ time echo zuul | ruby18 spellcheck-set.rb
  zuul

  real 0m1.796s
  user 0m1.748s
  sys 0m0.048s

  $ time echo zuul | ruby19 spellcheck-hash.rb
  zuul

  real 0m1.694s
  user 0m1.624s
  sys 0m0.070s
  
  $ time echo zuul | ruby19 spellcheck-set.rb
  zuul

  real 0m1.465s
  user 0m1.411s
  sys 0m0.053s

  $ time echo zuul | ~/usr/bin/ruby -rite spellcheck-hash.rb
  zuul

  real 0m1.541s
  user 0m1.485s
  sys 0m0.054s

  $ time echo zuul | ~/usr/bin/ruby -rite spellcheck-set.rb
  zuul

  real 0m1.178s
  user 0m1.108s
  sys 0m0.062s

YARV is not that faster in this case. I guess ruby's collection
types (Array, Hash, Set...) are not sufficiently optimized than python.
I've heard that python 2.4 boosted 'list' performance upto 30% than
previous version from a python developer. I got a strong impression
on much lower memory usage through 'generator' too.

Anyway, ruby is very slower than many other languages.
But we can prove not so slower than currently presented in that site.
How about discussing about 'howto raise ruby ranking in the shootout
site'? :wink:

···

--
http://nohmad.sub-port.net

52

Hello Ryan,

> But still, at this point in the state of computing, I would not use Ruby
> in certain applications:

> - operating system level code.
> - heavy duty 3D rendering.
> - device drivers.
> - any major number crunching (math, video processing, low-level image
> manipulation.)

Don't forget

From a person that maintains a Ruby IDE, you're comments always surprise me...

- code that must do a lot of parsing

I've actually used Ruby for a pretty hefty parsing application in my work. It was my experience that this is an area where Ruby really shines. Funny to see you bring it up as the opposite of that now.

- code that works on large datasets (a GC problem)

I don't have much experience here, so I'll take your word for it.

- any algorithmic problem

You serious?! I have to say that I think that's absurd. Browsing RubyForge and RAA makes me think I'm not alone either. Have you looked at the Ruby Quiz solutions?

So basically what you're saying is that Ruby is unusable for most applications? I think it's safe to say you're in the minority with this opinion.

I hear you, that Ruby is slow. I agree. I work with Java and it's definitely faster. I've benchmarked similar Perl scripts and they're faster. Ruby may even be the slowest language I use and the amount of time this actually affects me is incredibly low. I want YARV as much as you do, I'm sure, but your comments seem ridiculous to me.

James Edward Gray II

···

On Jun 9, 2005, at 4:15 PM, Lothar Scholz wrote:

53

Well, maybe not so new. There was a Lisp machine back in the 80's as I
recall. There was also a Forth chip made by Chuck Moore back then too.
Of course there are also HW implementations of the JVM (PicoJava).

Don't forget the SOAR (Smalltalk On A RISC) project. There was even a _very_ interesting book published about it.

Also, given the distance (in clock cycles) between main memory and the CPU, I'm expecting to see a programmable microcode design come out. Such things were built in the 70's & 80's with some success. And, being a fan of Forth, using it as the internal microcode could be really interesting.

Now that we've got fairly inexpensive FPGAs (like the Spartan 3 family
from Xilinx) it's possible that you can do this sort of thing in FPGAs.
It's certainly an intriguing idea.

I would enjoy learning how to design with these. And, I would love to write a bunch of tools in Ruby for developing with them.

-- Matt
Nothing great was ever accomplished without _passion_

···

On Fri, 10 Jun 2005, Phil Tomson wrote:

54

Glenn Parker wrote:

Isaac Gouy wrote:
>
> As I said - There isn't a problem with benchmarks per-se; there can be
> a problem with how one chooses to interpret benchmarks.

That's a meaningless distinction. You are suggesting that people who
read your benchmarks will somehow interpret them and find information
that is simply not there. The benchmarks paint an overly simplified
picture, nothing more. The only way to interpret the numbers correctly
is to ignore them and look for more pragmatic examples.

I do think there is some small value in the comparitive coding samples,
but the rigidity of the rules means that there is very little ingenuity
allowed in the code, and thus all the code ends up looking extremely
similar, regardless of the language employed. If you enjoy probing
variations in whitespace and the placement of semi-colons, you will have
lots of fun.

> Faster to develop-in than what other languages - Smalltalk? Lisp?
> Mathematica?

Faster to develop in than C++, Perl, Java, Forth, Assembler, and indeed,
Lisp. God help you if you decide to implement a web framework or a
compiler in Mathematica.

My mistake, I didn't realize Ruby was fixed to Rails, somewhere I got
the impression it was more general than that.

The real problem with the alioth benchmarks is that they are run by
amateurs that allow themselves to be bound by rigid pseudo-academic
dogma, but they can't research and fix trivial problems on their own,
and then they whine when nobody does it for them. If you guys aren't
highly motivated to fix things, why do you think the rest of us will be?

I gave up on them a while ago.

Who's whining? Those who wish to contribute will, those who don't won't.

55

it's 42 right?

-a

···

On Mon, 13 Jun 2005, Curt Hibbs wrote:

Austin Ziegler wrote:

If we're not getting the performance we need, we fix the damn problem.
We don't rely on "benchmarks" -- we rely on real world measurements of
our real problems. Not on pseudo-crap like gigaflops or specmark or
the speed of an airborne swallow. Actually, strike that. The last is
useful.

Yeah, I rely on the speed of airborne swallows at least three times as often as Ackermann functions. :wink:

--

email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
My religion is very simple. My religion is kindness.
--Tenzin Gyatso

===============================================================================

56

Austin Ziegler wrote:
-snip-
> Never *once* have I needed to implement an Ackermann function. Not
> once. In my entire career. I look at the crap that is on alioth and
> there's very little that represents common use. There's some neat
> things -- the new DNA transformation ones -- but exactly how many
> people will actually be using that in their work?
Never once needed to implement a recursive function?

Not an Ackermann recursive. Only simple recursive. Most of the time, I
haven't even needed that. There's an important point there. Indeed, I
have sometimes gone in and changed recursive into iterative because it
was too expensive to implement as recursion.

But what are you really testing with this? If you want to test stack
winding and unwinding speeds, then you're probably testing the wrong
thing here. If you're testing something else, it's not clear. And
that's the damnable thing about the whole alioth shootout -- you
refuse to take an editorial stance anywhere (well, sort of) on the
interpretation of the numbers, leaving them to stand on their own --
which leads to people making stupid assumptions about them. You state
that you refuse to take an editorial stance, but you end up doing so
-- both by what tests you accept for the shootout and by what
implementations you'll accept.

I stand by what I said, though: the alioth shootout is a waste of
everyone's time. Frankly, I think it would be better if it were just
taken down.

Some details are always specific to a problem domain and application,
but the same representations and approaches are used across problem
domains and across applications - yes the details are from DNA sequence
analysis, but the programs process ascii strings and hashtables.

Perhaps. But more often than not, the approaches aren't perfectly portable.

-austin

···

On 6/12/05, Isaac Gouy <igouy@yahoo.com> wrote:
--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

57

Bully for you. Were you there when they mixed unit systems, too? Did
they do *that* with benchmarks?

Sorry, but I don't buy the argument that they did this with
benchmarks. Measurements, perhaps, of the performance of their own
programs (or even prototypes), but *not* with benchmarks. In other
words, I think you're full of it and I'm calling you on it. Provide
some evidence that the decisions were made with benchmarks -- "A
standard by which something can be measured or judged."

Tell me that they did prototyping and I'll believe you. Tell me that
they measured *performance* and I'll believe you. Tell me that
benchmarks were involved in the process and I won't believe you unless
you provide some evidence of this claim.

-austin

···

On 6/12/05, Steven Jenkins <steven.jenkins@ieee.org> wrote:

Austin Ziegler wrote:
> Huh. They didn't do it with benchmarks.
They did. I was there; you weren't. But don't let facts get in the way.

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

58

Hello Michael,

Also, I agree with the fact the optimization comes later in the
dev cycles. But we need to have the doors open to optimize.
Moreover, I read that in Ruby you can always go and code the hot
spots in C. What if you don't have these resources? Which are
the costs to bring such a resource and make him productive?

How does this issue relate to *ruby*? The same charge could be
levied at python, even J2EE. "What happens if you don't have the
resources to speed up something slow?"

If a language is a few times faster then the risk factor that this
happens is just lower. This is a management descision.

Sometimes yes, sometimes no. More often than I would like, but less
often, I imagine, than you think.

There are many things to keep in balance when you start a project.
Development time is just one of them and often not that important.

Actually, in my experience -- both with cusomisable largeware (a
billing system for ISPs) and with consumer-oriented software --
development time is one of the top items of concern. Not *the* top,
but definitely a component in the top (which is usually support
costs, a combination of tech support time, QA time, and developer
time, the cost of each rising).

This is a domain where Ruby is unfortunately not useable at the
moment (missing a good GUI toolkit is the second reason). And i
would like to see it possible to use it there too.

I'd argue that Ruby's speed is secondary to the lack of a good
cross-platform GUI kit. I, too, would like to see Ruby perform
faster than it does. But I don't think that satisfying the alioth
shootout is going to make that happen. The problem *is* known, and
the solutions are at hand. If moving slowly.

-austin

···

On 6/13/05, Lothar Scholz <mailinglists@scriptolutions.com> wrote:

On 6/12/05, Alexandru Popescu <the_mindstorm@evolva.ro> wrote:

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

59

I would agree it would be cool if Ruby was faster, but for most web
applications, performance problems can be solved with adding more
hardware.

When you have to make the trade off between more developer time, ease
of maintainence vs. more hardware, most companies go with the more
hardware approach as it usually is cheaper. People worry about
performance often too much when choosing the language, rather than
focusing on good design and the product itself.

60

hmmm. since set uses hash yet has at least another level of method calls
that's hard to believe... this is what i'm getting:

   harp:~ > ruby a.rb
   ========< set >========
   3.818156

   ========< hash >========
   2.572087

   harp:~ > ruby a.rb
   ========< set >========
   3.824544

   ========< hash >========
   2.600603

   harp:~ > cat a.rb
   require 'set'

   def bench label
     fork {
       GC.disable
       STDOUT.sync = true
       puts "========< #{ label } >========"
       a = Time::now.to_f
       yield
       b = Time::now.to_f
       printf "%f\n", b - a
       puts
     }
     Process::wait
   end

   words = IO::readlines('/usr/share/dict/words').map{|word| word.strip}

   set = Set::new
   hash = Hash::new

   words.each{|word| set << hash[word] = word}

   bench('set'){ 42.times{ words.each{|word| set.include? word}}}
   bench('hash'){ 42.times{ words.each{|word| hash.has_key? word}}}

note that times are the time it takes to look up every single word in the list
42 times.

cheers.

-a

···

On Fri, 10 Jun 2005, Gyoung-Yoon Noh wrote:

Using 'set' is somewhat faster than Hash in 1.9.
I've rewritten that code using 'set' as following:

require 'set'

d = Set.new

File.open('/usr/share/dict/words') do |f|
   d << $_.chop! while f.gets
end

STDIN.each do |l|
   puts l unless d.include? l.chop!
end

Here are timed results in my machine.

--

email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
My religion is very simple. My religion is kindness.
--Tenzin Gyatso

===============================================================================