Need IO Optimization help

ruby-talk
21

Do you think that mmap will get me speeds near cp?

I have the source code for FreeBSD’s cp on my machine. If the file is under
8M it mmap’s the source for reading, and calls write() repeatedly on the
target file until it has been written completely. Otherwise it goes into a
simple read() / write() loop with a buffer of MAXBSIZE, which is 65536

There is a comment which says:

    /*
     * Mmap and write if less than 8M (the limit is so we don't totally
     * trash memory on big files.  This is really a minor hack, but it
     * wins some CPU back.
     */

I guess the main difference is that instead of two syscalls for each 64K of
data, you have one syscall for whatever the maximum size write() will accept
at once.

Thanks for that info. From the above, I don’t think I am going to worry
about it.

If you’re doing things where this is significant, then Ruby is probably the
wrong language for you - unless you write your entire read-process-write
loop in C as a single function, in which case using Ruby as a convenient way
to start it is fine. If it calls ‘yield’ on each line of the input then that
is almost certain to dominate…

Our performance is significant mostly from a human perspective and not from
any real-time processing constraints. We now use Perl for this
processing and it is very hard to beat with just read/write.

I have put the read part in RubyC and we have now surpassed Perl.
Plus, we get the added benefit of a common parser library and
much cleaner code.

Beating Perl was the main impetus here. Not driving the time down
to tens of seconds. If that was important, we would have chosen
assembly. :slight_smile:

It’s the old trade-off between performance, maintainability and
development time. With ruby and a C library to speed up the routine,
we have lowered the cost in each category. A big win in my book.

···

On Thursday, 17 April 2003 at 23:02:08 +0900, Brian Candler wrote:

On Thu, Apr 17, 2003 at 10:11:55PM +0900, David King Landrith wrote:

–
Jim Freeze

You know you’re a little fat if you have stretch marks on your car.
– Cyrus, Chicago Reader 1/22/82

22

In my experience, the fastest way to access files (by far) is
mmap.

Jim, I may be mis-remembering, but wasn’t your original
task/goal/test to compare perl, ruby, (and others) at IO? If so,
isn’t using Ruby+C extensions…cheating, somehow?

···

Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo

23

“Brian Candler” B.Candler@pobox.com schrieb im Newsbeitrag
news:20030411093207.A35304@linnet.org…

Below are the results I get from your benchmarks. I added
the last test which seems to be the fastest, AND is what
I was doing in my tests. To bad perl is 50% faster.

Can this be spead up with a C routine?

I did some profiling of a very similar program, results posted at

http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb?key=-pg&cginame=namazu.rb&s
ubmit=Search&dbname=ruby-talk&max=50&whence=0

Conclusion: I can see no obvious bottlenecks. When you drill down and
work
out how much time is spent in object creation and garbage collection,
file
reading and scanning etc. it seems all pretty well balanced to me. The
reading and breaking-into-individual-lines accounts for 1.09 seconds out
of
2.53 seconds

Even if there are no improvements possible you should judge the effort of
improving speed of ruby vs. perl or others agains the effort to maintain a
changed ruby and the efforts saved by rubys higher development speed. For
some script it’s surely more efficient to write it in one hour and have it
run in another than to tweak a high performance perl script in four hours
that then runs in 15 minutes.

Regards

robert
···

On Fri, Apr 11, 2003 at 03:10:27AM +0900, Jim Freeze wrote:

24

I did some profiles as well and came to the similar conclusions for these
simple tests. I’m not sure that if I dropped to rb_io_readline (or
whatever), that it would go any faster. However, I plan to repeat the
tests with ‘code’ inside the loop to see if Ruby catches up with Perl
when actual work is being done.

···

On Friday, 11 April 2003 at 17:32:17 +0900, Brian Candler wrote:

On Fri, Apr 11, 2003 at 03:10:27AM +0900, Jim Freeze wrote:

Below are the results I get from your benchmarks. I added
the last test which seems to be the fastest, AND is what
I was doing in my tests. To bad perl is 50% faster.

Can this be spead up with a C routine?

I did some profiling of a very similar program, results posted at
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb?key=-pg&cginame=namazu.rb&submit=Search&dbname=ruby-talk&max=50&whence=0

Conclusion: I can see no obvious bottlenecks. When you drill down and work
out how much time is spent in object creation and garbage collection, file
reading and scanning etc. it seems all pretty well balanced to me. The
reading and breaking-into-individual-lines accounts for 1.09 seconds out of
2.53 seconds

–
Jim Freeze

Never put off till tomorrow what you can avoid all together.

25

I agree completely, because I’ve run essentially that program (but using
Ruby’s standard I/O library) through a profiler. It spends around 5% of its
time doing I/O.

The problems are:

  • you want to process the file a line at a time
  • you are allocating a new String object for each line
  • you are calling ‘yield’ on a Ruby block for each line

Regards,

Brian.

···

On Thu, Apr 17, 2003 at 10:34:53PM +0900, ts wrote:

require ‘mmapio’
MMapIO.new(writefile, “w”) { |f|

Good luck, for this

MMapIO.new(readfile).each { |line
# start rockin’
f << line
}
}

Has this been written before I start to do it myself?

yes, but don’t expect that it will be faster …

26

Absolutely. But I liked the idea of “beating Perl” as being the underlying
objective, rather than meeting any other particular criteria :slight_smile:

If you’re still interested in mmap vs read you could try running the
attached program, once using read_normal and once with read_mmap, against
your 260MB file, and take the difference (in seconds, not as a percentage),
which is similar to the kind of difference you could expect (in seconds) in
your main application.

However, once you run this repeatedly and you have enough RAM then your file
is likely to remain in cache anyway, so this may not be representative of
the case where you run the job once (where I/O is likely to dominate).

Calculating a sum of the contents is a simple way to prove that the main
loop actually reads the bytes and hasn’t been optimised away…

Regards,

Brian.

test1.c (982 Bytes)

···

On Thu, Apr 17, 2003 at 11:22:33PM +0900, Jim Freeze wrote:

It’s the old trade-off between performance, maintainability and
development time. With ruby and a C library to speed up the routine,
we have lowered the cost in each category. A big win in my book.

27

Why would this be cheating?

···

On Friday, 18 April 2003 at 3:19:08 +0900, Michael Campbell wrote:

In my experience, the fastest way to access files (by far) is
mmap.

Jim, I may be mis-remembering, but wasn’t your original
task/goal/test to compare perl, ruby, (and others) at IO? If so,
isn’t using Ruby+C extensions…cheating, somehow?

–
Jim Freeze

Nothing is illegal if one hundred businessmen decide to do it.
– Andrew Young

28

For new scripts what you say is true. But I can’t accept the fact
that there are no improvements possible. And Ruby has no advantage
when the Perl scripts already exist and are not being changed.

···

On Friday, 11 April 2003 at 17:47:39 +0900, Robert Klemme wrote:

“Brian Candler” B.Candler@pobox.com schrieb im Newsbeitrag
news:20030411093207.A35304@linnet.org…

On Fri, Apr 11, 2003 at 03:10:27AM +0900, Jim Freeze wrote:

Below are the results I get from your benchmarks. I added

Even if there are no improvements possible you should judge the effort of
improving speed of ruby vs. perl or others agains the effort to maintain a
changed ruby and the efforts saved by rubys higher development speed. For
some script it’s surely more efficient to write it in one hour and have it
run in another than to tweak a high performance perl script in four hours
that then runs in 15 minutes.

–
Jim Freeze

“This is a test of the Emergency Broadcast System. If this had been an
actual emergency, do you really think we’d stick around to tell you?”

29

For a 260MB file, ruby spent 30% of its time in io and
70% processing the lines. (In this test case we were
doing minimal processing.)

So, of that 30% (which was 80 seconds), what would a fair
estimate be of time saved if we swapped mmap for rb_io_gets?

···

On Thursday, 17 April 2003 at 22:46:04 +0900, Brian Candler wrote:

On Thu, Apr 17, 2003 at 10:34:53PM +0900, ts wrote:

require ‘mmapio’
MMapIO.new(writefile, “w”) { |f|

Good luck, for this

MMapIO.new(readfile).each { |line
# start rockin’
f << line
}
}

Has this been written before I start to do it myself?

yes, but don’t expect that it will be faster …

I agree completely, because I’ve run essentially that program (but using
Ruby’s standard I/O library) through a profiler. It spends around 5% of its
time doing I/O.

The problems are:

  • you want to process the file a line at a time
  • you are allocating a new String object for each line
  • you are calling ‘yield’ on a Ruby block for each line

–
Jim Freeze

One difference between a man and a machine is that a machine is quiet
when well oiled.

30

The attached code did not run on Sun for mmap.

I ran it on freebsd and got the following results:

normal
sum -278244398

real 0m1.803s
user 0m1.308s
sys 0m0.476s

mmap
sum -278244398

real 0m1.797s
user 0m1.714s
sys 0m0.063s

-rw-r–r-- 1 jfreeze jfreeze 81499718 Apr 14 22:41 f

···

On Thursday, 17 April 2003 at 23:45:38 +0900, Brian Candler wrote:

–
Jim Freeze

Man is the only animal that can remain on friendly terms with the
victims he intends to eat until he eats them.
– Samuel Butler (1835-1902)

31

Jim Freeze jim@freeze.org writes:

Good luck, for this

MMapIO.new(readfile).each { |line
# start rockin’
f << line
}

Has this been written before I start to do it myself?

yes, but don’t expect that it will be faster …

I agree completely, because I’ve run essentially that program (but
using
Ruby’s standard I/O library) through a profiler. It spends around 5%
of its
time doing I/O.

The problems are:

  • you want to process the file a line at a time
  • you are allocating a new String object for each line
  • you are calling ‘yield’ on a Ruby block for each line

I’ve found that in the real world the speed advantages of using mmap
are more significant than you seem to allow for. (see “My experiments”
below.)

As a guess, I’d say your savings would be minimal. Should you decide
to mmap
the whole 260MB into your address space at once you may actually get
worse
overall performance, unless you have that amount of free RAM available.

This is a real possibility, if not probability. I believe in my
original email, I alluded to this as a disadvantage of using mmap.
However, it will depend very heavily upon how your system implements
mmap, and there are wildly varying implementations. For example, Linux
Kernels before 2.4.20 probably don’t handle files that size well. Mmap
can also have widely varying results on different systems, and with
different functions on the same system. I’m using MacOS X, for
example, and its mmap is happy with strncpy(x, mmapVar, n) but takes a
significant performance hit when you do strncat(x, mmapVar, n); I have
no idea why. I do not remember such a problem with Linux.

My experiments

···

On Thursday, April 17, 2003, at 09:46 AM, Brian Candler wrote:

On Thu, Apr 17, 2003 at 10:34:53PM +0900, ts wrote:

I wrote a function as a Ruby extension (Text_CSV.decode) that uses a
state machine to parse files one byte at a time and convert each line
from a character delimited record to an array. At a high level, it
looks like this: Accumulate bytes for a given field into a c string.
When field terminator is found, convert c string to ruby string and
push it onto a ruby array. When record terminator is found, either
yield the array or push a clone of it onto another array. (There are
also other issues like quotes, escaped characters, and the ability to
filter non-visible ASCII character). Thus, there are a lot of
comparisons, conditionals, and strncpy calls from mmap to the field
accumulator. Moreover, it tests for whether a block is given and
yields a completed row as a ruby array if so.

My early versions buffered IO from the the low level open and close
commands in unistd.h. I switched to mmap because I was frustrated with
the performance. Unfortunately, I don’t have the early code that used
buffered IO because I never bothered to check them in (shame on me). I
ran into the following problems with buffered IO that mmap immediately
solved:

  1. The code that handles the mmap in C is much simpler, because it
    eliminates the need for a buffer. As far as the rest of my code is
    concerned, I’m simply iterating through a char*
  2. I was unable to utilize a buffer of more than 10k (10,240 chars) to
    read data without getting a segfault. (I was allocating it using an
    instant array allocation at the top of my function – I did not try
    using malloc.)
  3. Since I’m analyzing the file byte by byte, using a buffer requires
    testing for the end of the buffer/end of the file after each byte,
    which is more slightly more complicated than simply comparing an
    incremented value to a limit.

While I was developing, I gauged performance using a file with 20,000
lines, 424,250 words, and 3,443,296 characters (which I do still have).
I did both a bulk read (returning an array of arrays of the entire
file) and an iterative read (where each row is yielded to a block as an
array; the block in this case joins the row with a tab.) For a file
this size, there was no noticeable difference. Performance parsing
this file using either read method (iterative or bulk) was as follows:

  1. Using C open/close with file descriptor: approx. 10 seconds to
    process
  2. Using mmap: just over 2 seconds to process

Simple profiling gave the following results:

  1. running ruby -rprofile for bulk read gave me the following:
    % cumulative self self total
    time seconds seconds calls ms/call ms/call name
    71.59 3.78 3.78 1 3780.00 5200.00 Text_CSV.decode
    26.89 5.20 1.42 20000 0.07 0.07 Array#clone
    0.95 5.25 0.05 5 10.00 16.00 Kernel.require
    …
    (Keep in mind that the program runs significantly slower using
    -rprofile, so that the time indicated above do not reflect the real,
    non-profiled execution time)

  2. running ruby -rprofile for iterative read give me the following:
    % cumulative self self total
    time seconds seconds calls ms/call ms/call name
    75.94 4.64 4.64 1 4640.00 6050.00 Text_CSV.decode
    23.08 6.05 1.41 20000 0.07 0.07 Array#join
    0.49 6.08 0.03 5 6.00 12.00 Kernel.require
    …
    (Keep in mind that the program runs significantly slower using
    -rprofile, so that the time indicated above do not reflect the real,
    non-profiled execution time)

Lastly, if my memory serves me correctly, then the Perl Text_CSV module
(which uses stdio.h) is about the same speed as my ruby that used the
unistd.h functions, and so it is much slower than my ruby routine that
uses mmap.

Best,

Dave

David King Landrith
(w) 617.227.4469x213
(h) 617.696.7133

One useless man is a disgrace, two
are called a law firm, and three or more
become a congress – John Adams

public key available upon request

32

As I said, I may not understand what you are trying to do. But if
you’re comparing perl to ruby, compare perl to ruby, not perl to ruby

  • C. If you “win” with ruby + C, could you not do the exact same
    enhancement with perl, and end up where you started?

As a contrived example, if it’s a simple “copy” test, and ‘cp’ is
rippingly fast, then why not just use system(‘cp foo bar’)? I mean,
that’s ruby, right?

···

— Jim Freeze jim@freeze.org wrote:

On Friday, 18 April 2003 at 3:19:08 +0900, Michael Campbell wrote:

In my experience, the fastest way to access files (by far)
is
mmap.

Jim, I may be mis-remembering, but wasn’t your original
task/goal/test to compare perl, ruby, (and others) at IO? If so,
isn’t using Ruby+C extensions…cheating, somehow?

Why would this be cheating?

Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo

33

Even if there are no improvements possible you should judge the effort of
improving speed of ruby vs. perl or others agains the effort to maintain a
changed ruby and the efforts saved by rubys higher development speed. For
some script it’s surely more efficient to write it in one hour and have it
run in another than to tweak a high performance perl script in four hours
that then runs in 15 minutes.

For new scripts what you say is true. But I can’t accept the fact
that there are no improvements possible. And Ruby has no advantage
when the Perl scripts already exist and are not being changed.

I think this really depends on the scripts, what they do and what they’re
used for. If you have to run your script very frequently and depend on its
output, then 15 minutes or an hour running time is really a big
difference. If you use these scripts all the time, then it’s worth
spending a couple extra hours on its implementation/optimization.

Ruben

34

But Ruby has some fundamental disadvantages compared to Perl for this sort
of program. For example,
count += 1
requires a method dispatch, and
gets
requires the creation of a new object each time round the loop (and the
corresponding cleaning up of old objects), whereas when perl assigns to $_
I think it’s just overwriting a global string buffer.

Both these ‘handicaps’ spring from Ruby’s design which makes the writing of
large programs far more elegant, and I suspect when you’re not just chucking
strings and around but doing operations with complex objects, the
performance difference will become less significant.

I suppose you could tweak the language to make it like Perl - for example
when reading lines have a method to overwrite an existing string instead of
allocating a new one:

   while $_.gets!
     .. perhaps
   end

But who wants to write ugly code like that for a small performance gain? You
might as well just use Perl in the first place.

In Perl a scalar is a scalar (except when it’s a reference!!) - in Ruby
everything’s an object. It’s a big win for writing concise code quicky, and
I’m happy to take the performance hit. If it’s “too slow to be useful” at
speed X then it’s probably too slow to be useful at speed 2X either. In that
case you need a fundamental rethink to make it run at speed 10X or 100X.

Or just buy a dual-CPU box :slight_smile:

Regards,

Brian.

···

On Fri, Apr 11, 2003 at 06:47:27PM +0900, Jim Freeze wrote:

On Friday, 11 April 2003 at 17:47:39 +0900, Robert Klemme wrote:

Even if there are no improvements possible you should judge the effort of
improving speed of ruby vs. perl or others agains the effort to maintain a
changed ruby and the efforts saved by rubys higher development speed. For
some script it’s surely more efficient to write it in one hour and have it
run in another than to tweak a high performance perl script in four hours
that then runs in 15 minutes.

For new scripts what you say is true. But I can’t accept the fact
that there are no improvements possible. And Ruby has no advantage
when the Perl scripts already exist and are not being changed.

35

I don’t think that’s a meaningful question, because mmap() doesn’t do the
same thing as rb_io_gets.

I think what you are asking is about rewriting rb_io_gets to do the same
things it currently does (i.e. divide the input into lines, allocate a
string for each one), but using mmap() instead of fread() to access the
file.

Did your program actually spend 30% of its time in fread()? Or was it 30% of
its time in rb_io_gets? It’s an important distinction.

As a guess, I’d say your savings would be minimal. Should you decide to mmap
the whole 260MB into your address space at once you may actually get worse
overall performance, unless you have that amount of free RAM available.

Regards,

Brian.

···

On Thu, Apr 17, 2003 at 10:54:08PM +0900, Jim Freeze wrote:

The problems are:

  • you want to process the file a line at a time
  • you are allocating a new String object for each line
  • you are calling ‘yield’ on a Ruby block for each line

For a 260MB file, ruby spent 30% of its time in io and
70% processing the lines. (In this test case we were
doing minimal processing.)

So, of that 30% (which was 80 seconds), what would a fair
estimate be of time saved if we swapped mmap for rb_io_gets?

36

I’ve found that in the real world the speed advantages of using mmap
are more significant than you seem to allow for. (see “My experiments”
below.)

This is interesting.

When you say your original version used “buffered IO” do you mean it was
using getc() and friends? (i.e. buffered IO at the Unix layer, not a Ruby IO
object)

It’s interesting to note that the gets function in Ruby 1.8.0p2 is quite
different from 1.6.8; where available, it pokes into the FILE* structure to
work out how many characters are available. In 1.6.8 it did individual
getc()'s, each one of which had to interact with threading, which was quite
an overhead. And from what others have posted, that seemed to work really
badly under Windoze.

  1. The code that handles the mmap in C is much simpler, because it
    eliminates the need for a buffer. As far as the rest of my code is
    concerned, I’m simply iterating through a char*

But remember that it is also less general, as you can’t mmap() a fifo, a
character device etc.

  1. I was unable to utilize a buffer of more than 10k (10,240 chars) to
    read data without getting a segfault. (I was allocating it using an
    instant array allocation at the top of my function – I did not try
    using malloc.)

You couldn’t allocate more than 10K on the stack?? I do remember someone
posting a message here a month or too back about the default stack size for
MacOSX applications being extraordinarily small, and I think someone showed
how to raise it. Or just malloc, as you suggested.

  1. Since I’m analyzing the file byte by byte, using a buffer requires
    testing for the end of the buffer/end of the file after each byte,
    which is more slightly more complicated than simply comparing an
    incremented value to a limit.

This is true, although you would have the same overhead if you wanted to
read an enormous file which was too big to mmap in at once, so you had to
mmap it in say 10MB segments.

I believe the main advantage that mmap has over the buffered I/O libraries
is that it doesn’t need actually to copy the data from the vbuf’s to your
application’s buffer.

While I was developing, I gauged performance using a file with 20,000
lines, 424,250 words, and 3,443,296 characters (which I do still have).
I did both a bulk read (returning an array of arrays of the entire
file) and an iterative read (where each row is yielded to a block as an
array; the block in this case joins the row with a tab.) For a file
this size, there was no noticeable difference. Performance parsing
this file using either read method (iterative or bulk) was as follows:

  1. Using C open/close with file descriptor: approx. 10 seconds to
    process
  2. Using mmap: just over 2 seconds to process

Simple profiling gave the following results:

  1. running ruby -rprofile for bulk read gave me the following:
    % cumulative self self total
    time seconds seconds calls ms/call ms/call name
    71.59 3.78 3.78 1 3780.00 5200.00 Text_CSV.decode
    26.89 5.20 1.42 20000 0.07 0.07 Array#clone
    0.95 5.25 0.05 5 10.00 16.00 Kernel.require
    …
    (Keep in mind that the program runs significantly slower using
    -rprofile, so that the time indicated above do not reflect the real,
    non-profiled execution time)

  2. running ruby -rprofile for iterative read give me the following:
    % cumulative self self total
    time seconds seconds calls ms/call ms/call name
    75.94 4.64 4.64 1 4640.00 6050.00 Text_CSV.decode
    23.08 6.05 1.41 20000 0.07 0.07 Array#join
    0.49 6.08 0.03 5 6.00 12.00 Kernel.require
    …
    (Keep in mind that the program runs significantly slower using
    -rprofile, so that the time indicated above do not reflect the real,
    non-profiled execution time)

Ruby’s profiler is a bit dodgy IMO, as I’ve seen it do strange things in the
past, so you might be better running a C profiler instead on the Ruby
interpreter itself - or just use the ‘time’ command,
time ./foo …
although of course this doesn’t factor out how much time is spent in
‘decode’ and how much in the rest of the program.

Clearly you had a faster implementation with mmap. It might however have
been possible to squeeze the difference by tweaking the code which used
standard I/O (with a 64K buffer, say)

Lastly, if my memory serves me correctly, then the Perl Text_CSV module
(which uses stdio.h) is about the same speed as my ruby that used the
unistd.h functions, and so it is much slower than my ruby routine that
uses mmap.

This is all perhaps worth revisiting anyway, as I don’t really like the API
to the CSV functions which are in RAA (but I don’t really have the time to
rewrite it myself :slight_smile: So if you want to release a super-CSV that would be
great…

Cheers,

Brian.

···

On Fri, Apr 18, 2003 at 01:11:07AM +0900, David King Landrith wrote:

37

Michael Campbell wrote:

In my experience, the fastest way to access files (by far)

is

mmap.

Jim, I may be mis-remembering, but wasn’t your original
task/goal/test to compare perl, ruby, (and others) at IO? If so,
isn’t using Ruby+C extensions…cheating, somehow?

Why would this be cheating?

As I said, I may not understand what you are trying to do. But if
you’re comparing perl to ruby, compare perl to ruby, not perl to ruby

  • C. If you “win” with ruby + C, could you not do the exact same
    enhancement with perl, and end up where you started?

[snipt]

Well, for one thing, writing C extensions for ruby is remarkably easy
(not sure how perl compares in that regard). And, for me, at least, the
ability to painlessly add low-level extensions when necessary translates
directly into a real-world performance benefit.

Of course, for benchmarking purposes, if you’re writing C extensions in
ruby, it’s only fair to do the same in perl… and compare the final
results in terms of runtime, and programmer time… :wink:

···

— Jim Freeze jim@freeze.org wrote:

On Friday, 18 April 2003 at 3:19:08 +0900, Michael Campbell wrote:

38

We use Perl in a business environment. My quest is not an academic
contest between Perl and Ruby. Perl is the incumbent language
used to write filters for processing large files. The code is
very ‘hackish’ and not easy to read or maintain. I would like
to replace it with a much more readable language, ie, Ruby.
But, if Ruby is not fast enough, then it will never be
considered. If it is close in speed, then it could be considered
because of it’s readability, but it will still leave an aftertaste
with some because it is not a clear winner.

However, if Ruby is faster and more readable and easier to maintain,
then it is a true win. Forward progress with Perl will have to be
justified, with much effort.

If we write a library in Ruby or extend it with C, it doesn’t
matter. Both are easy to write (unlike Perl). The user of the
library stays in a Ruby environment.

So, my comparison is with between what I can do with each language
with similar effort. If one language let’s me optimize and publish
common libraries with ease, then that’s just an added advantage
that that language possesses. If we could do the same with Perl,
we would. We are not artificially constraining it in any of these
tests. (We have several perl experts at our company that have
used the language for years and like it very much. Yet, none of them
know how to extend the language with C for performance. Either
it was not needed, or it is not something a Perl programmer
typically does.)

···

On Friday, 18 April 2003 at 9:19:25 +0900, Michael Campbell wrote:

— Jim Freeze jim@freeze.org wrote:

On Friday, 18 April 2003 at 3:19:08 +0900, Michael Campbell wrote:

In my experience, the fastest way to access files (by far)
is
mmap.

Jim, I may be mis-remembering, but wasn’t your original
task/goal/test to compare perl, ruby, (and others) at IO? If so,
isn’t using Ruby+C extensions…cheating, somehow?

Why would this be cheating?

As I said, I may not understand what you are trying to do. But if
you’re comparing perl to ruby, compare perl to ruby, not perl to ruby

  • C. If you “win” with ruby + C, could you not do the exact same
    enhancement with perl, and end up where you started?

As a contrived example, if it’s a simple “copy” test, and ‘cp’ is
rippingly fast, then why not just use system(‘cp foo bar’)? I mean,
that’s ruby, right?

–
Jim Freeze

A professor is one who talks in someone else’s sleep.

39

I’ve been thinking about this a bit since the last time the “++”
debate came up; the primary use for “++” is “count += 1”, at least
in Ruby and Perl; might there be a call for a “Counter” class that
performs a similar purpose, but keeps the counting internal until
#value is called?

I’m not in a position to be able to write an extension that does
this at this point, but I’m wondering if it would be faster than
using Fixnum#+ and object assignment.

-austin
– Austin Ziegler, austin@halostatue.ca on 2003.04.11 at 12:28:59

···

On Fri, 11 Apr 2003 19:59:28 +0900, Brian Candler wrote:

But Ruby has some fundamental disadvantages compared to Perl for
this sort of program. For example, count += 1 requires a method
dispatch,

40

In article 20030411115918.A35958@linnet.org,

Even if there are no improvements possible you should judge the effort of
improving speed of ruby vs. perl or others agains the effort to maintain a
changed ruby and the efforts saved by rubys higher development speed. For
some script it’s surely more efficient to write it in one hour and have it
run in another than to tweak a high performance perl script in four hours
that then runs in 15 minutes.

For new scripts what you say is true. But I can’t accept the fact
that there are no improvements possible. And Ruby has no advantage
when the Perl scripts already exist and are not being changed.

But Ruby has some fundamental disadvantages compared to Perl for this sort
of program. For example,
count += 1
requires a method dispatch, and
gets
requires the creation of a new object each time round the loop (and the
corresponding cleaning up of old objects), whereas when perl assigns to $_
I think it’s just overwriting a global string buffer.

Both these ‘handicaps’ spring from Ruby’s design which makes the writing of
large programs far more elegant, and I suspect when you’re not just chucking
strings and around but doing operations with complex objects, the
performance difference will become less significant.

I suppose you could tweak the language to make it like Perl - for example
when reading lines have a method to overwrite an existing string instead of
allocating a new one:

  while $_.gets!
    .. perhaps
  end

But who wants to write ugly code like that for a small performance gain? You
might as well just use Perl in the first place.

Well, if you’d prefer to use Ruby then putting up with a couple of lines
of ugliness so you could benefit from using Ruby for the rest of the
program might be worth it. This idea (to overwrite an existing string)
might very well be a good way to acheive the speed-up that Jim is looking
for. Perhaps a C extension which does this would be a big win - call it
FastIO or something.

You have to remember that he mentioned the need to read in files
which are several hundred MB in size - any (even small) performance gain
will help save a lot of time when it comes to reading large files.

In Perl a scalar is a scalar (except when it’s a reference!!) - in Ruby
everything’s an object. It’s a big win for writing concise code quicky, and
I’m happy to take the performance hit. If it’s “too slow to be useful” at
speed X then it’s probably too slow to be useful at speed 2X either. In that
case you need a fundamental rethink to make it run at speed 10X or 100X.

However, when it comes to comparing OO Perl speed and Ruby, method
dispatch in Ruby is faster.

Or just buy a dual-CPU box :slight_smile:

Wouldn’t help in this case.

Phil

···

Brian Candler B.Candler@pobox.com wrote:

On Fri, Apr 11, 2003 at 06:47:27PM +0900, Jim Freeze wrote:

On Friday, 11 April 2003 at 17:47:39 +0900, Robert Klemme wrote: