Ruby in Performance Testing

ruby-talk
1

Rubyists,

I’m trying to setup a performance/load test to bombard a server with
a ton of requests. The idea is to generate as many parallel requests
as possible to really stress the server. However my code already bombs
out at around 300 threads with:

naft_s.rb:140: [BUG] rb_sys_fail() - errno == 0
ruby 1.7.3 (2002-11-17) [i386-mswin32]

I also get “Bad file descriptor” errors at times.

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

The stripped code making 500 parallel requests; each in turn doing 10
sequential requests.

#!/usr/bin/ruby -w

require 'socket’
require ‘thread’

threads = []

(1…500).each do |c|
threads << Thread.new© do |client|
(1…10).each do |request|
begin
TCPSocket::open(‘localhost’, 80) { |s|
# perform some socket reading writing here
# and print some output
}
rescue Exception => e
$stderr.puts e
end
end
end
end

threads.each { |t|
t.join
}

Emiel

···

–
E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

2

I know this is a Ruby list, but if you use Apache it comes bundled with a
program called ‘ab’ (Apache Benchmark) which does load testing with any
number of concurrent requests. You may wish to take a look at it.

Tim Bates

···

On Wed, 26 Feb 2003 6:57 pm, E F van de Laar wrote:

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

–
tim@bates.id.au

3

From what you write I conclude that the server creates a thread for each
request. This might indicate a design problem in the server, because the
overhead of creating threads is often significant. Typical server
implementations use a thread pool with a fixed or at least limited amount
of threads that handle all requests, which they receive from an event
queue.

Kind regards

robert

“E F van de Laar” emiel@il.fontys.nl schrieb im Newsbeitrag
news:20030226082715.GA529@il.fontys.nl…

···

Rubyists,

I’m trying to setup a performance/load test to bombard a server with
a ton of requests. The idea is to generate as many parallel requests
as possible to really stress the server. However my code already bombs
out at around 300 threads with:

naft_s.rb:140: [BUG] rb_sys_fail() - errno == 0
ruby 1.7.3 (2002-11-17) [i386-mswin32]

I also get “Bad file descriptor” errors at times.

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

The stripped code making 500 parallel requests; each in turn doing 10
sequential requests.

#!/usr/bin/ruby -w

require 'socket’
require ‘thread’

threads = []

(1…500).each do |c|
threads << Thread.new© do |client|
(1…10).each do |request|
begin
TCPSocket::open(‘localhost’, 80) { |s|
# perform some socket reading writing here
# and print some output
}
rescue Exception => e
$stderr.puts e
end
end
end
end

threads.each { |t|
t.join
}

Emiel

E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

4

Rubyists,

I’m trying to setup a performance/load test to bombard a server with
a ton of requests. The idea is to generate as many parallel requests
as possible to really stress the server. However my code already bombs
out at around 300 threads with:

naft_s.rb:140: [BUG] rb_sys_fail() - errno == 0
ruby 1.7.3 (2002-11-17) [i386-mswin32]

Is this the error message from the server, or the client? From the high
source line number, I can only assume the error is showing up on the
server, in which case it’s difficult to determine exactly what might be
causing the problem without the source. However, I’ll assume that the
basic issue is symmetrical – that is, both the client and the server
begin to show the same symtoms after a high number of concurrent
connections are open.

I also get “Bad file descriptor” errors at times.

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

I’m not sure how one would fix the problem under Windows, but I know
that under most Linux and BSD systems, there’s a kernel-controlled
limit on the number of file descriptors a single process can have open at
any one time. Either a kernel recompile or a simple configuration option
usually fixes this on the open source OSes, but obviously, under Windows,
you probably won’t have that option.

[code deleted]

If the problem is indeed a hard limit to the number of file descriptors
one Windows process can have open, then you’ll need to fork several
’slave’ processes, each of which handles a number of connections
lower than this limit. However, you definitely don’t want to fork a new
process for each connection, as the 300+ simultaneous execution
requests would almost certainly bring the system to its knees for a
while, if not thrash the disk enough swapping to seriously risk your
filesystem integrity.

In general, though, I highly recommend the articles linked at the bottom
of the following page for general high-load server tuning ideas:

http://www.nightmare.com/medusa/index.html

The author of the above page also wrote the Medusa asynchronous
networking library for Python, but there are a number of excellent
articles about building servers capable of handling rediculous numbers
(i.e., thousands or tens of thousands) of concurrent connections. Some
of the content is a bit dated, but the basic techniques are all very sound.
(The short summary, for the impatient: the ‘select()’ method is your
friend and helper, esp. with continuations and/or lightweight threading.)

Emiel

E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

Lennon Day-Reynolds
lennon@day-reynolds.com

···

On Wed, 26 Feb 2003 17:27:17 +0900 E F van de Laar emiel@il.fontys.nl wrote:

5
> I also get "Bad file descriptor" errors at times.

#!/usr/bin/ruby -w

require 'socket’
require ‘thread’

threads = []

(1…500).each do |c|
threads << Thread.new© do |client|
(1…10).each do |request|
begin
TCPSocket::open(‘localhost’, 80) { |s|
# perform some socket reading writing here
# and print some output
^^^^^^^^^^^^^^^^^
}
rescue Exception => e
$stderr.puts e
^^^^^^^^^^^^^^
end
end
end
end

threads.each { |t|
t.join
}

i’m not sure, but accessing stdout/stderr from this many threads at once may
be a problem - i’ve had problems with it before. one thing i’ve done in my
threaded problems is something like this :

#!/usr/bin/env ruby
require ‘thread’

tds = []
toutput = []
sem = Mutex.new

4.times do |tid|
toutput[tid] = []
tds <<
Thread.new (tid) do
2.times do |i|
Thread.pass while not sem.try_lock
toutput[tid] << "msg #{i} from thread #{tid}…"
sem.unlock
end
end
end

toutput.each do |msgs|
msgs.each do |msg|
puts msg
end
puts
end

which would output :

msg 0 from thread 0…
msg 1 from thread 0…

msg 0 from thread 1…
msg 1 from thread 1…

msg 0 from thread 2…
msg 1 from thread 2…

msg 0 from thread 3…
msg 1 from thread 3…

this technique alone has helped my concurrent programming skiils since i can
read the bloody output!

-a

···

On Wed, 26 Feb 2003, E F van de Laar wrote:

Emiel

E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

–

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ahoward@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

6

I’m aware of “ab” but this is a custom tcp server. Not a web server.
Requests are made up of a custom format. Though it might not be a bad
idea to modify “ab” or “siege” to do this for me.

Thanks,

Emiel

···

On Wed, 26 Feb 2003 6:57 pm, E F van de Laar wrote:

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

I know this is a Ruby list, but if you use Apache it comes bundled with a
program called ‘ab’ (Apache Benchmark) which does load testing with any
number of concurrent requests. You may wish to take a look at it.

–
E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

7

Rubyists,

I’m trying to setup a performance/load test to bombard a server with
a ton of requests. The idea is to generate as many parallel requests
as possible to really stress the server. However my code already bombs
out at around 300 threads with:

naft_s.rb:140: [BUG] rb_sys_fail() - errno == 0
ruby 1.7.3 (2002-11-17) [i386-mswin32]

Is this the error message from the server, or the client? From the high
source line number, I can only assume the error is showing up on the
server, in which case it’s difficult to determine exactly what might be
causing the problem without the source. However, I’ll assume that the
basic issue is symmetrical – that is, both the client and the server
begin to show the same symtoms after a high number of concurrent
connections are open.

Error occurs in the client side. The server is a java implementation
that I have very little control over. It does use a pool of native
threads however.

I also get “Bad file descriptor” errors at times.

Does anyone have any other approaches for this kind of test? Is ruby
not up to this sort of task? Would fork be a better approach?
How do I raise this limit? Any tips are welcome.

I’m not sure how one would fix the problem under Windows, but I know
that under most Linux and BSD systems, there’s a kernel-controlled
limit on the number of file descriptors a single process can have open at
any one time. Either a kernel recompile or a simple configuration option
usually fixes this on the open source OSes, but obviously, under Windows,
you probably won’t have that option.

I’ll move to a UNIX machine and explore some of theses limits. Your
link has opened a new world for me and I’m already understanding
my problem a lot better.

[code deleted]

If the problem is indeed a hard limit to the number of file descriptors
one Windows process can have open, then you’ll need to fork several
’slave’ processes, each of which handles a number of connections
lower than this limit. However, you definitely don’t want to fork a new
process for each connection, as the 300+ simultaneous execution
requests would almost certainly bring the system to its knees for a
while, if not thrash the disk enough swapping to seriously risk your
filesystem integrity.

I’m going to try this soon. My parallel programming skills are just
now starting to develop. :slight_smile:

In general, though, I highly recommend the articles linked at the bottom
of the following page for general high-load server tuning ideas:

http://www.nightmare.com/medusa/index.html

The author of the above page also wrote the Medusa asynchronous
networking library for Python, but there are a number of excellent
articles about building servers capable of handling rediculous numbers
(i.e., thousands or tens of thousands) of concurrent connections. Some
of the content is a bit dated, but the basic techniques are all very sound.
(The short summary, for the impatient: the ‘select()’ method is your
friend and helper, esp. with continuations and/or lightweight threading.)

Thanks for the info.

Emiel

···

On Wed, 26 Feb 2003 17:27:17 +0900 > E F van de Laar emiel@il.fontys.nl wrote:
–
E F van de Laar
+31648183479
www.il.fontys.nl/~emiel

8
> I also get "Bad file descriptor" errors at times.

i’m not sure, but accessing stdout/stderr from this many threads at once may
be a problem - i’ve had problems with it before. one thing i’ve done in my
threaded problems is something like this :

#!/usr/bin/env ruby
require ‘thread’

tds = []
toutput = []
sem = Mutex.new

4.times do |tid|
toutput[tid] = []
tds <<
Thread.new (tid) do
2.times do |i|
Thread.pass while not sem.try_lock
toutput[tid] << "msg #{i} from thread #{tid}…"
sem.unlock

sem.synchronize do
toutput[tid] << "msg #{i} from thread #{tid}…"
end

I feel Thread.* are sometimes too low-level.
[snip]

this technique alone has helped my concurrent programming skiils since i can
read the bloody output!

This makes me think of one doubt I’ve had for some time:
what is the granularity of thread-switching?
or
What can I assume to happen atomically?

Obviously calls to methods implemented in C and such must be atomic
since Ruby’s using green threads.

However, I’ve found in my code things like this:
(complete example at
http://www.rubygarden.org/ruby?ObjectPoolingAndThreading)

def run(debug = false)
ref = []
handler = nil
@mutex.synchronize do
info = false
if @pool.size == 0
@handleravail.wait @mutex
info = true if debug
end
handler = @pool.shift
puts “Got handler #{handler} after getting blocked.” if info
@threads[handler] = ref # hack cause want this
# inside synchronize; perhaps not needed
end
ref << Thread.new do
begin
yield handler
ensure
@mutex.synchronize do
@pool << handler
@threads.delete handler
@handleravail.signal
end
end
end
ref[0]
end

The kludge with ‘ref’ is to ensure that the state of @threads (which is
a hash) is consistent.
What I really want to do is
@threads[handler] << Thread.new { }
but if that section is not protected with a Mutex I don’t feel safe.
Probably being too cautious and thinking too much as if we had were native
threads. But it might pay off if someday Ruby has real threading…

···

On Thu, Feb 27, 2003 at 12:15:37AM +0900, ahoward wrote:

On Wed, 26 Feb 2003, E F van de Laar wrote:

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

*** PUBLIC flooding detected from erikyyy
THAT’s an erik, pholx… :wink:
– Seen on #LinuxGER

9

“ahoward” ahoward@fsl.noaa.gov schrieb im Newsbeitrag
news:Pine.LNX.4.53.0302261440230.14239@eli.fsl.noaa.gov…

i’m not sure, but accessing stdout/stderr from this many threads at once
may
be a problem - i’ve had problems with it before. one thing i’ve done in
my
threaded problems is something like this :

In order to get a grip on ruby threading I implemented a similar version
that does also write the output along the way. Feel free to use and modify
it as you like.

robert

require ‘thread’

class Queue
def initialize()
@q = []
@mutex = Mutex.new
@cond = ConditionVariable.new
end

def enqueue(*elems)
@mutex.synchronize do
@q.push *elems
@cond.signal
end
end

def dequeue()
@mutex.synchronize do
while @q.empty? do
@cond.wait(@mutex)
end

  return @q.shift
end

end
end

class Logger
def initialize(io, close=false)
@closeOnTerminate = close
@queue = Queue.new
@io = io

@t = Thread.new do
  while true do
    @io.puts(@queue.dequeue)
  end
end

end

def log(*args)
checkRunning
@queue.enqueue(*args)
end

def terminate()
checkRunning

@t.exit
@t = nil

@io.close if @closeOnTerminate
@io = nil

@queue = nil

end

def checkRunning
raise “Logger already terminated” unless @t
end
end

l = Logger.new($stdout)

l.log(“hello”)
l.log(“this is it”)
l.log(%w{this is it})

l.terminate

sleep(5)

10

In order to get a grip on ruby threading I implemented a similar version
that does also write the output along the way. Feel free to use and modify
it as you like.

Thread logging would be a good addition to the thread module IMHO… something
similar to what you’ve written…

-a

···

On Wed, 26 Feb 2003, Robert Klemme wrote:

robert

require ‘thread’

class Queue
def initialize()
@q = []
@mutex = Mutex.new
@cond = ConditionVariable.new
end

def enqueue(*elems)
@mutex.synchronize do
@q.push *elems
@cond.signal
end
end

def dequeue()
@mutex.synchronize do
while @q.empty? do
@cond.wait(@mutex)
end

  return @q.shift
end

end
end

class Logger
def initialize(io, close=false)
@closeOnTerminate = close
@queue = Queue.new
@io = io

@t = Thread.new do
  while true do
    @io.puts(@queue.dequeue)
  end
end

end

def log(*args)
checkRunning
@queue.enqueue(*args)
end

def terminate()
checkRunning

@t.exit
@t = nil

@io.close if @closeOnTerminate
@io = nil

@queue = nil

end

def checkRunning
raise “Logger already terminated” unless @t
end
end

l = Logger.new($stdout)

l.log(“hello”)
l.log(“this is it”)
l.log(%w{this is it})

l.terminate

sleep(5)

–

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ahoward@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

11

“ahoward” ahoward@fsl.noaa.gov schrieb im Newsbeitrag
news:Pine.LNX.4.53.0302261440230.14239@eli.fsl.noaa.gov…

i’m not sure, but accessing stdout/stderr from this many threads at once
may
be a problem - i’ve had problems with it before. one thing i’ve done in
my
threaded problems is something like this :

In order to get a grip on ruby threading I implemented a similar version
that does also write the output along the way. Feel free to use and modify
it as you like.

robert

[snip]

This is nice, could it get into the Wiki?

···

On Thu, Feb 27, 2003 at 02:36:03AM +0900, Robert Klemme wrote:

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Save yourself from the ‘Gates’ of hell, use Linux." – like that one.
– The_Kind @ LinuxNet

12

done: http://www.rubygarden.org/ruby?MultiThreading

(I’m not very familiar with the categoryzing scheme, so please feel free to
change it.)

robert

“Mauricio Fernández” batsman.geo@yahoo.com schrieb im Newsbeitrag
news:20030226221941.GD23049@student.ei.uni-stuttgart.de…

“ahoward” ahoward@fsl.noaa.gov schrieb im Newsbeitrag
news:Pine.LNX.4.53.0302261440230.14239@eli.fsl.noaa.gov…

i’m not sure, but accessing stdout/stderr from this many threads at
once

may

be a problem - i’ve had problems with it before. one thing i’ve done
in

my

threaded problems is something like this :

In order to get a grip on ruby threading I implemented a similar version
that does also write the output along the way. Feel free to use and
modify

···

On Thu, Feb 27, 2003 at 02:36:03AM +0900, Robert Klemme wrote:

it as you like.

robert

[snip]

This is nice, could it get into the Wiki?

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Save yourself from the ‘Gates’ of hell, use Linux." – like that one.
– The_Kind @ LinuxNet

13

DISCLAIMER = <<EOF
sending to the list as direct mailing failed w/
<<< 550 {mx005-rz3} The recipient does not accept mails from 'yahoo.com’
over foreign mailservers
550 bob.news@gmx.net… User unknown
EOF

Thanks
Hopefully Gavin Sinclair (who introduced the Category concept in this
Wiki) will check it :wink:

···

On Thu, Feb 27, 2003 at 05:18:35PM +0900, Robert wrote:

done: http://www.rubygarden.org/ruby?MultiThreading

(I’m not very familiar with the categoryzing scheme, so please feel free to
change it.)

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

  • Linux Viruscan…
    Windows 95 found. Remove it? (Y/y)
    – Unknown source
14

I removed the “tutorial” categorisation because the English-Ruby ratio
on the page is about 2-3%, which doesn’t seem like a tutorial to me!

The bosy of sample code on the wiki
[http://www.rubygarden.org/ruby?search=CategorySampleCode] is looking
good.

Cheers,
Gavin

···

On Thursday, February 27, 2003, 7:40:43 PM, Mauricio wrote:

(I’m not very familiar with the categoryzing scheme, so please feel free to
change it.)

Thanks
Hopefully Gavin Sinclair (who introduced the Category concept in this
Wiki) will check it :wink:

15

You can always take that categorization as an invitation to add some
documentation in English :wink:

···

On Mon, Mar 03, 2003 at 01:44:43AM +0900, Gavin Sinclair wrote:

On Thursday, February 27, 2003, 7:40:43 PM, Mauricio wrote:

(I’m not very familiar with the categoryzing scheme, so please feel free to
change it.)

Thanks
Hopefully Gavin Sinclair (who introduced the Category concept in this
Wiki) will check it :wink:

I removed the “tutorial” categorisation because the English-Ruby ratio
on the page is about 2-3%, which doesn’t seem like a tutorial to me!

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

I’m telling you that the kernel is stable not because it’s a kernel,
but because I refuse to listen to arguments like this.
– Linus Torvalds

16

“Gavin Sinclair” gsinclair@soyabean.com.au schrieb im Newsbeitrag
news:107438575057.20030303034338@soyabean.com.au…

(I’m not very familiar with the categoryzing scheme, so please feel
free to

change it.)

Thanks
Hopefully Gavin Sinclair (who introduced the Category concept in this
Wiki) will check it :wink:

I removed the “tutorial” categorisation because the English-Ruby ratio
on the page is about 2-3%, which doesn’t seem like a tutorial to me!

True. Thanks for correcting that. Meanwhile I found time to add some
explanatory text. I don’t think this is a tutorial yet. If I find some
more spare time I will continue adding a bit here and there. (Iterative
tutorial develoment…)

The bosy of sample code on the wiki

“bosy” - what’s that? “amount”, “quality”… ?

[http://www.rubygarden.org/ruby?search=CategorySampleCode] is looking
good.

Regards

robert
···

On Thursday, February 27, 2003, 7:40:43 PM, Mauricio wrote: