Ruby replacement for ab?

ruby-talk
1

So I've been hunting for a way to stress test a web application and haven't been terribly satisfied so far.

I'd really love something where I could define a script that determined the way an agent would behave. Basically a set of requests where each request had the potential to use information contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting ruby's benchmark.rb, mechanize, maybe drb to manage multiple test hosts.... Some sort of ruby DSL would be sweet. But before I get too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions on HTTP requests as part of an agent script, but I don't see any facility to do computations on the returned content from a previous request.

Thoughts? Suggestions?

Thanks in advance,
Mat

2

Did you see, : http://rfuzz.rubyforge.org/

···

On 12/22/06, Mat Schaffer <schapht@gmail.com> wrote:

So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts? Suggestions?

Thanks in advance,
Mat

--
gnufied
-----------
There was only one Road; that it was like a great river: its springs
were at every doorstep, and every path was its tributary.

3

So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts? Suggestions?

I'm not really maintaining it at this point, but RWB might help. it's
available as a gem. I've depracated it in favor of RFuzz, but RFuzz
still hasn't caught all the features.

···

On 12/21/06, Mat Schaffer <schapht@gmail.com> wrote:

Thanks in advance,
Mat

--
thanks,
-pate
-------------------------

4

rfuzz is nice but isn't really what he's asking about.

What he is asking about is something that I have wanted for a long time. I want to be able to quickly and easily setup a test script for a site.

Go to a URL. Follow a link on the page returned. Maybe pick randomly from a few choices of links to follow. Repeat a few times to simulate people clicking around a site. Go to a page with a form, and fill in form fields, etc...

Basically, a DSL to write scripts for driving interactions to a site or application that would somewhat simulate what real traffic is expected to look like.

That would be dandy.

Kirk Haines

···

On Fri, 22 Dec 2006, hemant wrote:

On 12/22/06, Mat Schaffer <schapht@gmail.com> wrote:

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Did you see, : http://rfuzz.rubyforge.org/

5

Both RFuzz and RWB look like really good starts toward what I'm after.
Tough to say if I'll decide to really code something at this point.
But I'll certainly start browsing the source of both projects to get
some ideas.

I'd love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.

But given the state of things, it'd be a big problem to tackle.
Possibly too big for me at this point.

Thanks for the feedback. This list never disappoints!
-Mat

···

On 12/21/06, pat eyler <pat.eyler@gmail.com> wrote:

I'm not really maintaining it at this point, but RWB might help. it's
available as a gem. I've depracated it in favor of RFuzz, but RFuzz
still hasn't caught all the features.

6

It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.

-Fred

···

On 12/21/06, khaines@enigo.com <khaines@enigo.com> wrote:

On Fri, 22 Dec 2006, hemant wrote:

> On 12/22/06, Mat Schaffer <schapht@gmail.com> wrote:
>>
>> I'd really love something where I could define a script that
>> determined the way an agent would behave. Basically a set of
>> requests where each request had the potential to use information
>> contained in the return of a previous request.
>>
>> My brain is slowly churning around the possibilities of putting
>> ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
>> hosts.... Some sort of ruby DSL would be sweet. But before I get
>> too ahead of myself: Does anyone here know of a tool does this?
>>
>> I'm looking at JMeter right now. It at least allows for assertions
>> on HTTP requests as part of an agent script, but I don't see any
>> facility to do computations on the returned content from a previous
>> request.
>>
>
> Did you see, : http://rfuzz.rubyforge.org/

rfuzz is nice but isn't really what he's asking about.

What he is asking about is something that I have wanted for a long time.
I want to be able to quickly and easily setup a test script for a site.

Go to a URL. Follow a link on the page returned. Maybe pick randomly
from a few choices of links to follow. Repeat a few times to simulate
people clicking around a site. Go to a page with a form, and fill in form
fields, etc...

Basically, a DSL to write scripts for driving interactions to a site or
application that would somewhat simulate what real traffic is expected to
look like.

That would be dandy.

Kirk Haines

7

> I'm not really maintaining it at this point, but RWB might help. it's
> available as a gem. I've depracated it in favor of RFuzz, but RFuzz
> still hasn't caught all the features.

Both RFuzz and RWB look like really good starts toward what I'm after.
Tough to say if I'll decide to really code something at this point.
But I'll certainly start browsing the source of both projects to get
some ideas.

If you're planning on writing something, either coordinate with Zed
to work on RFuzz, or take over RWB and go for it.

I'd love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.

That's sort of where I was headed with RWB. It's already multithreaded,
but it really should use dRb to coordinate across multiple nodes. I
don't think there's too much code needed to do that.

But given the state of things, it'd be a big problem to tackle.
Possibly too big for me at this point.

If you decide to go with RWB, I'd be happy to answer any questions.
It might be a plateful, but you can take small bites and it won't be
that hard. (Same is true of RFuzz, except that I can't answer the
questions.)

···

On 12/21/06, Mat Schaffer <schapht@gmail.com> wrote:

On 12/21/06, pat eyler <pat.eyler@gmail.com> wrote:

Thanks for the feedback. This list never disappoints!
-Mat

--
thanks,
-pate
-------------------------

8

Do you know Watir?

-- fxn

···

On Dec 21, 2006, at 11:51 PM, khaines@enigo.com wrote:

Go to a URL. Follow a link on the page returned. Maybe pick randomly from a few choices of links to follow. Repeat a few times to simulate people clicking around a site. Go to a page with a form, and fill in form fields, etc...

Basically, a DSL to write scripts for driving interactions to a site or application that would somewhat simulate what real traffic is expected to look like.

9

Yeah, possibly. Mechanize or hpricot could be part of a solution, but the real challenge is in devising a DSL that makes it _easy_ for me to, for example:

1) Hit the homepage.

2) Go to a random URL on the site.
    Repeat 2-5 times.

3) Go to the fund information page.

4 Click through to the distributions display.

5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information, etc....

Then, if I can run that on two or three machines at the same time, and can collate their logs and run an analysis on the collated logs, I'd be a happy fellow.

Another thing that I'd love to be able to do is to run something like the above set of actions, but do it more slowly, with pauses between each request, so that the site may only get, on average, one request a second but I can let it run for days.

I have encountered failures on occasion that didn't seem to happen as a result of short term heavy loading, but more as a result of steady hits over a longer period of time.

I'd like to be able to easily simulate that usage pattern in a development environment.

Kirk Haines

···

On Fri, 22 Dec 2006, Fred Wulff wrote:

It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.

10

Watir (http://wtr.rubyforge.org/\) and FireWatir (which uses jSSh) look
promising. Those along with the tools mentioned earlier might be a start.

Nate

···

On 12/21/06, khaines@enigo.com <khaines@enigo.com> wrote:

Yeah, possibly. Mechanize or hpricot could be part of a solution, but the
real challenge is in devising a DSL that makes it _easy_ for me to, for
example:

1) Hit the homepage.

2) Go to a random URL on the site.
    Repeat 2-5 times.

3) Go to the fund information page.

4 Click through to the distributions display.

5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information,
etc....

Then, if I can run that on two or three machines at the same time, and can
collate their logs and run an analysis on the collated logs, I'd be a
happy fellow.

Another thing that I'd love to be able to do is to run something like the
above set of actions, but do it more slowly, with pauses between each
request, so that the site may only get, on average, one request a second
but I can let it run for days.

I have encountered failures on occasion that didn't seem to happen as a
result of short term heavy loading, but more as a result of steady hits
over a longer period of time.

I'd like to be able to easily simulate that usage pattern in a development
environment.

Kirk Haines

11

Those actually drive a real browser session, though. That's not really what I need.

Kirk Haines

···

On Fri, 22 Dec 2006, Nathan Smith wrote:

Watir (http://wtr.rubyforge.org/\) and FireWatir (which uses jSSh) look
promising. Those along with the tools mentioned earlier might be a start.

12

Um, then what DO you need that Rails unit and integration testing can't
handle? Any other type of testing *requires* a browser instance.

Jason

···

On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:

On Fri, 22 Dec 2006, Nathan Smith wrote:

> Watir (http://wtr.rubyforge.org/\) and FireWatir (which uses jSSh) look
> promising. Those along with the tools mentioned earlier might be a
start.

Those actually drive a real browser session, though. That's not really
what I need.

Kirk Haines

13

For a non-Ruby example of a scriptable test system: WebTest:
<http://webtest.canoo.com/&gt;

There's also a Firefox plugin to capture actions to use as the basis
for a test script: <http://webtestrecorder.canoo.com/&gt;

Once your script is done, you run it from the command line, or from
your build environment.

FWIW!

···

On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:

Those actually drive a real browser session, though. That's not really
what I need.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com

14

I can't answer Kirk's question, but personally I'm looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.
-Mat

···

On 12/22/06, Jason Roelofs <jameskilton@gmail.com> wrote:

On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:
>
> On Fri, 22 Dec 2006, Nathan Smith wrote:
>
> > Watir (http://wtr.rubyforge.org/\) and FireWatir (which uses jSSh) look
> > promising. Those along with the tools mentioned earlier might be a
> start.
>
> Those actually drive a real browser session, though. That's not really
> what I need.
>
> Kirk Haines
>

Um, then what DO you need that Rails unit and integration testing can't
handle? Any other type of testing *requires* a browser instance.

Jason

15

>
> Those actually drive a real browser session, though. That's not really
> what I need.

Um, then what DO you need that Rails unit and integration testing can't
handle? Any other type of testing *requires* a browser instance.

See below. Rails unit testing is irrelevant. I don't use Rails. It's inadequate for my needs.

I can't answer Kirk's question, but personally I'm looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.

What I would like, and what Mat wants, I think, have a great deal of overlap.

To repeat some of what I said in an earlier post, I'd like to.

1) Hit the homepage of a site.
2) Go to a random URL on the site.
    Repeat 2-5 times.
3) Go to the fund information page.
4 Click through to the distributions display.
5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information, etc....

Or, even more ambitiously:

1) Hit the homepage of a site.
2) Follow the link to the advisor login page.
3) Login (an action that requires sending a form response, and receiving a cookie in response that will be sent on further requests).
4) View fund information for some random funds.
5) Go into advisor tools and feed some numbers into a calculator.
6) Goto the logout link.

If I can create scripts that define actions like those outlined above, which would can simulate expected real world usage over a period of days, or can simulate hordes of users in a more realistic manner than the ab/httperf mechanism of hammering a single URL, it would be very valuable.

I don't need to drive an actual IE or Firefox session, and for this sort of thing that's impractical.

Thanks,

Kirk Haines

···

On Sat, 23 Dec 2006, Mat Schaffer wrote:

On 12/22/06, Jason Roelofs <jameskilton@gmail.com> wrote:

On 12/22/06, khaines@enigo.com <khaines@enigo.com> wrote:

16

khaines@enigo.com wrote:

What I would like, and what Mat wants, I think, have a great deal of overlap.

To repeat some of what I said in an earlier post, I'd like to.

1) Hit the homepage of a site.
2) Go to a random URL on the site.
   Repeat 2-5 times.
3) Go to the fund information page.
4 Click through to the distributions display.
5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information, etc....

Or, even more ambitiously:

1) Hit the homepage of a site.
2) Follow the link to the advisor login page.
3) Login (an action that requires sending a form response, and receiving a cookie in response that will be sent on further requests).
4) View fund information for some random funds.
5) Go into advisor tools and feed some numbers into a calculator.
6) Goto the logout link.

If I can create scripts that define actions like those outlined above, which would can simulate expected real world usage over a period of days, or can simulate hordes of users in a more realistic manner than the ab/httperf mechanism of hammering a single URL, it would be very valuable.

I don't need to drive an actual IE or Firefox session, and for this sort of thing that's impractical.

What you are describing is a load testing tool. I work with one such tool, Segue (now Borland) SilkPerformer. Another well-known one is Mercury LoadRunner. As far as I know, there are no open source tools that have the flexibility in script *capture* these tools have. Essentially, you fire up a script recorder, it fires up a browser, you exercise your app, you close the recorder and it generates a script. There are also tools for load test monitoring, server monitoring and performance/bottleneck analysis. These things are expensive, though.

The closest thing I've found in the open source world is jMeter. But as far as I know, you have to code your own scripts, your own benchmarks and your own performance analysis tools with jMeter. It's a tradeoff of money versus effort. Quite frankly, I'd recommend spending the money. Email me off-line if you want the details.

Now if someone wants to *write* a full-featured web application load testing tool -- and I mean *full-featured* -- in Ruby and open source it, allow me to encourage them. I sure as heck don't have the time to do it. :slight_smile:

···

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given rabbits fire.

17

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :slight_smile: Yes, it's Java, but if you're considering JMeter...

Caveat: I haven't used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.

···

On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

.... As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com

18

Hassan Schroeder wrote:

···

On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

.... As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :slight_smile: Yes, it's Java, but if you're considering JMeter...

Caveat: I haven't used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.

Yeah ... I missed it when I first posted but I have just looked at it. Unfortunately, the recorder apparently only works with Firefox, which is a deal-killer. However much you may hate IE and Microsoft ...

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given rabbits fire.

19

WebTest, from my cursory look, seems to be more oriented at validating the correctness of a web site/application than stress testing, though. Did I miss something?

Kirk Haines

···

On Sat, 23 Dec 2006, Hassan Schroeder wrote:

On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

.... As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :slight_smile: Yes, it's Java, but if you're considering JMeter...

20

Could you explain why this is a "deal killer"? I would hope that per-browser
functionality testing is a seperate suite from load testing, unless you're
sending different websites depending on the browser type.

Jason

···

On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

Hassan Schroeder wrote:
> On 12/22/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:
>
>> .... As far as I know, there are no open source tools
>> that have the flexibility in script *capture* these tools have.
>> Essentially, you fire up a script recorder, it fires up a browser, you
>> exercise your app, you close the recorder and it generates a script.
>
> Perhaps you missed my prior post about WebTest and associated
> WebTestRecorder? :slight_smile: Yes, it's Java, but if you're considering
JMeter...
>
> Caveat: I haven't used either of the commercial tools you mentioned,
> so grain of salt, YMMV, etc.
>
Yeah ... I missed it when I first posted but I have just looked at it.
Unfortunately, the recorder apparently only works with Firefox, which is
a deal-killer. However much you may hate IE and Microsoft ...

--
M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.