[ANN] WWW::Mechanize alike in Ruby

Hi there,

I just read the Web-testing thread in the latest "Ruby Weekly News" and want
to tell the interested people that I hacked up a little library which I use
in some small projects. Web::Unit didn't work for my purposes, and it was
easier to reinvent the wheel than to hack Web::Unit (of course I first tried
to hack it...).

It depends on narf-lib (>= 0.6.3) more exactly, it's version of htmltools, and
requires Ruby-current (due to limitations in the net/http library of 1.8.2).
If you want to try it out with Ruby 1.8.2, just get the net/*.rb files from
Ruby-current and use these instead (or use local overrides:
$LOAD_PATH.unshift './net-overrides' ... ).

Features:
  * Cookies (very preliminary, no expire etc.)
  * automatic redirect
  * Forms, Links

Be aware that it's just a few-hour hack, to get my project done.

It's still very generic and misses lots of usability methods (I use lots of
"find" and use explicit parameters instead of implicit, e.g. last visited
page).

Here's a simple example, that logs into your rubyforge.org acount:

  require 'mechanize'

  agent = WWW::Mechanize.new {|a| a.log = Logger.new(STDERR) }
  page = agent.get('http://rubyforge.org/')
  link = page.links.find {|l| l.node.text =~ /Log In/ }
  page = agent.click(link)
  form = page.forms[1]
  form.fields.find {|f| f.name == 'form_loginname'}.value = ARGV[0]
  form.fields.find {|f| f.name == 'form_pw'}.value = ARGV[1]
  page = agent.submit(form, form.buttons.first)
  puts page.body

Here are the sources:

  http://www.ntecs.de/viewcvs/viewcvs/WWW/

I've realized that Ruby has client libraries for www/http (cgi.rb, net/http)
and server libraries for http (webrick), but not a common library that puts
those things together...

Regards,

  Michael

Hi there,

I just read the Web-testing thread in the latest "Ruby Weekly News" and want
to tell the interested people that I hacked up a little library which I use
in some small projects. Web::Unit didn't work for my purposes, and it was
easier to reinvent the wheel than to hack Web::Unit (of course I first tried
to hack it...).

It depends on narf-lib (>= 0.6.3) more exactly, it's version of htmltools, and
requires Ruby-current (due to limitations in the net/http library of 1.8.2).

Isn't 1.8.2 the current Ruby? Or does "Ruby-current" mean something
other than what I'm understanding it to mean?

If you want to try it out with Ruby 1.8.2, just get the net/*.rb files from
Ruby-current and use these instead (or use local overrides:
$LOAD_PATH.unshift './net-overrides' ... ).

Features:
  * Cookies (very preliminary, no expire etc.)
  * automatic redirect
  * Forms, Links

[snip]

I've realized that Ruby has client libraries for www/http (cgi.rb, net/http)
and server libraries for http (webrick), but not a common library that puts
those things together...

Nice job, Michael. I'm looking forward to trying this one out.

- Jamis

···

On 22:04 Tue 18 Jan , Michael Neumann wrote:

--
Jamis Buck
jamis_buck@byu.edu
http://www.jamisbuck.org/jamis
------------------------------
"I am Victor of Borge. You will be assimil-nine-ed."

If it isn't 1.9 (CVS HEAD), then it's probably a post 1.8.2 release
stable snapshot.

-austin

···

On Tue, 18 Jan 2005 23:32:30 +0900, Jamis Buck <jamis_buck@byu.edu> wrote:

On 22:04 Tue 18 Jan , Michael Neumann wrote:
> Hi there,
>
> I just read the Web-testing thread in the latest "Ruby Weekly News" and want
> to tell the interested people that I hacked up a little library which I use
> in some small projects. Web::Unit didn't work for my purposes, and it was
> easier to reinvent the wheel than to hack Web::Unit (of course I first tried
> to hack it...).
>
> It depends on narf-lib (>= 0.6.3) more exactly, it's version of htmltools, and
> requires Ruby-current (due to limitations in the net/http library of 1.8.2).
Isn't 1.8.2 the current Ruby? Or does "Ruby-current" mean something
other than what I'm understanding it to mean?

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

Jamis Buck wrote:

Hi there,

I just read the Web-testing thread in the latest "Ruby Weekly News" and want to tell the interested people that I hacked up a little library which I use in some small projects. Web::Unit didn't work for my purposes, and it was easier to reinvent the wheel than to hack Web::Unit (of course I first tried to hack it...).

It depends on narf-lib (>= 0.6.3) more exactly, it's version of htmltools, and requires Ruby-current (due to limitations in the net/http library of 1.8.2).

Isn't 1.8.2 the current Ruby? Or does "Ruby-current" mean something
other than what I'm understanding it to mean?

With -current I mean the development version of Ruby. Sorry for the confusion. Maybe this could get back-ported into the stable-trunk?

If you want to try it out with Ruby 1.8.2, just get the net/*.rb files from Ruby-current and use these instead (or use local overrides: $LOAD_PATH.unshift './net-overrides' ... ).

Features:
* Cookies (very preliminary, no expire etc.)
* automatic redirect
* Forms, Links

[snip]

I've realized that Ruby has client libraries for www/http (cgi.rb, net/http) and server libraries for http (webrick), but not a common library that puts those things together...

Nice job, Michael. I'm looking forward to trying this one out.

Well, just download the net/ directory from the development branch of Ruby, and put it into a net-overrides directory (you only need net/http.rb, net/https.rb and net/protocol.rb). Then modify mechanize.rb:

   # put this at the top of mechanize.rb
   $LOAD_PATH.unshift "./net-overrides"

This should then work with Ruby 1.8.2.

Regards,

   Michael

···

On 22:04 Tue 18 Jan , Michael Neumann wrote:

Austin Ziegler wrote:

···

On Tue, 18 Jan 2005 23:32:30 +0900, Jamis Buck <jamis_buck@byu.edu> wrote:

On 22:04 Tue 18 Jan , Michael Neumann wrote:

Hi there,

I just read the Web-testing thread in the latest "Ruby Weekly News" and want
to tell the interested people that I hacked up a little library which I use
in some small projects. Web::Unit didn't work for my purposes, and it was
easier to reinvent the wheel than to hack Web::Unit (of course I first tried
to hack it...).

It depends on narf-lib (>= 0.6.3) more exactly, it's version of htmltools, and
requires Ruby-current (due to limitations in the net/http library of 1.8.2).

Isn't 1.8.2 the current Ruby? Or does "Ruby-current" mean something
other than what I'm understanding it to mean?

If it isn't 1.9 (CVS HEAD), then it's probably a post 1.8.2 release
stable snapshot.

I'm not sure whether it's in the stable branch. Just look for the get_field method, which must be present (net/http*).

Regards,

   Michael