Website screen scraping with Mechanize or Rubyful Soup

ruby-talk
1

I'm trying to get some website screen scraping working, but I'm
suffering from a lack of examples and documentation for either
WWW::Mechanize or Rubyful Soup.

With WWW::Mechanize, the only example I found was
http://www.zenspider.com/pipermail/ruby/2005-July/002068.html. I tried
to simplify this to the script below, but it just prints out "My wife
is ".

Rubyful Soup <http://www.crummy.com/software/RubyfulSoup/> also seems
like a great library, but there doesn't seem to be a single example
(only Python ones
<http://www.crummy.com/software/BeautifulSoup/examples.html>).

#!/usr/bin/env ruby

require 'mechanize'

agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows IE 6'

# get first page
page = agent.get('http://www.dankohn.com/')
md = page.body.match /My wife, (\w+\s\w+)<\/a>/m

printf "My wife is ", md

Thanks in advance for any help you can offer.

2

No quite what you wanted but

This can turn HTML in well formed XML

http://rubyforge.org/projects/tidy/

which is much easier to parse.

Maybe worth a look.

3

Dan Kohn wrote:

I'm trying to get some website screen scraping working, but I'm
suffering from a lack of examples and documentation for either
WWW::Mechanize or Rubyful Soup.

With WWW::Mechanize, the only example I found was
http://www.zenspider.com/pipermail/ruby/2005-July/002068.html\. I tried
to simplify this to the script below, but it just prints out "My wife
is ".

What are you actually trying to accomplish?

James

···

--

http://www.ruby-doc.org - The Ruby Documentation Site
http://www.rubyxml.com - News, Articles, and Listings for Ruby & XML
http://www.rubystuff.com - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com - Playing with Better Toys

4

Hi

···

On 9/12/05, Dan Kohn <dan@dankohn.com> wrote:

I'm trying to get some website screen scraping working, but I'm
suffering from a lack of examples and documentation for either
WWW::Mechanize or Rubyful Soup.

With WWW::Mechanize, the only example I found was
http://www.zenspider.com/pipermail/ruby/2005-July/002068.html\. I tried
to simplify this to the script below, but it just prints out "My wife
is ".

Rubyful Soup <http://www.crummy.com/software/RubyfulSoup/&gt; also seems
like a great library, but there doesn't seem to be a single example
(only Python ones
<http://www.crummy.com/software/BeautifulSoup/examples.html&gt;\).

#!/usr/bin/env ruby

require 'mechanize'

agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows IE 6'

# get first page
page = agent.get('http://www.dankohn.com/&#39;\)
md = page.body.match /My wife, (\w+\s\w+)<\/a>/m

printf "My wife is ", md

print "My wife is ", md[1]

Michel.

5

My ultimate goal is to create a series of screen scrapers that are able
to access airline websites (including entering username and password,
dealing with redirects, etc.), find my mileage and recent flights,
parse the data, put it in some variables, and save it to MySQL (with
rails).

I was trying to start with baby steps to understand the methods these
libraries support. Specifically, I was trying to fetch my own web
page, and then use a regex to match to my wife's name, "Julie Pullen",
since I have link text on www.dankohn.com saying "My wife, Julie
Pullen". I was then going to gradually increase the complexity of the
scraping.

Thanks in advance for any example scripts or documentation that you can
provide showing web scraping in ruby.

And Lyndon, I'm a huge fan of Tidy for cleaning up my own web pages,
but I'm not sure it's helpful here, as was aiming to use regexes to
parse the HTML rather than the DOM.

6

Thanks, Michel.

7

You're welcome!

···

On 9/12/05, Dan Kohn <dan@dankohn.com> wrote:

Thanks, Michel.

8

And Lyndon, I'm a huge fan of Tidy for cleaning up my own web pages,
but I'm not sure it's helpful here, as was aiming to use regexes to
parse the HTML rather than the DOM.

Well, DOM allows you to use XPath, which is a powerfull query mechanism.

This
http://www-128.ibm.com/developerworks/java/library/j-jtp03225.html?ca=dgr-jw26XQueryis
XQuery specific, but relies
on XPath.

And example from the article

//td[contains(a/small/text(), "New York, NY")]

9

If you can use a Windows box, Watir might be the best-fit.

Install (using Gem or the Windows setup program that is available at
http://wtr.rubyforge.org/) and IRB this:

require 'watir'
ie = Watir::IE.new
ie.goto "dankohn.com"
julie_lines = ie.text.scan(/.*Julie.*/)
link = ie.link(:text, /Julie/)
link.click

Cheers,
Dave

10

Looks cool, thanks. I'm developing on a Windows machine, but plan to
move to a "real" machine for production. So it looks like Mechanize,
REXML, and XQuery will be the best bet.