How to imitate a browser?

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?

···

--
Posted via http://www.ruby-forum.com/.

Use Mechanize:

http://mechanize.rubyforge.org/mechanize/

It manages cookies and allows you to fill forms, click links and so on.

Jesus.

···

On Sun, Oct 3, 2010 at 4:28 PM, Marek Osvald <dre.share@gmail.com> wrote:

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?

Hi Marek,

Have you tried the combo Nokogiri (http://nokogiri.org/) / Mechanize
(http://mechanize.rubyforge.org/mechanize/) to do the job?

I recommend you to take a look at:
  * http://railscasts.com/episodes/190-screen-scraping-with-nokogiri
  * http://railscasts.com/episodes/191-mechanize
Adriano

···

________________________________
From: Marek Osvald <dre.share@gmail.com>
To: ruby-talk ML <ruby-talk@ruby-lang.org>
Sent: Sun, October 3, 2010 11:28:38 AM
Subject: How to imitate a browser?

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?
--
Posted via http://www.ruby-forum.com/.

so I'm trying to write a simple script for extraction of my profile data
from one site.

Category: Browser testing - The Ruby Toolbox (watir,
especially)

···

--
Posted via http://www.ruby-forum.com/\.