LWP like library for Ruby?

ruby-talk
1

Can anybody tell me does Ruby support perl like LWP programming?

ie. scraping content of websites with support for Cookies

2

Robert Nicholson wrote:

Can anybody tell me does Ruby support perl like LWP programming?

ie. scraping content of websites with support for Cookies

I believe www-mechanize will do what you want.

Regards,

Dan

3

I cannot seem to install www-mechanize using gem install:

root@choochoo:~# apt-get install www-mechanize
Reading package lists... Done
Building dependency tree... Done
E: Couldn't find package www-mechanize
root@choochoo:~# gem install www-mechanize
Attempting local installation of 'www-mechanize'
Local gem file not found: www-mechanize*.gem
Attempting remote installation of 'www-mechanize'
Updating Gem source index for: http://gems.rubyforge.org
ERROR: While executing gem ... (Gem::GemNotFoundException)
    Could not find www-mechanize (> 0) in the repository
root@choochoo:~# gem search mechanize

*** LOCAL GEMS ***
root@choochoo:~#

Is it called something different in gems?

Daniel Berger wrote:

···

Robert Nicholson wrote:

Can anybody tell me does Ruby support perl like LWP programming?

ie. scraping content of websites with support for Cookies

I believe www-mechanize will do what you want.

Regards,

Dan

4

Actually, here is a more interesting failure:

root@choochoo:~# gem install mechanize
Attempting local installation of 'mechanize'
Local gem file not found: mechanize*.gem
Attempting remote installation of 'mechanize'
Install required dependency ruby-web? [Yn]
Successfully installed mechanize-0.4.5
Successfully installed ruby-web-1.1.1
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:21:in `require__': no such file to load -- rdoc/rdoc (LoadError)
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:21:in `require'
        from /usr/local/lib/site_ruby/1.8/rubygems/doc_manager.rb:43:in `generate_rdoc'
        from /usr/local/lib/site_ruby/1.8/rubygems/gem_commands.rb:215:in `execute'
        from /usr/local/lib/site_ruby/1.8/rubygems/gem_commands.rb:214:in `each' from /usr/local/lib/site_ruby/1.8/rubygems/gem_commands.rb:214:in `execute'
        from /usr/local/lib/site_ruby/1.8/rubygems/gem_commands.rb:153:in `each' from /usr/local/lib/site_ruby/1.8/rubygems/gem_commands.rb:153:in `execute'
        from /usr/local/lib/site_ruby/1.8/rubygems/command.rb:49:in `invoke'
        from /usr/local/lib/site_ruby/1.8/rubygems/cmd_manager.rb:94:in `process_args'
        from /usr/local/lib/site_ruby/1.8/rubygems/cmd_manager.rb:67:in `run'
        from /usr/local/lib/site_ruby/1.8/rubygems/gem_runner.rb:13:in `run'
        from /usr/bin/gem:17
root@choochoo:~# which mechanize
root@choochoo:~#

Xeno Campanoli wrote:

···

I cannot seem to install www-mechanize using gem install:

root@choochoo:~# apt-get install www-mechanize
Reading package lists... Done
Building dependency tree... Done
E: Couldn't find package www-mechanize
root@choochoo:~# gem install www-mechanize
Attempting local installation of 'www-mechanize'
Local gem file not found: www-mechanize*.gem
Attempting remote installation of 'www-mechanize'
Updating Gem source index for: http://gems.rubyforge.org
ERROR: While executing gem ... (Gem::GemNotFoundException)
    Could not find www-mechanize (> 0) in the repository
root@choochoo:~# gem search mechanize

*** LOCAL GEMS ***
root@choochoo:~#

Is it called something different in gems?

Daniel Berger wrote:

Robert Nicholson wrote:

Can anybody tell me does Ruby support perl like LWP programming?

ie. scraping content of websites with support for Cookies

I believe www-mechanize will do what you want.

Regards,

Dan

5

[snip]

Looks like it failed with the installation of the documentation. It's probably still ok. Note that WWW::Mechanize does not install a binary like lwp does.

try doing

require 'rubygems'
require 'mechanize'

www = WWW::Mechanize.new

···

On May 28, 2006, at 11:29 PM, Xeno Campanoli wrote:

Actually, here is a more interesting failure:

6

Try HTTPAccess2, it works for me...

http-access2 gives something like the functionality of libwww-perl (LWP) in
Ruby.

http://raa.ruby-lang.org/project/http-access2/

Horacio

Monday 29 May 2006 15:47、Logan Capaldo さんは書きました:

···

On May 28, 2006, at 11:29 PM, Xeno Campanoli wrote:
> Actually, here is a more interesting failure:

[snip]

Looks like it failed with the installation of the documentation. It's
probably still ok. Note that WWW::Mechanize does not install a binary
like lwp does.

try doing

require 'rubygems'
require 'mechanize'

www = WWW::Mechanize.new