# Pkg for getting web pages **URL:** \ **Category:** ruby-talk\ **Created:** [6 June 2003 15:50 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279 "2003-06-06T15:50:35Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [6 June 2003 15:50 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/1 "2003-06-06T15:50:35Z")
Hi: What is the ‘best’ package for getting pages from the web. Webfetcher looks like it has not been updated for a while and http-access is listed as not supporting cookies. Is there a common pkg that people are currently using? Thanks > **···** > > ## – Jim Freeze > > Be braver – you can’t cross a chasm in two small jumps. ---
**Author:** ![Gennady1](https://avatars.discourse-cdn.com/v4/letter/g/a4c791/32.png) [@Gennady1](https://rubytalk.org/u/Gennady1)\ **Post date:** [6 June 2003 18:48 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/2 "2003-06-06T18:48:59Z")
wget is good enough: [Wget - GNU Project - Free Software Foundation](http://www.gnu.org/software/wget/wget.html) Gennady. > **···** > > ----- Original Message ----- > From: “Jim Freeze” [jim@freeze.org](mailto:jim@freeze.org) > To: “ruby-talk ML” [ruby-talk@ruby-lang.org](mailto:ruby-talk@ruby-lang.org) > Sent: Friday, June 06, 2003 8:50 AM > Subject: Pkg for getting web pages > > > Hi: > > > > What is the ‘best’ package for getting pages > > from the web. Webfetcher looks like it has > > not been updated for a while and http-access > > is listed as not supporting cookies. > > > > Is there a common pkg that people are currently > > using? > > > > Thanks > > > > ## – Jim Freeze > > > > Be braver – you can’t cross a chasm in two small jumps. ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [7 June 2003 07:50 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/3 "2003-06-07T07:50:15Z")
There was one which was announced on ruby-talk around the beginning of this year, and I used it successfully to download a copy of the pickaxe as HTML (before I realised it was available for direct download anyway 🙂 Ah yes, found it: httpsnapshot-0.3.5.36.tar.gz I only used it once but it seemed to do the job (if mirroring a website is what you’re after) I can’t find the announcment on the ruby-talk archives, but it is on RAA. Cheers, Brian. > **···** > > On Sat, Jun 07, 2003 at 12:50:35AM +0900, Jim Freeze wrote: > > > What is the ‘best’ package for getting pages > > from the web. Webfetcher looks like it has > > not been updated for a while and http-access > > is listed as not supporting cookies. ---
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [6 June 2003 19:19 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/4 "2003-06-06T19:19:59Z")
Thanks. I was actually looking for a ruby pkg, but this looks very nice. > **···** > > On Saturday, 7 June 2003 at 3:48:59 +0900, Gennady wrote: > > > wget is good enough: [Wget - GNU Project - Free Software Foundation](http://www.gnu.org/software/wget/wget.html) > > ## – Jim Freeze > > 1.79 x 10^12 furlongs per fortnight – it’s not just a good idea, it’s > the law! ---
**Author:** ![Gennady1](https://avatars.discourse-cdn.com/v4/letter/g/a4c791/32.png) [@Gennady1](https://rubytalk.org/u/Gennady1)\ **Post date:** [6 June 2003 19:39 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/5 "2003-06-06T19:39:20Z")
Oops, I did not get it from your post. But anyway, Ruby is great for interfacing to other programs and if something is being done well by some specialized tool it is worth using it from Ruby instead of reinventing a bicycle in Ruby ;-), especially when the tool is readily available. Gennady. > **···** > > ----- Original Message ----- > From: “Jim Freeze” [jim@freeze.org](mailto:jim@freeze.org) > To: “ruby-talk ML” [ruby-talk@ruby-lang.org](mailto:ruby-talk@ruby-lang.org) > Sent: Friday, June 06, 2003 12:19 PM > Subject: Re: Pkg for getting web pages > > > On Saturday, 7 June 2003 at 3:48:59 +0900, Gennady wrote: > > > > > wget is good enough: [Wget - GNU Project - Free Software Foundation](http://www.gnu.org/software/wget/wget.html) > > > > Thanks. I was actually looking for a ruby pkg, but this > > looks very nice. > > > > ## – Jim Freeze > > > > 1.79 x 10^12 furlongs per fortnight – it’s not just a good idea, it’s > > the law! ---
**Author:** ![Josef\_Jupp\_SCHUGT](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@Josef\_Jupp\_SCHUGT](https://rubytalk.org/u/Josef_Jupp_SCHUGT)\ **Post date:** [7 June 2003 17:44 UTC](https://rubytalk.org/t/pkg-for-getting-web-pages/6279/6 "2003-06-07T17:44:50Z")
Saluton! - Gennady; 2003-06-06, 20:14 UTC: > But anyway, Ruby is great for interfacing to other programs and if > something is being done well by some specialized tool it is worth > using it from Ruby instead of reinventing a bicycle in Ruby ;-), > especially when the tool is readily available. The question is still open to discussion: _Which_ tool. Many people know w3 but that is not the only tool that fits. There are others, most notably _curl_ - it comes with a library (for C IIRC). \*curl \* is a client to get documents/files from or send documents to a server, using any of the supported protocols (HTTP, HTTPS, FTP, GOPHER, DICT, TELNET, LDAP or FILE). The command is designed to work without user interaction or any kind of interactivity. curl offers a busload of useful tricks like proxy support, user authentication, ftp upload, HTTP post, SSL (https:) connections, cookies, file transfer resume and more. [http://curl.haxx.se/](http://curl.haxx.se/) _snarf_ can transfer files through the http, gopher, finger, and ftp protocols without user interaction. It is small and fast and can force active FTP (default is passive), resume downloading a partially transferred file and spoof MSIE us well as Navigator user-agent string and checks the SNARF\_PROXY, FTP\_PROXY, GOPHER\_PROXY, HTTP\_PROXY, and PROXY environment variables. [http://www.xach.com/snarf/index.html](http://www.xach.com/snarf/index.html) Gis, Josef ‘Jupp’ Schugt