How to download file from web site

ruby-talk
1

Okay, it's easy to download a file from an FTP server with Net::FTP. Given a URL like this: "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there a way using pure Ruby to download this file?

2

harp:~ > cat a.rb
require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
   open(File.basename(uri), 'w') do |fout|
     while(buf = fin.read(8192))
       fout.write buf
     end
   end
end

harp:~ > ruby a.rb

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00 RMagick-1.15.3.tar.gz

-a

···

On Sat, 3 Mar 2007, Timothy Hunter wrote:

Okay, it's easy to download a file from an FTP server with Net::FTP. Given a URL like this: "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there a way using pure Ruby to download this file?

--
be kind whenever possible... it is always possible.
- the dalai lama

3

You might be surprised to learn that there is also Net::HTTP...

http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html

And then there's open-uri

http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html

  robert

···

On 03.03.2007 15:47, Timothy Hunter wrote:

Okay, it's easy to download a file from an FTP server with Net::FTP. Given a URL like this: "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there a way using pure Ruby to download this file?

4

Thanks again, Ara!

···

ara.t.howard@noaa.gov wrote:

On Sat, 3 Mar 2007, Timothy Hunter wrote:

Okay, it's easy to download a file from an FTP server with Net::FTP. Given a URL like this: "http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz" is there a way using pure Ruby to download this file?

harp:~ > cat a.rb
require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

harp:~ > ruby a.rb

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00 RMagick-1.15.3.tar.gz

-a

5

He asks naively...

  Why is this so complicated, e.g., the 8192 buffered write, etc.?

Later,

···

ara.t.howard@noaa.gov wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

--
Bil Kleb
http://fun3d.larc.nasa.gov

6

no way - thank you! i'm preparing a class right now, here's one of many
examples i've got put together:

···

On Sun, 4 Mar 2007, Timothy Hunter wrote:

-a

Thanks again, Ara!

#
# gem install RMagick @ http://rmagick.rubyforge.org/
#
   require "rubygems"
   require "RMagick"
#
# converts all images named on the command-line into thumbnails
#
   ARGV.each do |path|
     path = File.expand_path path
     dirname, basename = File.split path
     base, ext = basename.split %r/[.]/, 2

     img = Magick::ImageList.new path

     thumb = img.thumbnail 96, 96
     thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
     thumb.write thumb_path

     puts thumb_path
   end

thanks for rmagick.

-a
--
be kind whenever possible... it is always possible.
- the dalai lama

7

So you don't try to slurp a very large file into Ruby's memory I'm guessing.

James Edward Gray II

···

On Mar 4, 2007, at 10:10 AM, Bil Kleb wrote:

ara.t.howard@noaa.gov wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'
open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

8

well, this will do it too

   def wget src, dst
     open(dst,'w'){|b| open(src){|a| b.write a.read}}
   end

but if you

   wget 'http://host/huge_giant_output.cgi', 'big.out'

and then start 1000 of them in threads you might eat a lot of memory

the 8192 thing is just being robust in the face of potential giant docs. not
an issue for html probably but the OP was talking about downloading tar.gz
files, so those could be big. we've got plenty of them that are > 10GB for
download on our servers....

cheers.

-a

···

On Mon, 5 Mar 2007, Bil Kleb wrote:

ara.t.howard@noaa.gov wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

--
be kind whenever possible... it is always possible.
- the dalai lama

9

Bil Kleb wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet with open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio('http://rubydoc.org/'\) > rio('afile')

I suppose it's using the same buffering technique underneath that abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

···

ara.t.howard@noaa.gov wrote:

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

10

You're welcome!

The reason I'm asking these questions is because I'm working on a Ruby program to download and install ImageMagick, all its delegate libraries, and RMagick, on OS X, starting from scratch and not using MacPorts.

BTW, I saw your comment about the rmagick gem. You're right, it's screwy. I'm hoping to mothball it in RMagick 2.0.0, if I can figure out a way to replace my autoconf-generated configure script with pure Ruby.

···

ara.t.howard@noaa.gov wrote:

On Sun, 4 Mar 2007, Timothy Hunter wrote:

-a

Thanks again, Ara!

no way - thank you! i'm preparing a class right now, here's one of many
examples i've got put together:

#
# gem install RMagick @ http://rmagick.rubyforge.org/
#
  require "rubygems"
  require "RMagick"
#
# converts all images named on the command-line into thumbnails
#
  ARGV.each do |path|
    path = File.expand_path path
    dirname, basename = File.split path
    base, ext = basename.split %r/[.]/, 2

    img = Magick::ImageList.new path

    thumb = img.thumbnail 96, 96
    thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
    thumb.write thumb_path

    puts thumb_path
  end

thanks for rmagick.

-a

11

reminds me of an RCR i've had in mind:

   io.relay another_io, bufsize

seems like a nice one huh?

-a

···

On Mon, 5 Mar 2007, Joel VanderWerf wrote:

Bil Kleb wrote:

ara.t.howard@noaa.gov wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet with open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio('http://rubydoc.org/'\) > rio('afile')

I suppose it's using the same buffering technique underneath that abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

--
be kind whenever possible... it is always possible.
- the dalai lama

12

I just spent about a day fixing my RMagick install (on Mac OS X), that had become horribly broken, so releasing something like this would make you a hero in my eyes.

James Edward Gray II

···

On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:

The reason I'm asking these questions is because I'm working on a Ruby program to download and install ImageMagick, all its delegate libraries, and RMagick, on OS X, starting from scratch and not using MacPorts.

13

Yes, although I'm not sure whether "relay" is immediately clear to everyone - especially non native speakers. I briefly thought of "copy_to" but that is not correct in every case, i.e. not always it is actually a copy. What about "forward_to"? Hm...

Kind regards

  robert

···

On 04.03.2007 20:50, ara.t.howard@noaa.gov wrote:

On Mon, 5 Mar 2007, Joel VanderWerf wrote:

Bil Kleb wrote:

ara.t.howard@noaa.gov wrote:

require 'open-uri'
uri = 'http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz'

open(uri) do |fin|
  open(File.basename(uri), 'w') do |fout|
    while(buf = fin.read(8192))
      fout.write buf
    end
  end
end

He asks naively...

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn't it? Especially in the same snippet with open(uri), which abstracts so much.

Didn't the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio('http://rubydoc.org/'\) > rio('afile')

I suppose it's using the same buffering technique underneath that abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

reminds me of an RCR i've had in mind:

  io.relay another_io, bufsize

seems like a nice one huh?

14

James Edward Gray II wrote:

···

On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:

The reason I'm asking these questions is because I'm working on a Ruby program to download and install ImageMagick, all its delegate libraries, and RMagick, on OS X, starting from scratch and not using MacPorts.

I just spent about a day fixing my RMagick install (on Mac OS X), that had become horribly broken, so releasing something like this would make you a hero in my eyes.

James Edward Gray II

Ouch! Sorry you had trouble. Email me when you have a problem, James. I'll be glad to do what I can to help.

BTW, just last week I updated my "Installing RMagick on OS X HOWTO" here: http://rmagick.rubyforge.org/install-osx.html\. I guess it's too late to help you but maybe it'll help somebody else.

15

No worries. It wasn't your fault. I broke it.

James Edward Gray II

···

On Mar 3, 2007, at 11:53 AM, Timothy Hunter wrote:

James Edward Gray II wrote:

On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:

The reason I'm asking these questions is because I'm working on a Ruby program to download and install ImageMagick, all its delegate libraries, and RMagick, on OS X, starting from scratch and not using MacPorts.

I just spent about a day fixing my RMagick install (on Mac OS X), that had become horribly broken, so releasing something like this would make you a hero in my eyes.

James Edward Gray II

Ouch! Sorry you had trouble. Email me when you have a problem, James. I'll be glad to do what I can to help.

16

Here is a shell script I wrote a while ago to build ruby/imagemagick/rmagick on OSX without using macports. Feel free to use it if you want.

#!/bin/sh

# Install Ruby & IMageMagick & RMagick on Macintosh OS X 10.4.7 (Intel)
# Stock Apple Developer Tools (2.3)

# Readline
READLINE_VERSION="5.1"
wget ftp://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
tar xzvf readline-${READLINE_VERSION}.tar.gz
pushd readline-${READLINE_VERSION}
./configure --prefix=/usr/local
make
sudo make install
popd

# Ruby

RUBY_VERSION="1.8.5"

wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-${RUBY_VERSION}.tar.gz
tar xfz ruby-${RUBY_VERSION}.tar.gz

pushd ruby-${RUBY_VERSION}
./configure
make
sudo make install
popd

# Ruby Gems

GEM_VERSION="0.9.2"

wget http://rubyforge.org/frs/download.php/11289/rubygems-${GEM_VERSION}.tgz
tar xfz rubygems-${GEM_VERSION}.tgz

pushd rubygems-${GEM_VERSION}
sudo ruby setup.rb
popd

# ImageMagick

LPNG_VERSION="1.2.12"

wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-${LPNG_VERSION}.tar.gz
tar xfz libpng-${LPNG_VERSION}.tar.gz

pushd libpng-${LPNG_VERSION}
./configure
make
sudo make install
popd

JPEG_VERSION="6b"

wget http://www.ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz
tar xfz jpegsrc.v${JPEG_VERSION}.tar.gz

pushd jpeg-${JPEG_VERSION}
./configure
make
sudo make install
sudo make install-lib
popd

TIFF_VERSION="3.8.2"

wget ftp://ftp.remotesensing.org/pub/libtiff/tiff-${TIFF_VERSION}.tar.gz
tar xfz tiff-${TIFF_VERSION}.tar.gz

pushd tiff-${TIFF_VERSION}
./configure
make
sudo make install
popd

IM_VERSION_MAJOR="6.2.9"
IM_VERSION_MINOR="4"

IM_VERSION="${IM_VERSION_MAJOR}-${IM_VERSION_MINOR}"

wget ftp://ftp.imagemagick.net/pub/ImageMagick/ImageMagick-${IM_VERSION}.tar.gz
tar xfz ImageMagick-${IM_VERSION}.tar.gz

pushd ImageMagick-${IM_VERSION_MAJOR}
./configure
make
sudo make install
popd

sudo gem install rmagick

Cheers-

-- Ezra Zygmuntowicz-- Lead Rails Evangelist
-- ez@engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

···

On Mar 3, 2007, at 9:53 AM, Timothy Hunter wrote:

James Edward Gray II wrote:

On Mar 3, 2007, at 9:39 AM, Timothy Hunter wrote:

The reason I'm asking these questions is because I'm working on a Ruby program to download and install ImageMagick, all its delegate libraries, and RMagick, on OS X, starting from scratch and not using MacPorts.

I just spent about a day fixing my RMagick install (on Mac OS X), that had become horribly broken, so releasing something like this would make you a hero in my eyes.

James Edward Gray II

Ouch! Sorry you had trouble. Email me when you have a problem, James. I'll be glad to do what I can to help.

BTW, just last week I updated my "Installing RMagick on OS X HOWTO" here: http://rmagick.rubyforge.org/install-osx.html\. I guess it's too late to help you but maybe it'll help somebody else.

17

Ezra Zygmuntowicz wrote:

    Here is a shell script I wrote a while ago to build ruby/imagemagick/rmagick on OSX without using macports. Feel free to use it if you want.

Thanks, Ezra! That's a very nice gift. I'll incorporate all the best parts into my script.

There are a couple of these kinds of bash scripts for installing RMagick floating around. None of them implement my exact set of superstitions about installing ImageMagick[1], so I thought it would be useful to do my own. I'm using Ruby instead of bash, though, since I get very little opportunity to write Ruby code.

[1] "Speed up your RMagick apps in 1 easy step" http://rubyforge.org/forum/forum.php?thread_id=10975&forum_id=1618