Confusion trying to get IMG tags from html page

ruby-talk
1

I'm trying to download images from a web page that has them listed with
html like what I've pasted below. Basically, I want to iterate through
all the <IMG tags and grab the SRC= info and download those files.
I've tried a bunch of things with not much luck. Here is my last
attempt. Any help would be appreciated.

require 'net/http'
require 'rexml/document'

Net::HTTP.start('www.myphotowebsite.com') do |http|
  response =
http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
  puts "Code = #{response.code}"
  puts "Message = #{response.message}"
  #puts "Body = #{response.body}"

  #parser = HTMLTree::XMLParser.new(false,false)
  #parser.feed(client.getContent(url))
  xml=response.body

  xml.elements.each('//HREF]') do |node|

end

<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1516.jpg">IMG_1516.jpg</A> 28-Jul-2005 08:59
233k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1517.jpg">IMG_1517.jpg</A> 18-Jun-2005 08:03
819k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1518.jpg">IMG_1518.jpg</A> 28-Jul-2005 09:00
398k
<I

2

pkellner wrote:

I'm trying to download images from a web page that has them listed with
html like what I've pasted below. Basically, I want to iterate through
all the <IMG tags and grab the SRC= info and download those files.
I've tried a bunch of things with not much luck. Here is my last
attempt. Any help would be appreciated.

require 'net/http'
require 'rexml/document'

Net::HTTP.start('www.myphotowebsite.com') do |http|
  response =
http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
  puts "Code = #{response.code}"
  puts "Message = #{response.message}"
  #puts "Body = #{response.body}"

  #parser = HTMLTree::XMLParser.new(false,false)
  #parser.feed(client.getContent(url))
  xml=response.body

  xml.elements.each('//HREF]') do |node|

end

<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1516.jpg">IMG_1516.jpg</A> 28-Jul-2005 08:59
233k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1517.jpg">IMG_1517.jpg</A> 18-Jun-2005 08:03
819k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1518.jpg">IMG_1518.jpg</A> 28-Jul-2005 09:00
398k

That isn't valid XML (tags without matching end-tags must have a
trailing slash), so the parser probably doesn't understand it. Assuming
the HTML isn't too complicated, you should be able to get the info with
regular expressions.

3

pkellner wrote:

I'm trying to download images from a web page that has them listed with
html like what I've pasted below. Basically, I want to iterate through
all the <IMG tags and grab the SRC= info and download those files.

require 'uri'
require 'open-uri'
require 'html/htmltokenizer'

class WebPage
   attr_reader :images # URLs of all images on page

   # Get a web page from a specified URL
   def get(url)
     @uri = URI.parse(url)
     open(url) {|result| @body = result.read }
   end

   # Parse the web page, extracting links to images
   def parse
     if !@body
       return
     end
     tokenizer = HTMLTokenizer.new(@body)
     @images = Array.new
     while tag = tokenizer.getTag('img')
       url = tag.attr_hash['src']
       uri = @uri.merge(url)
       @images.push(uri.to_s)
     end
   end
end

wp = WebPage.new
wp.get('http://www.ruby-lang.org/en/&#39;\)
wp.parse
for u in wp.images
   puts u
end

mathew

···

--
<URL:http://www.pobox.com/~meta/&gt;
          WE HAVE TACOS

4

Or use the simplest variant:

require 'open-uri'

open('www.myphotowebsite.com') do |http|
response = http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
puts response.body.scan(/[^\t "'=]+\.(?:jpg|gif|png)/).flatten
end

regards,

Brian

···

On 30/07/05, pkellner <peter@peterkellner.net> wrote:

I'm trying to download images from a web page that has them listed with
html like what I've pasted below. Basically, I want to iterate through
all the <IMG tags and grab the SRC= info and download those files.
I've tried a bunch of things with not much luck. Here is my last
attempt. Any help would be appreciated.

require 'net/http'
require 'rexml/document'

Net::HTTP.start('www.myphotowebsite.com') do |http|
  response =
http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
  puts "Code = #{response.code}"
  puts "Message = #{response.message}"
  #puts "Body = #{response.body}"

  #parser = HTMLTree::XMLParser.new(false,false)
  #parser.feed(client.getContent(url))
  xml=response.body

  xml.elements.each('//HREF]') do |node|

end

<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1516.jpg">IMG_1516.jpg</A> 28-Jul-2005 08:59
233k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1517.jpg">IMG_1517.jpg</A> 18-Jun-2005 08:03
819k
<IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
HREF="IMG_1518.jpg">IMG_1518.jpg</A> 28-Jul-2005 09:00
398k
<I

--
http://ruby.brian-schroeder.de/

Stringed instrument chords: http://chordlist.brian-schroeder.de/

5

I was really hoping for some code or pseudo code. I'm new to ruby and
have been thrashing over this for hours. I promise to put some back
later when I know more about this. (and sadly, I'm not a regular
expression wizard)

Thanks

Charles Steinman wrote:

···

pkellner wrote:
> I'm trying to download images from a web page that has them listed with
> html like what I've pasted below. Basically, I want to iterate through
> all the <IMG tags and grab the SRC= info and download those files.
> I've tried a bunch of things with not much luck. Here is my last
> attempt. Any help would be appreciated.
>
> require 'net/http'
> require 'rexml/document'
>
> Net::HTTP.start('www.myphotowebsite.com') do |http|
> response =
> http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
> puts "Code = #{response.code}"
> puts "Message = #{response.message}"
> #puts "Body = #{response.body}"
>
> #parser = HTMLTree::XMLParser.new(false,false)
> #parser.feed(client.getContent(url))
> xml=response.body
>
> xml.elements.each('//HREF]') do |node|
>
> end
>
>
>
>
>
> <IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
> HREF="IMG_1516.jpg">IMG_1516.jpg</A> 28-Jul-2005 08:59
> 233k
> <IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
> HREF="IMG_1517.jpg">IMG_1517.jpg</A> 18-Jun-2005 08:03
> 819k
> <IMG SRC="/icons/image2.gif" ALT="[IMG]"> <A
> HREF="IMG_1518.jpg">IMG_1518.jpg</A> 28-Jul-2005 09:00
> 398k

That isn't valid XML (tags without matching end-tags must have a
trailing slash), so the parser probably doesn't understand it. Assuming
the HTML isn't too complicated, you should be able to get the info with
regular expressions.

6

> [snip]

Or use the simplest variant:

require 'open-uri'

open('www.myphotowebsite.com') do |http|
response = http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
puts response.body.scan(/[^\t "'=]+\.(?:jpg|gif|png)/).flatten
end

obviously that should have read

require 'net/http'

Net::HTTP.start('www.myphotowebsite.com') do |http|
response = http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
puts response.body.scan(/[^\t "']+\.(?:jpg|gif|png)/).flatten
end

···

regards,

Brian

--
http://ruby.brian-schroeder.de/

Stringed instrument chords: http://chordlist.brian-schroeder.de/

--
http://ruby.brian-schroeder.de/

Stringed instrument chords: http://chordlist.brian-schroeder.de/

7

pkellner wrote:

I was really hoping for some code or pseudo code. I'm new to ruby and
have been thrashing over this for hours. I promise to put some back
later when I know more about this. (and sadly, I'm not a regular
expression wizard)

I use WWW::Mechanize to slurp down numerous CafePress shop pages and snarf out the img info, which I use to automagically create the product pages for rubystuff.com.

The code sample here is a much simplified version.

Mechanize lets you use custom classes to encapsulate node types, which in turn makes it simpler to manipulate assorted HTML elements. I need to extract assorted data from image URLs, so I coded up some additional trickery not shown here.

Also note that some sites reject bots, spiders, etc. when the declared user-agent is not something acceptable. Hence the random selection from UA here.

#!/usr/local/bin/ruby

require 'mechanize'

UA = [
    'Windows IE 6' ,
    'Windows Mozilla',
    'Mac Safari' ,
    'Mac Mozilla' ,
    'Linux Mozilla',
    'Linux Konqueror' ]

# Wrap certain nodes in an Img class to make
# node attribute access a bit easier to grok.
class Img
   attr_reader :alt, :src

   def initialize( node )
     @node = node
     @alt = ''
     @src = ''

     if @node.attributes[ 'alt' ]
       @alt = @node.attributes[ 'alt' ].to_s.strip
     end
     if @node.attributes[ 'src' ]
       @src = @node.attributes[ 'src' ].to_s.strip
     end
   end
end

# Now with Rails tote bags and thongs and stuff!
url = 'http://www.cafepress.com/rubyonrailsshop&#39;

agent = WWW::Mechanize.new {|a| a.log = Logger.new( STDERR ) }
agent.user_agent_alias = UA[ rand( UA.size - 1 ) ]

# This tells Mechanize to watch for certain elements, and
# map matching nodes to the keyed class. Here, when an img
# element is encountered, mechanize will use the node to create
# an Img object and store it for us.
agent.watch_for_set = { 'img' => Img }

page = agent.get( url )

# Get the watch items we're interested in
images = page.watches[ 'img' ]

# What did we get?
images.each do |img|
   p img.src
end

···

#----------------

Hope this helps.

Get Mechanize from rubyforge.org, from the wee project page.

http://rubyforge.org/projects/wee/

James Britt

--

http://www.ruby-doc.org - The Ruby Documentation Site
http://www.rubyxml.com - News, Articles, and Listings for Ruby & XML
http://www.rubystuff.com - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com - Playing with Better Toys

8

Brian Schröder wrote:

require 'net/http'

Net::HTTP.start('www.myphotowebsite.com') do |http|
response = http.get('/terry/temp/2005-06-18%20Kiss%20of%20Death%203/')
puts response.body.scan(/[^\t "']+\.(?:jpg|gif|png)/).flatten
end

If you want only picture names that are in tags:

puts response.body.scan(/<[^>]*?([^\t "']+\.(?:jpg|gif|png))[^>]*>/)