Some problems with URI.extract?

ruby-talk
1

Hello,

I’ve written, some times ago, a Ruby code that allows me to follow web
links and to retrieve easily interesting files. This little software
works well. To extract the links from a downloaded webpage I use
URI.extract and I’ve noticed that URI.extract miss a lot of links. In
fact URI.extract doesn’t understand (resolve ?) relative links (for
example link). Am I wrong ? If I don’t,
what way do you advice to me to be sure to retrieve all the relative links ?

Thank you and good evening.

···

–
Nicolas Cavigneaux | GPG KeyID : F0954C41
bounga@altern.org | http://bounga.ath.cx

2

I guess you are using Ruby 1.9 from CVS ?

I just read in Oniguruma’s ChangeLog :
2004/05/25: [bug] (thanks Masahiro Sakai) [ruby-dev:23560]
ruby -ruri -ve ‘URI::ABS_URI =~
“http://example.org/Andr\xC3\xA9”’
nested STK_REPEAT type stack can’t backtrack repeat_stk.
add OP_REPEAT_INC_SG and OP_REPEAT_INC_NG_SG.

I have no idea what that problem was, only that it was URI related.

Does it work on Ruby 1.8.1/2 ?

···

Nicolas Cavigneaux bounga@altern.org wrote:

I’ve written, some times ago, a Ruby code that allows me to follow web
links and to retrieve easily interesting files. This little software
works well. To extract the links from a downloaded webpage I use
URI.extract and I’ve noticed that URI.extract miss a lot of links. In
fact URI.extract doesn’t understand (resolve ?) relative links (for
example link). Am I wrong ? If I don’t,
what way do you advice to me to be sure to retrieve all the relative links ?

–
Simon Strandgaard

3

“Nicolas Cavigneaux” bounga@altern.org schrieb im Newsbeitrag
news:pan.2004.05.25.20.40.03.951102@altern.org…

Hello,

I’ve written, some times ago, a Ruby code that allows me to follow web
links and to retrieve easily interesting files. This little software
works well. To extract the links from a downloaded webpage I use
URI.extract and I’ve noticed that URI.extract miss a lot of links. In
fact URI.extract doesn’t understand (resolve ?) relative links (for
example link). Am I wrong ? If I don’t,
what way do you advice to me to be sure to retrieve all the relative links
?

I think I remember there was a method URI.join which could join an absolute
URI and a relative one. Or was it URL.join?

robert
4

IIRC, URI.extract(str) just scans plain text for URIs. So, links in
html would have to be absolute, not relative, ie.
“http://google.com/help”, not just “/help”.

To get all the links out of html, you would probably need to create a
regular expression that finds all link-ish html attributes (, , , etc), parse them to see what
type of link they are, then construct a full URI based on the page’s
original location.

A quick, incomplete, untested example.

open-uri is nice

require ‘open-uri’

def get_URI_list(uri)

download the page at the uri passed

page_data = open(uri){|f|f.read}

scan it for the contents of html

attributes that are usually links

uris = page_data.scan(/(?:href|src|rel|)=“([^”]*)"/)

convert relative links to absolute links

uris.map do |item|
case item
when /^// # it’s relative to site root
“http://” + URI.parse(uri).host + item
when /^http:/ #it’s absolute
item
else # it’s relative to the current page
# merge the two uris here. This is left as an exercise :wink:
end
end
end

HTH,
Mark

···

On May 25, 2004, at 1:43 PM, Nicolas Cavigneaux wrote:

Hello,

I’ve written, some times ago, a Ruby code that allows me to follow web
links and to retrieve easily interesting files. This little software
works well. To extract the links from a downloaded webpage I use
URI.extract and I’ve noticed that URI.extract miss a lot of links. In
fact URI.extract doesn’t understand (resolve ?) relative links (for
example link). Am I wrong ? If I don’t,
what way do you advice to me to be sure to retrieve all the relative
links ?

5

No, I’m using Ruby 1.8.1.

···

On Wed, 26 May 2004 05:55:07 +0900, Simon Strandgaard wrote:

I guess you are using Ruby 1.9 from CVS ?

–
Nicolas Cavigneaux | GPG KeyID : F0954C41
bounga@altern.org | http://bounga.ath.cx

6

IIRC, URI.extract(str) just scans plain text for URIs. So, links in html
would have to be absolute, not relative, ie. “http://google.com/help”, not
just “/help”.

OK, that’s what I was thinking.

 else # it's relative to the current page
   # merge the two uris here. This is left as an exercise ;)
 end

eh eh :wink: Thank you for your help and for this little exercise :slight_smile:

Bye.

···

On Wed, 26 May 2004 07:22:52 +0900, Mark Hubbart wrote:

Nicolas Cavigneaux | GPG KeyID : F0954C41
bounga@altern.org | http://bounga.ath.cx