# New to ruby.. need a code **URL:** \ **Category:** ruby-talk\ **Created:** [1 April 2006 11:03 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324 "2006-04-01T11:03:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rati\_lion](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@rati\_lion](https://rubytalk.org/u/rati_lion)\ **Post date:** [1 April 2006 11:03 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/1 "2006-04-01T11:03:49Z")
Hi all I am new to Ruby. Found it as an intersting language. Can anyone help me with a simple code in Ruby to check for all the dead and live links in a website ? Thanks Rati ---
**Author:** ![Patrick\_Hurley1](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Patrick\_Hurley1](https://rubytalk.org/u/Patrick_Hurley1)\ **Post date:** [1 April 2006 14:39 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/2 "2006-04-01T14:39:41Z")
A wonderful language, a sort of rough request here is some (very) simple code to get you started, using some great ruby libraries, including rubyful soup ([http://www.crummy.com/software/RubyfulSoup/\](http://www.crummy.com/software/RubyfulSoup/%5C)), which is available as a gem. As written it only checks one page, you would need to make it "walk" the links to recursively check a whole site. Hope it helps pth require 'open-uri' require 'uri' require 'rubyful\_soup' url = '[http://www.yahoo.com/'](http://www.yahoo.com/'); uri = URI.parse(url) html = open(uri).read soup = BeautifulSoup.new(html) #Search the soup links = soup.find\_all('a').map { |a| a['href'] } # Remove javascript links.delete\_if { |href| href =~ /javascript/ } links.each do |l|   # resolve relative paths (there is probably a better way)   link = URI.parse(l)   link.scheme = 'http' unless link.scheme   link.host = uri.host unless link.host   link.path = uri.path + link.path unless link.path[0] == ?/   link = URI.parse(link.to\_s)   # check the link   begin     open(link).read     # if we made it here the link is probably good   rescue Exception =\> e     puts "#{link.to\_s}: #{e.to\_s}"   end end > **路路路** > > On 4/1/06, rati\_lion@yahoo.com \ wrote: > > > Hi all > > > > I am new to Ruby. Found it as an intersting language. Can anyone help > > me with a simple code in Ruby to check for all the dead and live links > > in a website ? > > > > Thanks > > Rati ---
**Author:** ![rati\_lion](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@rati\_lion](https://rubytalk.org/u/rati_lion)\ **Post date:** [1 April 2006 16:18 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/3 "2006-04-01T16:18:50Z")
Thank you. Will take care in future to not post a rough request 馃檪 Thanks again for this extended help. ---
**Author:** ![David\_Vallner](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@David\_Vallner](https://rubytalk.org/u/David_Vallner)\ **Post date:** [1 April 2006 20:06 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/4 "2006-04-01T20:06:36Z")
D艌a Sobota 01 Apr铆l 2006 16:39 Patrick Hurley nap铆sal: > **路路路** > > > On 4/1/06, rati\_lion@yahoo.com \ wrote: > > \> Hi all > > \> > > \> I am new to Ruby. Found it as an intersting language. Can anyone help > > \> me with a simple code in Ruby to check for all the dead and live links > > \> in a website ? > > \> > > \> Thanks > > \> Rati > > > > A wonderful language, a sort of rough request here is some (very) > > simple code to get you started, using some great ruby libraries, > > including rubyful soup ([http://www.crummy.com/software/RubyfulSoup/\](http://www.crummy.com/software/RubyfulSoup/%5C)), > > which is available as a gem. As written it only checks one page, you > > would need to make it "walk" the links to recursively check a whole > > site. > > > > Hope it helps > > pth > > > > require 'open-uri' > > require 'uri' > > require 'rubyful\_soup' > > > > url = '[http://www.yahoo.com/'](http://www.yahoo.com/'); > > uri = URI.parse(url) > > html = open(uri).read > > soup = BeautifulSoup.new(html) > > > > #Search the soup > > links = soup.find\_all('a').map { |a| a['href'] } > > > > # Remove javascript > > links.delete\_if { |href| href =~ /javascript/ } > > > > links.each do |l| > >   # resolve relative paths (there is probably a better way) > >   link = URI.parse(l) > >   link.scheme = 'http' unless link.scheme > >   link.host = uri.host unless link.host > >   link.path = uri.path + link.path unless link.path[0] == ?/ > >   link = URI.parse(link.to\_s) > > > >   # check the link > >   begin > >     open(link).read > >     # if we made it here the link is probably good > >   rescue Exception =\> e > >     puts "#{link.to\_s}: #{e.to\_s}" > >   end > > end > > Yay for intuitive and descriptive class and library names, eh? > > David Vallner ---
**Author:** ![Robin\_Stocker1](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@Robin\_Stocker1](https://rubytalk.org/u/Robin_Stocker1)\ **Post date:** [2 April 2006 01:55 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/5 "2006-04-02T01:55:03Z")
Hi, > links.each do |l| >   # resolve relative paths (there is probably a better way) Yes there is, see below 馃檪 >   link = URI.parse(l) >   link.scheme = 'http' unless link.scheme >   link.host = uri.host unless link.host >   link.path = uri.path + link.path unless link.path[0] == ?/ >   link = URI.parse(link.to\_s) link = uri.merge(l)    Robin ---
**Author:** ![Patrick\_Hurley1](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Patrick\_Hurley1](https://rubytalk.org/u/Patrick_Hurley1)\ **Post date:** [2 April 2006 00:41 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/6 "2006-04-02T00:41:01Z")
You are very welcome. And you are always welcome to request help. So I don't come off sounding too harsh, let me explain. This is a friendly community, generally populated by programmers and people striving to become programmers, so when asking for assistance in this manner, you will often receive more assistance when you show that you have spent a little time attempting to do this yourself, or asking for guidance on how to get started. As for the name rubyful\_soup there are other better named libraries, but this is based upon a Python library (same author) called Beautiful Soup -- its claim to fame is that it will handle invalid markup, that will give most other parsers fits (so it is useful in the real world when dealing with other peoples "html"). I knew the library because I had read about it here. Good luck again pth > **路路路** > > On 4/1/06, rati\_lion@yahoo.com \ wrote: > > > Thank you. Will take care in future to not post a rough request 馃檪 > > > > Thanks again for this extended help. ---
**Author:** ![Patrick\_Hurley1](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Patrick\_Hurley1](https://rubytalk.org/u/Patrick_Hurley1)\ **Post date:** [2 April 2006 02:59 UTC](https://rubytalk.org/t/new-to-ruby-need-a-code/26324/7 "2006-04-02T02:59:45Z")
Thanks I knew there had to be :-), I don't use URI much. pth > **路路路** > > On 4/1/06, Robin Stocker \ wrote: > > > Hi, > > > > \> links.each do |l| > > \> # resolve relative paths (there is probably a better way) > > Yes there is, see below 馃檪 > > > > \> link = URI.parse(l) > > \> link.scheme = 'http' unless link.scheme > > \> link.host = uri.host unless link.host > > \> link.path = uri.path + link.path unless link.path[0] == ?/ > > \> link = URI.parse(link.to\_s) > > link = uri.merge(l) > > > >    Robin