\
**Category:** ruby-talk\
**Created:** [11 July 2006 19:55 UTC](https://rubytalk.org/t/decent-html-parser/28863 "2006-07-11T19:55:05Z")\
**Posts on this page:** 18\
**Page:** 1
**Author:**  [@Kevin\_Weller](https://rubytalk.org/u/Kevin_Weller)\
**Post date:** [11 July 2006 19:55 UTC](https://rubytalk.org/t/decent-html-parser/28863/1 "2006-07-11T19:55:05Z")
Anybody have experience with a decent HTML parser for a Ruby application? I've looked around, and so far everything I've found is either unfinished, unstable, [relatively] undocumented, or just plain ugly in terms of API.
I'd like a parser that can take a partial HTML file and return an easily-traversable data structure, in the same order that the elements appear in the file. I don't want or need a callback mechanism, only something I can iterate and tree-search. Though I don't hold much hope it will work, I will try using REXML on my text and see what it produces...results to be posted here. Thanks in advance!
> **···**
>
> --
> Kevin Weller
> Information Technology Crucible
> [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Bruno\_Celeste1](https://rubytalk.org/u/Bruno_Celeste1)\
**Post date:** [11 July 2006 20:00 UTC](https://rubytalk.org/t/decent-html-parser/28863/2 "2006-07-11T20:00:13Z")
You can check rubyful soup library at
> **[Rubyful Soup: "The brush has got entangled in it!"](https://www.crummy.com/software/RubyfulSoup/)**
>
> Rubyful Soup: an HTML parser for Ruby, optimized for screen-scraping.
> **···**
>
> On 7/11/06, Kevin Weller \<"[http://www.itcrucible.com/contact](http://www.itcrucible.com/contact)"@ruby-lang.org\> wrote:
>
> > Anybody have experience with a decent HTML parser for a Ruby
> > application? I've looked around, and so far everything I've found is
> > either unfinished, unstable, [relatively] undocumented, or just plain
> > ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and return an
> > easily-traversable data structure, in the same order that the elements
> > appear in the file. I don't want or need a callback mechanism, only
> > something I can iterate and tree-search. Though I don't hold much hope
> > it will work, I will try using REXML on my text and see what it
> > produces...results to be posted here. Thanks in advance!
> >
> > --
> > Kevin Weller
> > Information Technology Crucible
> > [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Kenosis](https://rubytalk.org/u/Kenosis)\
**Post date:** [11 July 2006 21:45 UTC](https://rubytalk.org/t/decent-html-parser/28863/3 "2006-07-11T21:45:12Z")
To help us narrow things down, can you tell us which Ruby HTML mods
you've tried out?
Ken
Kevin Weller wrote:
> **···**
>
> > Anybody have experience with a decent HTML parser for a Ruby
> > application? I've looked around, and so far everything I've found is
> > either unfinished, unstable, [relatively] undocumented, or just plain
> > ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and return an
> > easily-traversable data structure, in the same order that the elements
> > appear in the file. I don't want or need a callback mechanism, only
> > something I can iterate and tree-search. Though I don't hold much hope
> > it will work, I will try using REXML on my text and see what it
> > produces...results to be posted here. Thanks in advance!
> >
> > --
> > Kevin Weller
> > Information Technology Crucible
> > [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@7rans](https://rubytalk.org/u/7rans)\
**Post date:** [11 July 2006 22:06 UTC](https://rubytalk.org/t/decent-html-parser/28863/4 "2006-07-11T22:06:43Z")
Kevin Weller wrote:
> Anybody have experience with a decent HTML parser for a Ruby
> application? I've looked around, and so far everything I've found is
> either unfinished, unstable, [relatively] undocumented, or just plain
> ugly in terms of API.
>
> I'd like a parser that can take a partial HTML file and return an
> easily-traversable data structure, in the same order that the elements
> appear in the file. I don't want or need a callback mechanism, only
> something I can iterate and tree-search. Though I don't hold much hope
> it will work, I will try using REXML on my text and see what it
> produces...results to be posted here. Thanks in advance!
I also metion Facets' tagiterator.rb (from Ⴗnyasu's tagiter.rb) For
example:
a = TagIterator.new(stext)
a.first("body") do |y|
y.nth("dl",2) do |dl|
dl.enumtag("dt") do |t|
puts t.text.strip
end
end
end
[http://facets.rubyforge.org/api/more/classes/TagIterator.html](http://facets.rubyforge.org/api/more/classes/TagIterator.html)
T.
---
**Author:**  [@Geoff\_Davis](https://rubytalk.org/u/Geoff_Davis)\
**Post date:** [11 July 2006 22:15 UTC](https://rubytalk.org/t/decent-html-parser/28863/5 "2006-07-11T22:15:10Z")
You might find Rubyful Soup useful:
> **[Rubyful Soup: "The brush has got entangled in it!"](https://www.crummy.com/software/RubyfulSoup/)**
>
> Rubyful Soup: an HTML parser for Ruby, optimized for screen-scraping.
It's a bit slow, but it is quite robust to bad HTML.
> **···**
>
> On Tue, 11 Jul 2006 13:52:13 -0600, Kevin Weller wrote:
>
> > Anybody have experience with a decent HTML parser for a Ruby
> > application? I've looked around, and so far everything I've found is
> > either unfinished, unstable, [relatively] undocumented, or just plain
> > ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and return an
> > easily-traversable data structure, in the same order that the elements
> > appear in the file. I don't want or need a callback mechanism, only
> > something I can iterate and tree-search. Though I don't hold much hope
> > it will work, I will try using REXML on my text and see what it
> > produces...results to be posted here. Thanks in advance!
---
**Author:**  [@Assaf](https://rubytalk.org/u/Assaf)\
**Post date:** [12 July 2006 08:10 UTC](https://rubytalk.org/t/decent-html-parser/28863/6 "2006-07-12T08:10:11Z")
Kevin,
I settled on using Tidy to clean up the HTML, then parsing it into a
tree using the HTML scanner that comes with Rails.
Tidy does all the hard stuff of dealing with bad HTML and straightening
it up. The HTML scanner is very lightweight and has a simple, clean
API. You don't need to run Rails, just require the scanner library
(look for html/document.rb).
It's two passes, but with Tidy being C++ and HTML scanner doing no
cleanup, it's amazingly fast. I'm processing around 500Kb/s (mobile Duo
Core 1.8GHz).
You can walk the DOM, or use XPath-like finders, or my preferred method
of looking up content: using CSS selectors.
If you're doing HTML scraping this library will do all the hard work
for you:
[http://blog.labnotes.org/2006/07/11/scraping-with-style-scrapi-toolkit-for-ruby/](http://blog.labnotes.org/2006/07/11/scraping-with-style-scrapi-toolkit-for-ruby/)
Assaf
> **[Labnotes (by Assaf Arkin)](https://labnotes.org)**
>
> 🔥 All about software design, development, and culture. A mix of insightful and funny. Goes great with coffee. ☕
Kevin Weller wrote:
> **···**
>
> > Anybody have experience with a decent HTML parser for a Ruby
> > application? I've looked around, and so far everything I've found is
> > either unfinished, unstable, [relatively] undocumented, or just plain
> > ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and return an
> > easily-traversable data structure, in the same order that the elements
> > appear in the file. I don't want or need a callback mechanism, only
> > something I can iterate and tree-search. Though I don't hold much hope
> > it will work, I will try using REXML on my text and see what it
> > produces...results to be posted here. Thanks in advance!
> >
> > --
> > Kevin Weller
> > Information Technology Crucible
> > [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Seth\_Thomas\_Rasmusse](https://rubytalk.org/u/Seth_Thomas_Rasmusse)\
**Post date:** [12 July 2006 13:10 UTC](https://rubytalk.org/t/decent-html-parser/28863/7 "2006-07-12T13:10:10Z")
It was just released recently, but Hpricot might be worth a look:
[http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html](http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html)
Kevin Weller wrote:
> **···**
>
> > Anybody have experience with a decent HTML parser for a Ruby
> > application? I've looked around, and so far everything I've found is
> > either unfinished, unstable, [relatively] undocumented, or just plain
> > ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and return an
> > easily-traversable data structure, in the same order that the elements
> > appear in the file. I don't want or need a callback mechanism, only
> > something I can iterate and tree-search. Though I don't hold much hope
> > it will work, I will try using REXML on my text and see what it
> > produces...results to be posted here. Thanks in advance!
> >
> > --
> > Kevin Weller
> > Information Technology Crucible
> > [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Rob](https://rubytalk.org/u/Rob)\
**Post date:** [13 July 2006 16:39 UTC](https://rubytalk.org/t/decent-html-parser/28863/8 "2006-07-13T16:39:48Z")
If you find the Ruby options don't cut it, then I recommend TagSoup
which is implemented in Java but "keeps on trucking" to convert HTML
found in the wild to well-formed XML files:
[http://home.ccil.org/~cowan/XML/tagsoup/](http://home.ccil.org/~cowan/XML/tagsoup/)
I use it when screenscraping, then load the resulting XML files with REXML.
Rob
> **···**
>
> On 7/11/06, Kevin Weller \<"[http://www.itcrucible.com/contact](http://www.itcrucible.com/contact)"@ruby-lang.org\> wrote:
>
> > Anybody have experience with a decent HTML parser for
> > a Ruby application? I've looked around, and so far
> > everything I've found is
> > either unfinished, unstable, [relatively] undocumented,
> > or just plain ugly in terms of API.
> >
> > I'd like a parser that can take a partial HTML file and
> > return an easily-traversable data structure, in the same
> > order that the elements appear in the file.
---
**Author:**  [@Alex\_Young](https://rubytalk.org/u/Alex_Young)\
**Post date:** [11 July 2006 20:10 UTC](https://rubytalk.org/t/decent-html-parser/28863/9 "2006-07-11T20:10:58Z")
Bruno Celeste wrote:
> You can check rubyful soup library at
> [Rubyful Soup: "The brush has got entangled in it!"](http://www.crummy.com/software/RubyfulSoup/)
... and if that doesn't help, Tidy + REXML does fine for me.
> **···**
>
> --
> Alex
>
> > On 7/11/06, Kevin Weller \> \<"[http://www.itcrucible.com/contact](http://www.itcrucible.com/contact)"@ruby-lang.org\> wrote:
> >
> > > Anybody have experience with a decent HTML parser for a Ruby
> > > application? I've looked around, and so far everything I've found is
> > > either unfinished, unstable, [relatively] undocumented, or just plain
> > > ugly in terms of API.
> > >
> > > I'd like a parser that can take a partial HTML file and return an
> > > easily-traversable data structure, in the same order that the elements
> > > appear in the file. I don't want or need a callback mechanism, only
> > > something I can iterate and tree-search. Though I don't hold much hope
> > > it will work, I will try using REXML on my text and see what it
> > > produces...results to be posted here. Thanks in advance!
> > >
> > > --
> > > Kevin Weller
> > > Information Technology Crucible
> > > [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Phillip\_Hutchings](https://rubytalk.org/u/Phillip_Hutchings)\
**Post date:** [11 July 2006 21:51 UTC](https://rubytalk.org/t/decent-html-parser/28863/10 "2006-07-11T21:51:30Z")
I'll second Rubyful Soup. It's not the fastest, but it tolerates bad
HTML. I used it for an intranet spider, it worked with anything I
could find.
> **···**
>
> On 7/12/06, Bruno Celeste \ wrote:
>
> > You can check rubyful soup library at
> > [Rubyful Soup: "The brush has got entangled in it!"](http://www.crummy.com/software/RubyfulSoup/)
>
> --
> Phillip Hutchings
> [http://www.sitharus.com/](http://www.sitharus.com/)
---
**Author:**  [@Kevin\_Weller](https://rubytalk.org/u/Kevin_Weller)\
**Post date:** [12 July 2006 00:50 UTC](https://rubytalk.org/t/decent-html-parser/28863/11 "2006-07-12T00:50:11Z")
Thanks for the reply. I've basically reviewed every potential match generated by:
[http://raa.ruby-lang.org/search.rhtml?search=html+parser](http://raa.ruby-lang.org/search.rhtml?search=html+parser)
I've since tried out a couple, and the option that seems to work best so far is Ned Konz' ruby-htmltools. Unfortunately, it does not seem to parse partial HTML documents well, so I've had to resort to parsing the whole thing, extracting a REXML document object, then using XPath to get to the content I care about. Seems like a waste of processing power when I can get the necessary markup in text with a simple file.grep operation and (theoretically) parse only the text that I want, but at least I have something that works until/unless something better comes along. Any recommendations?
Kenosis wrote:
> **···**
>
> > To help us narrow things down, can you tell us which Ruby HTML mods
> > you've tried out?
> >
> > Ken
> >
> > Kevin Weller wrote:
> >
> > > Anybody have experience with a decent HTML parser for a Ruby
> > > application? I've looked around, and so far everything I've found is
> > > either unfinished, unstable, [relatively] undocumented, or just plain
> > > ugly in terms of API.
---
**Author:**  [@Kevin\_Weller](https://rubytalk.org/u/Kevin_Weller)\
**Post date:** [12 July 2006 00:55 UTC](https://rubytalk.org/t/decent-html-parser/28863/12 "2006-07-12T00:55:05Z")
Geoff Davis wrote:
> **···**
>
> > On Tue, 11 Jul 2006 13:52:13 -0600, Kevin Weller wrote:
> >
> > > Anybody have experience with a decent HTML parser for a Ruby application? I've looked around, and so far everything I've found is either unfinished, unstable, [relatively] undocumented, or just plain ugly in terms of API.
> > > --- snip ---
> >
> > You might find Rubyful Soup useful:
> >
> > [Rubyful Soup: "The brush has got entangled in it!"](http://www.crummy.com/software/RubyfulSoup/)
> >
> > It's a bit slow, but it is quite robust to bad HTML.
>
> Ooooh, thanks, that might be just what the doctor ordered...especially if it handles a single text line of an HTML document. Right now I have a temporary solution that involves using ruby-htmltools to parse the entire document, then finding the part that I want with an XPath query. However, Rubyful Soup might turn out to be a better performer if it does what I want. Thanks so much!
---
**Author:**  [@Pere\_Noel1](https://rubytalk.org/u/Pere_Noel1)\
**Post date:** [12 July 2006 10:10 UTC](https://rubytalk.org/t/decent-html-parser/28863/13 "2006-07-12T10:10:05Z")
except it doesn't wipe out the charset even when it's wrong (afaik)
> **···**
>
> assaf.arkin@gmail.com \ wrote:
>
> > Tidy does all the hard stuff of dealing with bad HTML and straightening
> > it up.
>
> --
> une bévue
---
**Author:**  [@Kevin\_Weller](https://rubytalk.org/u/Kevin_Weller)\
**Post date:** [13 July 2006 15:25 UTC](https://rubytalk.org/t/decent-html-parser/28863/14 "2006-07-13T15:25:11Z")
That sounds like a very clean solution for future needs. I dig into Rails NEXT week for an upcoming Rails-based project (this one isn't)...nice to know what my options will be then. Thanks for the info!
> **···**
>
> assaf.arkin@gmail.com wrote:
>
> > Kevin,
> >
> > I settled on using Tidy to clean up the HTML, then parsing it into a
> > tree using the HTML scanner that comes with Rails.
>
> --
> Kevin Weller
> Information Technology Crucible
> [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Kevin\_Weller](https://rubytalk.org/u/Kevin_Weller)\
**Post date:** [13 July 2006 15:30 UTC](https://rubytalk.org/t/decent-html-parser/28863/15 "2006-07-13T15:30:04Z")
Seth Thomas Rasmussen wrote:
> It was just released recently, but Hpricot might be worth a look:
>
> [http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html](http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html)
Thanks Seth! Hpricot looks a little young, but promising. It apparently parses incomplete HTML documents, don't know how it handles imbalanced tags. I'll check it out...thanks again...
> **···**
>
> --
> Kevin Weller
> Information Technology Crucible
> [http://www.itcrucible.com](http://www.itcrucible.com)
---
**Author:**  [@Ezra\_Zygmuntowicz](https://rubytalk.org/u/Ezra_Zygmuntowicz)\
**Post date:** [11 July 2006 22:06 UTC](https://rubytalk.org/t/decent-html-parser/28863/16 "2006-07-11T22:06:08Z")
Have a look at Hpricot, \_why's new ruby/C html parser. Its fast and has nice features.
[http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html](http://redhanded.hobix.com/inspect/okayGiveHpricot02AGo.html)
-Ezra
> **···**
>
> On Jul 11, 2006, at 2:51 PM, Phillip Hutchings wrote:
>
> > On 7/12/06, Bruno Celeste \ wrote:
> >
> > > You can check rubyful soup library at
> > > [Rubyful Soup: "The brush has got entangled in it!"](http://www.crummy.com/software/RubyfulSoup/)
> >
> > I'll second Rubyful Soup. It's not the fastest, but it tolerates bad
> > HTML. I used it for an intranet spider, it worked with anything I
> > could find.
> > -- Phillip Hutchings
> > [http://www.sitharus.com/](http://www.sitharus.com/)
---
**Author:**  [@Alex\_Young](https://rubytalk.org/u/Alex_Young)\
**Post date:** [12 July 2006 10:58 UTC](https://rubytalk.org/t/decent-html-parser/28863/17 "2006-07-12T10:58:14Z")
Une bévue wrote:
> **···**
>
> > assaf.arkin@gmail.com \ wrote:
> >
> > > Tidy does all the hard stuff of dealing with bad HTML and straightening
> > > it up.
> >
> > except it doesn't wipe out the charset even when it's wrong (afaik)
>
> I've had it generate an incorrect xml processing instruction once before, and that's a simple gsub! to fix.
>
> --
> Alex
---
**Author:**  [@why\_the\_lucky\_stiff1](https://rubytalk.org/u/why_the_lucky_stiff1)\
**Post date:** [13 July 2006 17:58 UTC](https://rubytalk.org/t/decent-html-parser/28863/18 "2006-07-13T17:58:10Z")
=\> #\
{elem \ {text "A test of "} {elem \ {text "unbalanced"}} \}
{text " tags."}
{bogusetag \}}\>
Hpricot will attempt to match tags, keeping around the unmatched end tags as
BogusEtag objects (as HTree does.) When you output XHTML, these bogus tags are
ignored and you'll get back XHTML (though some work still has to be done to
be sure it's truly valid XHTML.)
Thankfully, I've been getting some really mangled HTML from interested people
sent to my inbox. So today's Hpricot has become much better at fixing all
manner of nutty tags. If you encounter anything Hpricot can't do, send me an
e-mail and with the URL you're trying to parse and the results you expect.
\_why
> **···**
>
> On Fri, Jul 14, 2006 at 12:30:04AM +0900, Kevin Weller wrote:
>
> > Thanks Seth! Hpricot looks a little young, but promising. It
> > apparently parses incomplete HTML documents, don't know how it handles
> > imbalanced tags. I'll check it out...thanks again...
>
> \>\> pp Hpricot("\\A test of \unbalanced\ tags.\")