# Best way to parse/update HTML file?
**URL:** \
**Category:** ruby-talk\
**Created:** [25 June 2005 01:15 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173 "2005-06-25T01:15:39Z")\
**Posts on this page:** 9\
**Page:** 1
**Author:**  [@Bucco](https://rubytalk.org/u/Bucco)\
**Post date:** [25 June 2005 01:15 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/1 "2005-06-25T01:15:39Z")
Sorry for the newbie question. I am trying to find the best metod for
parsing a HTML file and changinf one tag/item. Unfortunately, REXML
chokes on the file because of the incomplete tags. Completing the tag
is not an option either. What is the best way to find a specific tag
in an html file, change it's text and attribute settings?
Thanks:)
SA
---
**Author:**  [@daz](https://rubytalk.org/u/daz)\
**Post date:** [26 June 2005 09:40 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/2 "2005-06-26T09:40:39Z")
Bucco wrote:
> Sorry for the newbie question.
This has been answered once or twice before by this group 馃槈
> I am trying to find the best metod for parsing a HTML file
> and changinf one tag/item. Unfortunately, REXML chokes on
> the file because of the incomplete tags. Completing the tag
> is not an option either. What is the best way to find a specific
> tag in an html file, change it's text and attribute settings?
>
> Thanks:)
>
> SA
The best way tends to involve using a package although
you /could/ work your way through it using regular expressions.
If you're likely to be doing this kind of thing in the
future, you'll be glad you spent a bit of time installing;
then it's always available.
As I recall, a different package is often
recommended but I don't know which is best.
This is what some of us use:
[http://ruby-htmltools.rubyforge.org/](http://ruby-htmltools.rubyforge.org/) (Ned Konz +)
Examples are included but here's another ...
> **路路路**
>
> #-----------------------------------------------------------------
> EXAMPLE = \<\ \
> \
> \Page title\
> \
> \
> \
Bucco wrote:
> Sorry for the newbie question. I am trying to find the best metod for
> parsing a HTML file and changinf one tag/item. Unfortunately, REXML
> chokes on the file because of the incomplete tags. Completing the tag
> is not an option either. What is the best way to find a specific tag
> in an html file, change it's text and attribute settings?
Hi. I know I'm a bit late to the discussion, so 'sokay if you have an answer already.
A really fantastic HTML parser library is HTree by Tanaka Akira.
\<[http://cvs.m17n.org/~akr/htree/>](http://cvs.m17n.org/~akr/htree/>);
It's completely forgiving of bad HTML and you can import the document into REXML through the HTree parser.
require 'htree'
HTree.parse( "\Bad markup" ).to\_rexml
The only downside is that you'll need to install the iconv library, which can be a bit of a pain to track down on Windows. Other than that, it's a great package.
\_why
---
**Author:**  [@Brad\_Wilson](https://rubytalk.org/u/Brad_Wilson)\
**Post date:** [27 June 2005 19:35 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/5 "2005-06-27T19:35:13Z")
If you're comfortable "cleaning it up", why not tidy it to XHTML then
use the XML parser? This is the approach I took recently when I needed
it.
> **路路路**
>
> On 6/27/05, mathew \ wrote:
>
> > For invalid "tag soup" HTML, your best bet is probably to use
> > html/htmltokenizer.
---
**Author:**  [@daz](https://rubytalk.org/u/daz)\
**Post date:** [29 June 2005 07:00 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/6 "2005-06-29T07:00:42Z")
\_why wrote:
> [snip]
>
> A really fantastic HTML parser library is HTree by Tanaka Akira.
I'm glad you brought that in because I tried it last year and saw
that it was a serious "heavy horse" and perhaps a little bit
\_more\_ than I was looking for.
It was adding XHTML namespace prefixes to all tags, so a
horizontal rule, for example, became:
\<{[XHTML namespace](http://www.w3.org/1999/xhtml)}hr\>
> It's completely forgiving of bad HTML and you can import the
> document into REXML through the HTree parser.
>
> require 'htree'
> HTree.parse( "\Bad markup" ).to\_rexml
It may have been in its early stages of development, but my
assumption that HTree would be too strict is under review 馃檪
Applying your example, I get the result I was expecting
without all that namespace stuff.
> The only downside is that you'll need to install the iconv library,
> which can be a bit of a pain to track down on Windows.
Instead of hunting around for that, I'd made a dummy Ruby version
(no functionality for those who don't need any):
In my old version there are two files which require 'iconv' -
("text.rb" and "encoder.rb") which I changed to:
begin
require 'iconv'
rescue LoadError
require 'htree/iconv\_dummy'
end
Then, add this dummy file as:
lib\ruby\site\_ruby\1.8\htree\iconv\_dummy.rb
> **路路路**
>
> #-------------------------------------------------------------
> class Iconv
>
> ## For testing : Not part of the HTree package ##
> warn "Using dummy iconv lib: #{\_\_FILE\_\_}"
> IC\_DUMMY = true
>
> def Iconv.open(to, from)
> inst = Iconv.new
> block\_given? ? yield(inst) : inst
> end
> def Iconv.iconv(to, from, \*strs)
> strs.join
> end
> def Iconv.conv(to, from, str)
> str
> end
> def Iconv.list
> raise 'No Iconv.list'
> end
> def initialize(to, from)
> end
> def close
> ''
> end
> def iconv(str, strt = 0, len = -1)
> (len and !( len \< 0 )) or len = str.size - strt
> str[strt, len]
> end
>
> module Failure
> def initialize(\*args) # 3
> end
> def success
> end
> def failed
> end
> def inspect
> end
> end
>
> # class InvalidEncoding \< ArgumentError; end
> # class IllegalSequence \< ArgumentError; end
> # class InvalidCharacter \< ArgumentError; end
> # class OutOfRange \< RuntimeError; end
>
> def Iconv.charset\_map
> raise 'No Iconv.charset\_map'
> end
> end
> #-------------------------------------------------------------
>
> > \_why
>
> daz
---
**Author:**  [@Bill\_Guindon1](https://rubytalk.org/u/Bill_Guindon1)\
**Post date:** [9 July 2005 13:17 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/7 "2005-07-09T13:17:27Z")
> Bucco wrote:
>
> \>Sorry for the newbie question. I am trying to find the best metod for
> \>parsing a HTML file and changinf one tag/item. Unfortunately, REXML
> \>chokes on the file because of the incomplete tags. Completing the tag
> \>is not an option either. What is the best way to find a specific tag
> \>in an html file, change it's text and attribute settings?
> \>
> Hi. I know I'm a bit late to the discussion, so 'sokay if you have an
> answer already.
>
> A really fantastic HTML parser library is HTree by Tanaka Akira.
>
> \<[http://cvs.m17n.org/~akr/htree/>](http://cvs.m17n.org/~akr/htree/>);
>
> It's completely forgiving of bad HTML and you can import the document
> into REXML through the HTree parser.
>
> require 'htree'
> HTree.parse( "\Bad markup" ).to\_rexml
>
> The only downside is that you'll need to install the iconv library,
> which can be a bit of a pain to track down on Windows. Other than that,
> it's a great package.
There's a page on the Rails site that covers the iconv installation on Windows:
[http://wiki.rubyonrails.com/rails/show/iconv](http://wiki.rubyonrails.com/rails/show/iconv)
Once I had the iconv.so in a library path, and iconv.dll in
windows\system32, I ran the test-all.rb. Got an error due to a lack
of /dev/null, but that was fixed by creating a dev directory, and
adding an empty 'null' file to it.
Should swap that out to have it point to a temp dir, but with that
setup, all of the htree tests passed.
> **路路路**
>
> On 6/28/05, why the lucky stiff \ wrote:
>
> > \_why
>
> --
> Bill Guindon (aka aGorilla)
---
**Author:**  [@Bucco](https://rubytalk.org/u/Bucco)\
**Post date:** [28 June 2005 00:20 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/8 "2005-06-28T00:20:39Z")
I have a couple of more questions then:
1. I tried the example for the htmltokenizer and got an error around
assert. Where/what is the assert method?
2. What do you mean by "slurp" in the rest of the text?
3. Any better examples how to use htmltokenizer?
Thanks:)
SA
---
**Author:**  [@mathew](https://rubytalk.org/u/mathew)\
**Post date:** [28 June 2005 16:25 UTC](https://rubytalk.org/t/best-way-to-parse-update-html-file/19173/9 "2005-06-28T16:25:41Z")
Bucco wrote:
> 1. I tried the example for the htmltokenizer and got an error around
> assert. Where/what is the assert method?
An error around "assert" is likely an internal error of some kind. Assertions are pieces of code placed in software to detect invalid arguments to methods, internal data structure inconsistencies, and so on.
For example, consider the Ruby URI library. It doesn't support all kinds of URI. So, it would be a good idea if it were to assert that the URI it is being passed is one of the kinds it actually knows how to parse. That way, someone innocently using the library with the wrong kind of URI will discover the problem immediately, rather than being passed back bad data, or having some bizarre error occur in the middle of the library code.
So it could be that you're passing an invalid argument to a method of htmltokenizer. It's also possible that you're triggering a bug in the library.
> 2. What do you mean by "slurp" in the rest of the text?
"slurp" meaning "pull in the entire content of the file from the current file pointer onwards, without performing any processing on it".
As in file = File.new("something.gif")
data = file.read # slurp!
\ 3. Any better examples how to use htmltokenizer?
require 'html/htmltokenizer'
#[...]
# Parse all the images and links out of the web page
tokenizer = HTMLTokenizer.new(@body)
@images = Array.new
@links = Array.new
lastlink = ''
while tag = tokenizer.getTag('img', 'a')
if tag.tag\_name == 'img'
url = tag.attr\_hash['src']
uri = @uri.merge(url)
@images.push([uri.to\_s, lastlink])
else
url = tag.attr\_hash['href']
uri = @uri.merge(url)
@links.push(uri.to\_s)
lastlink = uri.to\_s
end
end
That's the only time I've used it, I'm afraid. Still, it might give you some ideas.
mathew