# Nokogiri help parsing HTML

**URL:** <https://rubytalk.org/t/nokogiri-help-parsing-html/67586>\
**Category:** ruby-talk\
**Created:** [26 March 2013 22:40 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586 "2013-03-26T22:40:00Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [26 March 2013 22:40 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/1 "2013-03-26T22:40:00Z")

</div>

I'm relatively new to Ruby (and therefore Nokogiri) and am trying to  
parse some HTML that will ultimately be written to a MySQL database. In  
the interim, I'm writing it to a text file for troubleshooting purposes.

Here's the relevant piece of the HTML I'd like to parse:

\<!-- body="start" --\>  
\<div class="mail"\>  
\<address class="headers"\>  
\<span id="from"\>  
\<dfn\>From\</dfn\>: Paul David Mena &lt;\<a  
href="mailto:pauldavidmena\_at\_gmail.com?Subject=Re:%20twilight"\>pauldavidmena\_at\_gmail.com\</a\>&gt;  
\</span\>\<br /\>  
\<span id="date"\>\<dfn\>Date\</dfn\>: Tue, 26 Mar 2013 18:13:21  
-0400\</span\>\<br /\>  
\</address\>  
\<p\>  
Line 1  
\<br /\>  
Line 2  
\<br /\>  
Line 3  
\<br /\>  
\<p\>\<pre\>

> **···**
>
> --  
> Paul David Mena  
> --------------------  
> pauldavidmena\_at\_gmail&#46;\<!--nospam--\>com  
> \</pre\>  
> \<span id="received"\>\<dfn\>Received on\</dfn\> Tue Mar 26 2013 - 22:13:23  
> EDT\</span\>  
> \</div\>  
> \<!-- body="end" --\>
> 
> My goal is to strip out everything between the "address" and "pre" tags  
> and to output only:
> 
> Line 1
> 
> Line 2
> 
> Line 3
> 
> My code, however, is stripping out one or the other, depending upon  
> where I place the definition. Here is the code:
> 
> #!/usr/bin/env ruby
> 
> require "nokogiri"
> 
> class PlainTextExtractor \< Nokogiri::XML::SAX::Document  
> &nbsp;&nbsp;attr\_reader :plaintext  
> &nbsp;&nbsp;# Initialize the state of interest variable with false  
> &nbsp;&nbsp;def initialize  
> &nbsp;&nbsp;&nbsp;&nbsp;@interesting = false  
> &nbsp;&nbsp;&nbsp;&nbsp;@pre = false  
> &nbsp;&nbsp;&nbsp;&nbsp;@address = false  
> &nbsp;&nbsp;&nbsp;&nbsp;@plaintext = ""  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def start\_element(name, attrs = [])  
> &nbsp;&nbsp;&nbsp;&nbsp;if name == "address"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@address = true  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def end\_element(name, attrs = [])  
> &nbsp;&nbsp;&nbsp;&nbsp;if name == "address"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@address = false  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def start\_element(name, attrs = [])  
> &nbsp;&nbsp;&nbsp;&nbsp;if name == "pre"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@pre = true  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def end\_element(name, attrs = [])  
> &nbsp;&nbsp;&nbsp;&nbsp;if name == "pre"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@pre = false  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;# This method is called whenever a comment occurs and  
> &nbsp;&nbsp;# the comments text is passed in as string.  
> &nbsp;&nbsp;def comment(string)  
> &nbsp;&nbsp;&nbsp;&nbsp;case string.strip # strip leading and trailing whitespaces  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;when /^body="start"/ # match starting comment  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@interesting = true  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;when /^body="end"/  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@interesting = false # match closing comment  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;# This callback method is called with any string between  
> &nbsp;&nbsp;# a tag.  
> &nbsp;&nbsp;def characters(string)  
> &nbsp;&nbsp;&nbsp;&nbsp;if @interesting and not @pre  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if @interesting and not @address  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@plaintext \<\< string  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end  
> end
> 
> fname = ARGV[0]  
> start\_column = 4  
> end\_column = 6
> 
> target\_range = (start\_column-1)..(end\_column-1)  
> IO.foreach(fname) do |line|  
> &nbsp;&nbsp;if line.match(/\<dfn\>Date\<\/dfn\>/)  
> &nbsp;&nbsp;&nbsp;&nbsp;pieces = line.split(" ")
> 
> &nbsp;&nbsp;&nbsp;&nbsp;@date\_string = pieces[target\_range].join("-")  
> # puts @date\_string  
> &nbsp;&nbsp;end  
> end
> 
> pte = PlainTextExtractor.new  
> parser = Nokogiri::HTML::SAX::Parser.new(pte)  
> parser.parse\_file ARGV[0]
> 
> # puts pte.plaintext
> 
> begin  
> &nbsp;&nbsp;file = File.open("snippet.txt", "w")  
> &nbsp;&nbsp;file.write(@date\_string)  
> &nbsp;&nbsp;file.write(pte.plaintext)  
> rescue IOError =\> e  
> &nbsp;&nbsp;#some error occur, dir not writable etc.  
> ensure  
> &nbsp;&nbsp;file.close unless file == nil  
> end
> 
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Jesus\_Gabriel\_y\_Gala](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@Jesus\_Gabriel\_y\_Gala](https://rubytalk.org/u/Jesus_Gabriel_y_Gala)\
**Post date:** [26 March 2013 23:05 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/2 "2013-03-26T23:05:17Z")

</div>

OK, so you want every tag that is a sibling of address and pre and is  
within those two. I have found this StackOverFlow answer:

> <https://stackoverflow.com/questions/6554261/xpath-expression-select-elements-between-a-href-expr-tags/6555376#6555376>

which applied to your problem:

1.9.2p290 :001 \> require 'nokogiri'  
=\> true  
1.9.2p290 :002 \> s = \<\<END  
1.9.2p290 :003"\> \<!-- body="start" --\>  
1.9.2p290 :004"\> \<div class="mail"\>  
[...snip...]  
1.9.2p290 :031 \> doc = Nokogiri::HTML(s)  
1.9.2p290 :039 \>  
doc.xpath("//address/following-sibling::node()[count(.|  
//pre/preceding-sibling::node())=count(//pre/preceding-sibling::node())]")  
=\> [#\<Nokogiri::XML::Text:0xdb0114 "\n"\>,  
#\<Nokogiri::XML::Element:0xdaff84 name="p"  
children=[#\<Nokogiri::XML::Text:0xdafc28 "\nLine 1\n"\>,  
#\<Nokogiri::XML::Element:0xdafa5c name="br"\>,  
#\<Nokogiri::XML::Text:0xdaf728 "\nLine 2\n"\>,  
#\<Nokogiri::XML::Element:0xdaf5ac name="br"\>,  
#\<Nokogiri::XML::Text:0xdaf228 "\nLine 3\n"\>,  
#\<Nokogiri::XML::Element:0xdaf0ac name="br"\>]\>,  
#\<Nokogiri::XML::Element:0xdaea80 name="p"\>]

will return a node set that contains the required nodes.

Hope this helps,

Jesus.

> **···**
>
> On Tue, Mar 26, 2013 at 11:40 PM, Paul Mena \<lists@ruby-forum.com\> wrote:
> 
> > I'm relatively new to Ruby (and therefore Nokogiri) and am trying to  
> > parse some HTML that will ultimately be written to a MySQL database. In  
> > the interim, I'm writing it to a text file for troubleshooting purposes.
> > 
> > Here's the relevant piece of the HTML I'd like to parse:
> > 
> > \<!-- body="start" --\>  
> > \<div class="mail"\>  
> > \<address class="headers"\>  
> > \<span id="from"\>  
> > \<dfn\>From\</dfn\>: Paul David Mena &lt;\<a  
> > href="mailto:pauldavidmena\_at\_gmail.com?Subject=Re:%20twilight"\>pauldavidmena\_at\_gmail.com\</a\>&gt;  
> > \</span\>\<br /\>  
> > \<span id="date"\>\<dfn\>Date\</dfn\>: Tue, 26 Mar 2013 18:13:21  
> > -0400\</span\>\<br /\>  
> > \</address\>  
> > \<p\>  
> > Line 1  
> > \<br /\>  
> > Line 2  
> > \<br /\>  
> > Line 3  
> > \<br /\>  
> > \<p\>\<pre\>  
> > --  
> > Paul David Mena  
> > --------------------  
> > pauldavidmena\_at\_gmail&#46;\<!--nospam--\>com  
> > \</pre\>  
> > \<span id="received"\>\<dfn\>Received on\</dfn\> Tue Mar 26 2013 - 22:13:23  
> > EDT\</span\>  
> > \</div\>  
> > \<!-- body="end" --\>
> > 
> > My goal is to strip out everything between the "address" and "pre" tags  
> > and to output only:

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 14:57 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/3 "2013-03-27T14:57:26Z")

</div>

Thanks to all for the help. The following seems to do most of what I  
want:

dom.xpath('//address/following-sibling::p//text()').each {|n| p n}

My next task is to capture only the desired text, and to write it to a  
file. Specifically:

Line 1\n  
Line 2\n  
Line 3\n

The above code writes the following to standard out:

#\<Nokogiri::XML::Text:0xba8f9c "\nLine 1\n"\>  
#\<Nokogiri::XML::Text:0xba8eac "\nLine 2\n"\>  
#\<Nokogiri::XML::Text:0xba8dd0 "\nLine 3\n"\>

So close!

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 15:18 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/4 "2013-03-27T15:18:15Z")

</div>

Isn't that as simple as "p n.text"?

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 16:44 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/5 "2013-03-27T16:44:59Z")

</div>

I should probably include the whole revised program for context. The  
argument is the path to an HTML file that contains relevant text between  
the body=start and body=end tags.

#!/usr/bin/env ruby

# some initializations

@interesting = false  
@my\_text = ""

# read the file between the two "body" tags and stash in "my\_text"

fname = ARGV[0]

IO.foreach(fname) do |line|  
&nbsp;&nbsp;if line.match(/body="start"/)  
&nbsp;&nbsp;&nbsp;&nbsp;@interesting = true  
&nbsp;&nbsp;end

# meanwhile let's grab the date string and process it

&nbsp;&nbsp;start\_column = 4  
&nbsp;&nbsp;end\_column = 6

&nbsp;&nbsp;target\_range = (start\_column-1)..(end\_column-1)

&nbsp;&nbsp;if line.match(/\<dfn\>Date\<\/dfn\>/)  
&nbsp;&nbsp;&nbsp;&nbsp;pieces = line.split(" ")  
&nbsp;&nbsp;&nbsp;&nbsp;@date\_string = pieces[target\_range].join("-")  
&nbsp;&nbsp;end

&nbsp;&nbsp;if line.match(/body="end"/)  
&nbsp;&nbsp;&nbsp;&nbsp;@interesting = false  
&nbsp;&nbsp;end

&nbsp;&nbsp;if @interesting  
&nbsp;&nbsp;&nbsp;&nbsp;@my\_text \<\< line  
&nbsp;&nbsp;end  
end  
# puts @haiku\_text

require "nokogiri"

doc = Nokogiri::HTML(@my\_text)  
doc.xpath('//address/following-sibling::p//text()').each {|n| p n}

# puts doc

begin  
&nbsp;&nbsp;file = File.open("snippet.txt", "w")  
&nbsp;&nbsp;file.write(@date\_string)  
&nbsp;&nbsp;file.write(doc)  
rescue IOError =\> e  
&nbsp;&nbsp;#some error occur, dir not writable etc.  
ensure  
&nbsp;&nbsp;file.close unless file == nil  
end

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 16:56 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/6 "2013-03-27T16:56:49Z")

</div>

How about this?

doc = Nokogiri::HTML(@my\_text)

output = @date\_string + $/  
doc.xpath('//address/following-sibling::p//text()').each { |line| output  
\<\< ( line.text.strip + $/ ) }

File.write("snippet.txt", output)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 17:55 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/7 "2013-03-27T17:55:27Z")

</div>

Just out of curiosity I had a go at writing this myself, with the  
exception of that complicated xpath because I don't really understand  
xpath yet 🙂

This is what I came up with:

require 'nokogiri'  
doc = Nokogiri::HTML File.read(ARGV[0])  
output = doc.css('span[@id="date"]').first.text[/\d+ \w+ \d+/].gsub('  
','-') + $/  
path = '//address/following-sibling::p//text()'  
doc.xpath(path).each { |line| output \<\< line.text.strip \<\< $/ }  
File.write("snippet.txt", output)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [27 March 2013 18:05 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/8 "2013-03-27T18:05:52Z")

</div>

This actually worked perfectly:

require "nokogiri"

doc = Nokogiri::HTML(@my\_text)  
output = @date\_string + $/  
doc.xpath('//address/following-sibling::p//text()').each { |line| output  
\<\< ( line.text.strip + $/ ) }

File.write("snippet.txt", output)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [28 March 2013 08:35 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/9 "2013-03-28T08:35:34Z")

</div>

You still have @my\_text and @date\_string there, which leads me to  
suspect that's only the last part of your script.  
The example I gave is the entire script...

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 09:28 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/10 "2013-03-29T09:28:53Z")

</div>

Since you're using Nokogiri anyway, you'd be better off using that for  
the whole process rather than looping through the HTML "manually". This  
is the sort of thing it's worth getting in the habit of: using the tools  
available to their fullest potential.  
You can do the whole thing in 5 lines (barring error-checking) as I  
demonstrated earlier.

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 16:39 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/11 "2013-03-29T16:39:29Z")

</div>

That makes quite a difference! Here's what I have after ripping out the  
old logic and extracting the date using Nokogiri:

#!/usr/bin/env ruby

require "nokogiri"

# get the date

doc = Nokogiri::HTML File.read(ARGV[0])  
output = doc.css('span[@id="date"]').first.text[/\d+ \w+ \d+/].gsub('  
','-') + $/

# get the remaining text

path = '//address/following-sibling::p//text()'  
doc.xpath(path).each { |line| output \<\< line.text.strip \<\< $/ }

File.write("snippet.txt", output)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 16:50 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/12 "2013-03-29T16:50:37Z")

</div>

Looks good, Ruby's pretty amazing when it comes to finding simple ways  
to do complex things.  
Are there any parts of that code you need clarifying? It'll help to  
understand all the methods used here so you can write your own more  
easily in future.  
For example, you can test regular expressions here:  
[http://www.rubular.com/](http://www.rubular.com/)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 16:55 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/13 "2013-03-29T16:55:16Z")

</div>

It really helped to run the code in IRB to see what Nokogiri was doing.  
It will take me a little bit longer to wrap my mind around how Ruby does  
regular expressions, but it certainly seems worth the effort.

Thanks for the link, and for all of the help!

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 22:49 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/14 "2013-03-29T22:49:46Z")

</div>

I do have a follow-up question, if you don't mind. I can see how the  
"address" tag is stripped, but not the "pre" tag. Amazing how much  
heavy lifting is accomplished with a simple (or, at least to me, not so  
simple) line of code.

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 23:02 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/15 "2013-03-29T23:02:54Z")

</div>

I'm not sure what you mean by stripping the tags. Firstly, the xpath is  
looking for \<p\> after \<address\>, which doesn't include \<pre\> in your  
example. Secondly if you ask Nokogiri for the "text", it won't include  
any html tags.

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\
**Post date:** [27 March 2013 08:25 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/16 "2013-03-27T08:25:00Z")

</div>

Your version also outputs \<p\> tags, doesn't it? A modified version of yours

irb(main):036:0\>  
dom.xpath('//address/following-sibling::\*//text()').each {|n| p n}  
#\<Nokogiri::XML::Text:0x..fc01de63e "\nLine 1\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de4d6 "\nLine 2\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de36e "\nLine 3\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de206 "\n"\>  
#\<Nokogiri::XML::Text:0x..fc01ddf04 "\n"\>  
=\> 0

irb(main):037:0\>  
dom.xpath('//address/following-sibling::p//text()').each {|n| p n}  
#\<Nokogiri::XML::Text:0x..fc01de63e "\nLine 1\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de4d6 "\nLine 2\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de36e "\nLine 3\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de206 "\n"\>  
=\> 0

Here's another approach: find everything under \<div class="mail"\> but  
not under \<address\>:

irb(main):032:0\>  
dom.xpath('//div[@class="mail"]//text()[not(ancestor::address)]').each  
{|n| p n}  
#\<Nokogiri::XML::Text:0x..fc01c07c4 "\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de7b0 "\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de63e "\nLine 1\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de4d6 "\nLine 2\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de36e "\nLine 3\n"\>  
#\<Nokogiri::XML::Text:0x..fc01de206 "\n"\>  
#\<Nokogiri::XML::Text:0x..fc01ddf04 "\n"\>  
=\> 0

TIMTOWTDI 🙂

Kind regards

robert

> **···**
>
> On Wed, Mar 27, 2013 at 12:05 AM, Jesús Gabriel y Galán \<jgabrielygalan@gmail.com\> wrote:
> 
> > On Tue, Mar 26, 2013 at 11:40 PM, Paul Mena \<lists@ruby-forum.com\> wrote:
> 
> > > My goal is to strip out everything between the "address" and "pre" tags  
> > > and to output only:
> > 
> > OK, so you want every tag that is a sibling of address and pre and is  
> > within those two. I have found this StackOverFlow answer:
> > 
> > [html - XPath Expression: Select elements between A HREF="expr" tags - Stack Overflow](http://stackoverflow.com/a/6555376)
> > 
> > which applied to your problem:
> > 
> > 1.9.2p290 :001 \> require 'nokogiri'  
> > =\> true  
> > 1.9.2p290 :002 \> s = \<\<END  
> > 1.9.2p290 :003"\> \<!-- body="start" --\>  
> > 1.9.2p290 :004"\> \<div class="mail"\>  
> > [...snip...]  
> > 1.9.2p290 :031 \> doc = Nokogiri::HTML(s)  
> > 1.9.2p290 :039 \>  
> > doc.xpath("//address/following-sibling::node()[count(.|  
> > //pre/preceding-sibling::node())=count(//pre/preceding-sibling::node())]")  
> > =\> [#\<Nokogiri::XML::Text:0xdb0114 "\n"\>,  
> > #\<Nokogiri::XML::Element:0xdaff84 name="p"  
> > children=[#\<Nokogiri::XML::Text:0xdafc28 "\nLine 1\n"\>,  
> > #\<Nokogiri::XML::Element:0xdafa5c name="br"\>,  
> > #\<Nokogiri::XML::Text:0xdaf728 "\nLine 2\n"\>,  
> > #\<Nokogiri::XML::Element:0xdaf5ac name="br"\>,  
> > #\<Nokogiri::XML::Text:0xdaf228 "\nLine 3\n"\>,  
> > #\<Nokogiri::XML::Element:0xdaf0ac name="br"\>]\>,  
> > #\<Nokogiri::XML::Element:0xdaea80 name="p"\>]
> > 
> > will return a node set that contains the required nodes.
> 
> --  
> remember.guy do |as, often| as.you\_can - without end  
> [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [28 March 2013 12:07 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/17 "2013-03-28T12:07:13Z")

</div>

Joel Pearson wrote in post #1103448:

> You still have @my\_text and @date\_string there, which leads me to  
> suspect that's only the last part of your script.  
> The example I gave is the entire script...

You're right. Here's the whole thing:

#!/usr/bin/env ruby

# some initializations

@interesting = false  
@my\_text = ""

# read the file between the two "body" tags

fname = ARGV[0]

IO.foreach(fname) do |line|  
&nbsp;&nbsp;if line.match(/body="start"/)  
&nbsp;&nbsp;&nbsp;&nbsp;@interesting = true  
&nbsp;&nbsp;end

# meanwhile let's grab the date string and process it

&nbsp;&nbsp;start\_column = 4  
&nbsp;&nbsp;end\_column = 6

&nbsp;&nbsp;target\_range = (start\_column-1)..(end\_column-1)

&nbsp;&nbsp;if line.match(/\<dfn\>Date\<\/dfn\>/)  
&nbsp;&nbsp;&nbsp;&nbsp;pieces = line.split(" ")  
&nbsp;&nbsp;&nbsp;&nbsp;@date\_string = pieces[target\_range].join("-")  
&nbsp;&nbsp;end

&nbsp;&nbsp;if line.match(/body="end"/)  
&nbsp;&nbsp;&nbsp;&nbsp;@interesting = false  
&nbsp;&nbsp;end

&nbsp;&nbsp;if @interesting  
&nbsp;&nbsp;&nbsp;&nbsp;@my\_text \<\< line  
&nbsp;&nbsp;end  
end

require "nokogiri"

doc = Nokogiri::HTML(@my\_text)  
output = @date\_string + $/  
doc.xpath('//address/following-sibling::p//text()').each { |line| output  
\<\< ( line.text.strip + $/ ) }

File.write("snippet.txt", output)

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![7stud2](https://avatars.discourse-cdn.com/v4/letter/7/9de053/32.png) [@7stud2](https://rubytalk.org/u/7stud2)\
**Post date:** [29 March 2013 15:17 UTC](https://rubytalk.org/t/nokogiri-help-parsing-html/67586/18 "2013-03-29T15:17:59Z")

</div>

Joel Pearson wrote in post #1103626:

> Since you're using Nokogiri anyway, you'd be better off using that for  
> the whole process rather than looping through the HTML "manually". This  
> is the sort of thing it's worth getting in the habit of: using the tools  
> available to their fullest potential.  
> You can do the whole thing in 5 lines (barring error-checking) as I  
> demonstrated earlier.

I completely missed that in your earlier post! It definitely makes  
sense to use Nokogiri to do exactly what it's good at.

Thanks!

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).