Hpricot html parsing

ruby-talk
1

hi all,
I have the following html fragment
I want to get the inner html content inside the
<p> <img></p> tag , not the between the <p> </p> tag.
for example in the following example i want to get the result as
"this is fun". I dont want to get the result including "NO FUN".
how to do with Hpricot

example html fragment:

···

----------------------

<p class=posted>
this is fun
<img src="" class="dhans"/>
</p>
<p class=posted>
NO FUN
</p>

thanks in advance,
dhanasekaran

--
Posted via http://www.ruby-forum.com/.

2

Dhanasekaran Vivekanandhan wrote:

hi all,
I have the following html fragment
I want to get the inner html content inside the
<p> <img></p> tag , not the between the <p> </p> tag.
for example in the following example i want to get the result as
"this is fun". I dont want to get the result including "NO FUN".
how to do with Hpricot

example html fragment:
----------------------

<p class=posted>
this is fun
<img src="" class="dhans"/>
</p>
<p class=posted>
NO FUN
</p>

I did not quite get you. You want the text of the first <p> because it
has an image?
Or what is the exact criterion to accept/reject <p>'s?

Peter

···

__
http://www.rubyrailways.com

3

yes, I want the text of the first <p> because it
has an image. and reject if <p> has no image.
thanks,
Dhanasekaran

···

--
Posted via http://www.ruby-forum.com/.

4

Dhanasekaran Vivekanandhan wrote:

yes, I want the text of the first <p> because it
has an image. and reject if <p> has no image.
thanks,

I see. Try this:

···

===============================================
require 'rubygems'
require 'hpricot'

doc = Hpricot %q{<p class=posted>
this is fun
<img src="" class="dhans"/>
</p>
<p class=posted>
NO FUN
</p>
<p class=posted>
  fun again!
  <img src=""/>
</p>
<p class=posted>
  NO FUN AT ALL!
</p>
}

paragraphs = doc/'p'

good_elems = paragraphs.map.reject {|elem| ((elem/"img").empty?) }
good_elems.each { |elem| puts elem.inner_text.strip }

output:

************
this is fun
fun again!
************

You will need hpricot 0.4.84 because of inner_text - if you don't want
to install it (I did not experience any difficulties, so I can recommend
it) then you have to roll your own inner_text, but I guess this is not a
big problem.

Cheers,
Peter

5

You can try something like this:

if p.search("img").length > 0
   puts p.inner_html
end

Dhanasekaran Vivekanandhan wrote:

···

yes, I want the text of the first <p> because it
has an image. and reject if <p> has no image.
thanks,
Dhanasekaran

--
Posted via http://www.ruby-forum.com/\.

6

Dhanasekaran Vivekanandhan wrote:

yes, I want the text of the first <p> because it
has an image. and reject if <p> has no image.

Hpricot might be able to do this, but you can also do it on your own, and
know why the solution works.

···

---------------------------------------

#!/usr/bin/ruby -w

data = File.read("test.html")

array = data.scan(%r{<p>([^<]+?)<img .*?/></p>})

p array

---------------------------------------

Input text:

<p>don't want this text</p>

<p>want this text<img src=""/></p>

<p>don't want this text either</p>

<p>want this text too<img src=""/></p>

Output:

[["want this text"], ["want this text too"]]

--
Paul Lutus
http://www.arachnoid.com

7

Peter Szinek wrote:

paragraphs = doc/'p'

good_elems = paragraphs.map.reject {|elem| ((elem/"img").empty?) }

Which once again makes me wish paragraphs = doc/'//p[img]/text()'
worked. This could be doable if you asked Hpricot to provide you with
the REXML document (it's probably out of scope for the intendedly simple
XPath engine Hpricot uses natively), but unfortunately I can't for the
heck of it figure out how to make REXML accept the final /text(), even
though the parser claims to support XPath 1.0 except a few exceptions,
that one not being noted.

David Vallner

8

Thanks Peter ,
Your solution worked. and I just wanted to know , where can I find the
syntax for Hpricot like the one you gave as a solution,

thanks,
dhanasekaran

···

--
Posted via http://www.ruby-forum.com/.

9

Ask:

http://code.whytheluckystiff.net/hpricot/ticket/32

text in xpath should return a text node if present. For example:
(doc/"/html/body/div[1]/*/table[0]/tr[0]/*/b[9]/text")

Currently I am using the search and next_node:

    doc.search("/html/body/div[1]/*/table[0]/tr[0]/td/b"){|x|
@movie_plot=x.next_node.to_s.strip if x.inner_html=="Plot Outline:" }

And receive

Author:
    why
Message:

        * lib/hpricot/elements.rb: added support for selecting text
nodes with text(): //p/text(), //p[a]//text(), etc.
        * lib/hpricot/traverse.rb: ditto.
        * lib/hpricot/tag.rb: the pathname method reports the path
fragment needed to get to this node.
        * lib/hpricot/parse.rb: handle possible empty processing instruction.
http://code.whytheluckystiff.net/hpricot/changeset/87

···

On 12/13/06, David Vallner <david@vallner.net> wrote:

Peter Szinek wrote:
> paragraphs = doc/'p'
>
> good_elems = paragraphs.map.reject {|elem| ((elem/"img").empty?) }

Which once again makes me wish paragraphs = doc/'//p[img]/text()'
worked. This could be doable if you asked Hpricot to provide you with
the REXML document (it's probably out of scope for the intendedly simple
XPath engine Hpricot uses natively), but unfortunately I can't for the
heck of it figure out how to make REXML accept the final /text(), even
though the parser claims to support XPath 1.0 except a few exceptions,
that one not being noted.

David Vallner

10

Dhanasekaran Vivekanandhan wrote:

Thanks Peter ,
Your solution worked. and I just wanted to know , where can I find the syntax for Hpricot like the one you gave as a solution,

Hmm, except of what can be found on the Hpricot page, I am using

1) rdoc, ri
2) p SomeHpricotClass.methods.sort
3) my kind-of-decent XPath knowledge
4) source code browsing (you don't have to be a pro (I am a newbie myself) and you can get a surprisingly lot from there))
5) common sense
6) ruby mailing list

Roughly in this order... A cheatsheet or something would be handy.. maybe there is already one somewhere?

Cheers,
Peter

···

__
http://www.rubyrailways.com

11

ruby talk wrote:

Ask:

http://code.whytheluckystiff.net/hpricot/ticket/32

text in xpath should return a text node if present. For example:
(doc/"/html/body/div[1]/*/table[0]/tr[0]/*/b[9]/text")

Well, it's 'text()' not 'text'. Luckily _why noticed.

       * lib/hpricot/elements.rb: added support for selecting text
nodes with text(): //p/text(), //p[a]//text(), etc.

W00t :stuck_out_tongue_winking_eye:

Thanks for pointing this out.

David Vallner

PS: Your email address name confuses the heck out of me. Please use
something that doesn't cause a mental namespace clash?

12

Citát Peter Szinek <peter@rubyrailways.com>:

Roughly in this order... A cheatsheet or something would be handy..
maybe there is already one somewhere?

http://code.whytheluckystiff.net/hpricot/wiki/SupportedXpathExpressions ?

Also, I'd take that in preference to point 3, using an XPath -ish sort of query
and then using a syntax element the implementation happens to not understand is
rather infuriating. (Aght REXML not supporting text() in a POLS way, if at
all.)

David Vallner

13

Sorry, I have been archiving ruby talk at rubytalk@gmail.com since 10/14/04.

Stephen Becker IV

···

On 12/16/06, David Vallner <david@vallner.net> wrote:

ruby talk wrote:
> Ask:
>
> http://code.whytheluckystiff.net/hpricot/ticket/32
>
> text in xpath should return a text node if present. For example:
> (doc/"/html/body/div[1]/*/table[0]/tr[0]/*/b[9]/text")
>

Well, it's 'text()' not 'text'. Luckily _why noticed.

> * lib/hpricot/elements.rb: added support for selecting text
> nodes with text(): //p/text(), //p[a]//text(), etc.

W00t :stuck_out_tongue_winking_eye:

Thanks for pointing this out.

David Vallner

PS: Your email address name confuses the heck out of me. Please use
something that doesn't cause a mental namespace clash?