# REXML libraries and parsing issues **URL:** \ **Category:** ruby-talk\ **Created:** [24 June 2005 00:54 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152 "2005-06-24T00:54:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![B\_Angell](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@B\_Angell](https://rubytalk.org/u/B_Angell)\ **Post date:** [24 June 2005 00:54 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/1 "2005-06-24T00:54:51Z")
First off, let me say right up front that I am a newbie wrt Ruby. I am trying to parse an XML file, however, am having all kinds of trouble. I am using the REXML libraries and the sax2parser/listener. In the sax2listener, I can use the character/text part of the method, however, I cannot for the life of me figure out how to parse out JUST WHAT I WANT. Here is what the file looks like as follows: \\\ this is the text I need \\\ If I use :character, %w{PDAT} {|text| puts text} ... I get the text "this is the text I need" printed out. If I use the B110 or any combination, I cannot get it to work. Anyone know how to get the sax2parser/listener to parse the file and allow me to be selective about what I parse out of the file? Thanks for any/all help in this endeavor!!!!!!!!!! -Bob Angell- [angellr@mac.com](mailto:angellr@mac.com) ---
**Author:** ![James\_Britt4](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@James\_Britt4](https://rubytalk.org/u/James_Britt4)\ **Post date:** [24 June 2005 01:50 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/2 "2005-06-24T01:50:19Z")
BA wrote: > First off, let me say right up front that I am a newbie wrt Ruby. > > I am trying to parse an XML file, however, am having all kinds of trouble. I am using the REXML libraries and the sax2parser/listener. In the sax2listener, I can use the character/text part of the method, however, I cannot for the life of me figure out how to parse out JUST WHAT I WANT. Here is what the file looks like as follows: > > \\\ this is the text I need \\\ > > If I use :character, %w{PDAT} {|text| puts text} ... I get the text "this is the text I need" printed out. If I use the B110 or any combination, I cannot get it to work. Anyone know how to get the sax2parser/listener to parse the file and allow me to be selective about what I parse out of the file? Thanks for any/all help in this endeavor!!!!!!!!!! What, exactly, do you want? To extract the text from the PDAT element? How predictable is the XML? Are the files as small as your example? Are regular expressions an option? Or using a DOM and XPath? How did you decide to use the listner? James > **···** > > -- > > [http://www.ruby-doc.org](http://www.ruby-doc.org) - The Ruby Documentation Site > [http://www.rubyxml.com](http://www.rubyxml.com) - News, Articles, and Listings for Ruby & XML > [http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff > [http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys ---
**Author:** ![B\_Angell](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@B\_Angell](https://rubytalk.org/u/B_Angell)\ **Post date:** [24 June 2005 01:59 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/3 "2005-06-24T01:59:44Z")
Yes, I want to extract the PDAT element, however, I want to use the B110 tag to find this element. The XML \*is\* predictable, however, there are variations in the placement of the elements (there could be several different address fields and/or many paragraphs that need to be parsed/searched). The files are \*extremely\* large (some could be as large as 1-2GB). I would prefer to do all of the processing in Ruby if this is possible (want to use the OO functionality for the text processing I want to do) and would like to also incorporate regex if possible (started doing this by parsing the file line by line, however, ran into malformed XML where I decided that I needed to use the database functionality of XML. Not sure if DOM would work. Could not get XPath to work. The listener was, quite frankly, a SWAG. Thanks. > **···** > > On Jun 23, 2005, at 7:50 PM, James Britt wrote: > > > BA wrote: > > > > > First off, let me say right up front that I am a newbie wrt Ruby. > > > I am trying to parse an XML file, however, am having all kinds of trouble. I am using the REXML libraries and the sax2parser/listener. In the sax2listener, I can use the character/text part of the method, however, I cannot for the life of me figure out how to parse out JUST WHAT I WANT. Here is what the file looks like as follows: > > > \\\ this is the text I need \\\ > > > If I use :character, %w{PDAT} {|text| puts text} ... I get the text "this is the text I need" printed out. If I use the B110 or any combination, I cannot get it to work. Anyone know how to get the sax2parser/listener to parse the file and allow me to be selective about what I parse out of the file? Thanks for any/all help in this endeavor!!!!!!!!!! > > > > What, exactly, do you want? To extract the text from the PDAT element? > > > > How predictable is the XML? > > > > Are the files as small as your example? > > > > Are regular expressions an option? Or using a DOM and XPath? > > > > How did you decide to use the listner? > > > > James > > > > -- > > > > [http://www.ruby-doc.org](http://www.ruby-doc.org) - The Ruby Documentation Site > > [http://www.rubyxml.com](http://www.rubyxml.com) - News, Articles, and Listings for Ruby & XML > > [http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff > > [http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys ---
**Author:** ![Bucco](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@Bucco](https://rubytalk.org/u/Bucco)\ **Post date:** [24 June 2005 03:45 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/4 "2005-06-24T03:45:38Z")
How about something like: require 'rexml/document' doc = REXML::Document.new(File.open('someXMLFile.xml')) info = doc.elements["//B110/DNUM/PDAT"].text puts info SA 🙂 ---
**Author:** ![James\_Britt4](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@James\_Britt4](https://rubytalk.org/u/James_Britt4)\ **Post date:** [24 June 2005 05:02 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/5 "2005-06-24T05:02:25Z")
BA wrote: > Yes, I want to extract the PDAT element, however, I want to use the B110 tag to find this element. The XML \*is\* predictable, however, there are variations in the placement of the elements (there could be several different address fields and/or many paragraphs that need to be parsed/searched). The files are \*extremely\* large (some could be as large as 1-2GB). I would prefer to do all of the processing in Ruby if this is possible (want to use the OO functionality for the text processing I want to do) and would like to also incorporate regex if possible (started doing this by parsing the file line by line, however, ran into malformed XML where I decided that I needed to use the database functionality of XML. Not sure if DOM would work. Could not get XPath to work. The listener was, quite frankly, a SWAG. Thanks. OK, I got the picture. I would suggest the pull parser. Open up a file stream and keep pulling events. When you get a start\_element event, check the element name. If it is B110, then, loop and pull events until the PDAT element. Then pull until text event. Grab text and store it or whatever. Go back to main loop, looking again for that B110 element. Something like this: #!/usr/bin/env ruby require 'rexml/parsers/pullparser' include REXML::Parsers $text = def pdat( parser )    while parser.has\_next?      pull\_event = parser.pull      $text.push( pull\_event[0] ) if pull\_event.text?   end end def get\_text parser    while parser.has\_next?      pull\_event = parser.pull      b110( parser ) if pull\_event.start\_element? and                        pull\_event[0] =~ /B110/    end end def b110( parser )    while parser.has\_next?      pull\_event = parser.pull      pdat( parser ) if pull\_event.start\_element? and                        pull\_event[0] =~ /PDAT/    end end File.open( "pdat.xml", "r") { |f|    parser = PullParser.new( f )    b110( parser ) } puts $text.join( "\n" ) James > **···** > > -- > > [http://www.ruby-doc.org](http://www.ruby-doc.org) - The Ruby Documentation Site > [http://www.rubyxml.com](http://www.rubyxml.com) - News, Articles, and Listings for Ruby & XML > [http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff > [http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys ---
**Author:** ![mathew](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@mathew](https://rubytalk.org/u/mathew)\ **Post date:** [27 June 2005 17:50 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/6 "2005-06-27T17:50:41Z")
BA wrote: > Yes, I want to extract the PDAT element, however, I want to use the B110 tag to find this element. The XML \*is\* predictable, however, there are variations in the placement of the elements (there could be several different address fields and/or many paragraphs that need to be parsed/searched). The files are \*extremely\* large (some could be as large as 1-2GB). Any time you're faced with a huge XML file and you only want to get small pieces of it, you should think about using a stream parser, a la Java's SAX2. The idea behind a stream parser is that the parser runs through the entire file exactly once, and never has to seek forwards or backwards; it throws the data to you in whatever size chunks are most convenient for it. This means the parser does as little work as possible, which means so long as your code is efficient, the end result should be maximally efficient. The downside is that you have to do a bit more work. For example, depending on how the parser buffers things internally, it might send you a piece of text inside an XML element in two or more pieces, and expect you to glue them together. It's also up to you to deal with any position-based restrictions on which elements you're interested in. I assumed you wanted the text inside any PDAT element that was \*somewhere\* inside a B110 element, so I simply track which elements are currently "open" and by how many levels. Here's the code. > **···** > > --- > require 'rexml/document' > require 'rexml/parsers/streamparser' > > class MyListener > >    def initialize >      # Hash to record which elements we are inside at any given moment, and >      # how many of them we are inside >      @inside = Hash.new >      @textbuffer = '' >    end > >    def tag\_start(name, attrs) >      if @inside[name] >        @inside[name] += 1 >      else >        @inside[name] = 1 >      end >    end > >    def text(text) >      if @inside['B110'] and @inside['PDAT'] >        @textbuffer += text >      end >    end > >    def tag\_end(name) >      if name == 'PDAT' >        # Output the text if we just closed a PDAT inside a B110 >        if @inside['B110'] and @inside['PDAT'] >          puts @textbuffer >        end >        # Clear the buffer any time we close a PDAT >        @textbuffer = '' >      end >      # Decrement count, set to nil if zero >      # so @inside['foo'] works as a boolean >      if @inside[name] == 1 >        @inside[name] = nil >      else >        @inside[name] -= 1 >      end >    end > end > > listener = MyListener.new > source = File.new "mydoc.xml" > REXML::Document.parse\_stream(source, listener) > --- > > Here's a sample file: > > --- > > \ >    \\ >      \ >        \This is the text you want\This isn't. >      \ >    \ >      \This is sneaky good text\ >    \ >    \This is bad text\ > \ > > --- > > Output: > > --- > > This is the text you want > This is sneaky good text > > --- > > Note that this uses the native REXML stream parser API, not the SAX2 clone, because the SAX2 clone is slower according to the documentation. > > Disclaimers: > > The above code is only lightly tested. Although a stream parser should theoretically be the fastest option, I haven't actually benchmarked it against (say) the pull parser. (Which is also documented as having an unstable API, so personally I'd avoid it anyway.) > > The above code will break if you have a PDAT somewhere inside a PDAT. I'm assuming that's not allowed. If it is, you'll have to make your text buffer be a stack of strings rather than a simple string, append to @textbuffer[@inside['PDAT']], and take the performance hit. > > Also, if you need to do more elaborate selection of which elements to process, you'll obviously need to make changes to how the current position is tracked... e.g. implementing "process PDAT elements only if they are not buried more than 2 other elements deep inside a B110 element" is left as an exercise for the reader 🙂 > > \> (started doing this by parsing the file line by line, however, > > > ran into malformed XML where I decided that I needed to use the database functionality of XML. > > If by "malformed XML" you mean syntactically invalid XML, such as unescaped \< \> characters, then you may be hosed, as REXML's parsers will likely choke on it. > > mathew > -- > \ **Author:** ![James\_Edward\_Gray\_II](https://avatars.discourse-cdn.com/v4/letter/j/ea5d25/32.png) [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)\ **Post date:** [24 June 2005 04:02 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/7 "2005-06-24T04:02:24Z") For 2 Gig files?! Good luck! James Edward Gray II > **···** > > On Jun 23, 2005, at 10:45 PM, Bucco wrote: > > > How about something like: > > > > require 'rexml/document' > > doc = REXML::Document.new(File.open('someXMLFile.xml')) > > info = doc.elements["//B110/DNUM/PDAT"].text > > puts info ---
**Author:** ![Ryan\_Leavengood2](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@Ryan\_Leavengood2](https://rubytalk.org/u/Ryan_Leavengood2)\ **Post date:** [24 June 2005 04:09 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/8 "2005-06-24T04:09:02Z")
James Edward Gray II said: > **···** > > > On Jun 23, 2005, at 10:45 PM, Bucco wrote: > > > > > How about something like: > > > > > > require 'rexml/document' > > > doc = REXML::Document.new(File.open('someXMLFile.xml')) > > > info = doc.elements["//B110/DNUM/PDAT"].text > > > puts info > > > > For 2 Gig files?! Good luck! > > Hahahaha, I must agree with this. Of course the irony is that if those 2 > gig XML files were in YAML, they would only be 5 megs 😉 > > Ryan ---
**Author:** ![James\_Britt4](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@James\_Britt4](https://rubytalk.org/u/James_Britt4)\ **Post date:** [24 June 2005 05:15 UTC](https://rubytalk.org/t/rexml-libraries-and-parsing-issues/19152/9 "2005-06-24T05:15:50Z")
Ryan Leavengood wrote: > ... > Hahahaha, I must agree with this. Of course the irony is that if those 2 > gig XML files were in YAML, they would only be 5 megs 😉 How \*does\* one do stream parsing and partial reading of YAML files? Is there a SAY parser? James > **···** > > -- > > [http://www.ruby-doc.org](http://www.ruby-doc.org) - The Ruby Documentation Site > [http://www.rubyxml.com](http://www.rubyxml.com) - News, Articles, and Listings for Ruby & XML > [http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff > [http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys