\
**Category:** ruby-talk\
**Created:** [24 October 2004 17:39 UTC](https://rubytalk.org/t/parsing-html/14368 "2004-10-24T17:39:08Z")\
**Posts on this page:** 5\
**Page:** 1
**Author:**  [@Martin\_Pfeffer](https://rubytalk.org/u/Martin_Pfeffer)\
**Post date:** [24 October 2004 17:39 UTC](https://rubytalk.org/t/parsing-html/14368/1 "2004-10-24T17:39:08Z")
hi
my problem is i need a file with german words and so i try to create a file parsing html sites and write extracted words to a database so my questizn is what is the easyest way to extract text from html pages?
thx
Martin
---
**Author:**  [@Stefan\_Schmiedl](https://rubytalk.org/u/Stefan_Schmiedl)\
**Post date:** [24 October 2004 21:49 UTC](https://rubytalk.org/t/parsing-html/14368/2 "2004-10-24T21:49:08Z")
> hi
> my problem is i need a file with german words and so i try to create a
> file parsing html sites and write extracted words to a database so my
> questizn is what is the easyest way to extract text from html pages?
> thx
> Martin
there's a /usr/share/dict/ngerman on my Debian box
> wc ngerman
308860 308860 3998536 ngerman
which tells me that the average word length is about 13 (!) letters.
Unvorstellbar!
s.
> **···**
>
> On Sun, 24 Oct 2004 17:35:46 GMT, Martin Pfeffer \ wrote:
---
**Author:**  [@Brian\_SchrA\_der](https://rubytalk.org/u/Brian_SchrA_der)\
**Post date:** [25 October 2004 09:20 UTC](https://rubytalk.org/t/parsing-html/14368/3 "2004-10-25T09:20:41Z")
If you don't mind senseless words like "img" that come from html markup:
--8\<---
require 'open-uri' open('[http://ruby.brian-schroeder.de](http://ruby.brian-schroeder.de)').read.scan(/[-\wöäüß]+/i)
--8\<---
If you have valid xhtml:
--8\<---
require 'rexml/document'
require 'open-uri'
include REXML
Document.new(open('[http://ruby.brian-schroeder.de](http://ruby.brian-schroeder.de)')).
elements.to\_a('//').
map{|e| e.texts.map{|t|t.value} }.
join(' ').
scan(/[-\wöäüß]+/i).
sort.
uniq
--8\<---
hth,
Brian
PS: I'm shure the text-extraction with rexml can be done in a nicer/more efficent way.
> **···**
>
> On Mon, 25 Oct 2004 02:39:08 +0900 Martin Pfeffer \ wrote:
>
> > hi
> > my problem is i need a file with german words and so i try to create a
> > file parsing html sites and write extracted words to a database so my
> > questizn is what is the easyest way to extract text from html pages?
> > thx
> > Martin
---
**Author:**  [@Ben\_Giddings1](https://rubytalk.org/u/Ben_Giddings1)\
**Post date:** [26 October 2004 21:13 UTC](https://rubytalk.org/t/parsing-html/14368/4 "2004-10-26T21:13:00Z")
Martin Pfeffer wrote:
> my problem is i need a file with german words and so i try to create a file parsing html sites and write extracted words to a database so my questizn is what is the easyest way to extract text from html pages?
My "htmltokenizer" module (available on RAA and Rubyforge) is pretty good at extracting text from HTML pages.
Ben
---
**Author:**  [@Alexander\_Kellett2](https://rubytalk.org/u/Alexander_Kellett2)\
**Post date:** [27 October 2004 07:16 UTC](https://rubytalk.org/t/parsing-html/14368/5 "2004-10-27T07:16:33Z")
aye. it rocks. thanks for that 🙂
Alex
> **···**
>
> On Wed, Oct 27, 2004 at 06:13:00AM +0900, Ben Giddings wrote:
>
> > Martin Pfeffer wrote:
> > \>my problem is i need a file with german words and so i try to create a
> > \>file parsing html sites and write extracted words to a database so my
> > \>questizn is what is the easyest way to extract text from html pages?
> >
> > My "htmltokenizer" module (available on RAA and Rubyforge) is pretty
> > good at extracting text from HTML pages.