# Memory leak

**URL:** <https://rubytalk.org/t/memory-leak/55866>\
**Category:** ruby-talk\
**Created:** [20 October 2009 01:10 UTC](https://rubytalk.org/t/memory-leak/55866 "2009-10-20T01:10:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [20 October 2009 01:10 UTC](https://rubytalk.org/t/memory-leak/55866/1 "2009-10-20T01:10:09Z")

</div>

Hi all,  
I'm writing web crawler with threads support. And after working with  
some amount of link memory usage increase more and more. When program  
just started in use 20mb of mem. After crawling of 150-200 link, memory  
usage ~100. When 1000 link crawled my program may use up to 1GB of mem.  
Help me please find out why?

require 'rubygems'  
require 'mechanize'  
require 'hpricot'  
require 'yaml'  
require 'net/http'  
require 'uri'  
require 'modules/common'

Thread.abort\_on\_exception = true  
$config = YAML.load\_file "config.yml"  
links = IO.read("bases/3+4.txt").split("\n")  
threads = []

links.each do |link|  
&nbsp;&nbsp;if Thread.list.size \< 50 then  
&nbsp;&nbsp;&nbsp;&nbsp;threads \<\< Thread.new(link) { |myLink|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it(myLink)  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;sleep(1)  
&nbsp;&nbsp;&nbsp;&nbsp;threads.each { |t|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;&nbsp;puts "total threads: " + Thread.list.size.to\_s  
&nbsp;&nbsp;&nbsp;&nbsp;redo  
&nbsp;&nbsp;end  
end

threads.each { |t| t.join() }

What in "Common" module:  
1. Crawler (net/http or mechanize - I tried both, results the same)  
2. HTML parser (Hpricot or Nokogir - I tried both again, with same bad  
results)  
so I extract some data from page and save it to the file. Nothing  
special as you see.

When I start this program without threads I getting the same results ☹

Please help, is this my fault or something wrong in the libraries ?

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Aaron\_Patterson1](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@Aaron\_Patterson1](https://rubytalk.org/u/Aaron_Patterson1)\
**Post date:** [20 October 2009 02:13 UTC](https://rubytalk.org/t/memory-leak/55866/2 "2009-10-20T02:13:13Z")

</div>

Rob Doug wrote:

> Hi all,  
> I'm writing web crawler with threads support. And after working with  
> some amount of link memory usage increase more and more. When program  
> just started in use 20mb of mem. After crawling of 150-200 link, memory  
> usage ~100. When 1000 link crawled my program may use up to 1GB of mem.  
> Help me please find out why?
> 
> require 'rubygems'  
> require 'mechanize'  
> require 'hpricot'  
> require 'yaml'  
> require 'net/http'  
> require 'uri'  
> require 'modules/common'
> 
> Thread.abort\_on\_exception = true  
> $config = YAML.load\_file "config.yml"  
> links = IO.read("bases/3+4.txt").split("\n")  
> threads =
> 
> links.each do |link|  
> &nbsp;&nbsp;if Thread.list.size \< 50 then  
> &nbsp;&nbsp;&nbsp;&nbsp;threads \<\< Thread.new(link) { |myLink|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it(myLink)  
> &nbsp;&nbsp;&nbsp;&nbsp;}  
> &nbsp;&nbsp;else  
> &nbsp;&nbsp;&nbsp;&nbsp;sleep(1)  
> &nbsp;&nbsp;&nbsp;&nbsp;threads.each { |t|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;&nbsp;}  
> &nbsp;&nbsp;&nbsp;&nbsp;puts "total threads: " + Thread.list.size.to\_s  
> &nbsp;&nbsp;&nbsp;&nbsp;redo  
> &nbsp;&nbsp;end  
> end
> 
> threads.each { |t| t.join() }
> 
> What in "Common" module:  
> 1. Crawler (net/http or mechanize - I tried both, results the same)  
> 2. HTML parser (Hpricot or Nokogir - I tried both again, with same bad  
> results)  
> so I extract some data from page and save it to the file. Nothing  
> special as you see.
> 
> When I start this program without threads I getting the same results ☹
> 
> Please help, is this my fault or something wrong in the libraries ?

One culprit could be mechanize. Mechanize keeps a history of the pages  
you've visited, and if you don't limit that length it will grow infinitely.

Try setting the max history:

&nbsp;&nbsp;[http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173](http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173)

> **···**
>
> --  
> Aaron Patterson  
> [http://tenderlovemaking.com/](http://tenderlovemaking.com/)

---

<div class="post-metadata">

**Author:** ![Ryan\_Davis1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/ryan_davis1/32/1848_2.png) [@Ryan\_Davis1](https://rubytalk.org/u/Ryan_Davis1)\
**Post date:** [20 October 2009 08:13 UTC](https://rubytalk.org/t/memory-leak/55866/3 "2009-10-20T08:13:26Z")

</div>

> links = IO.read("bases/3+4.txt").split("\n")  
> threads =
> 
> links.each do |link|  
> if Thread.list.size \< 50 then  
> &nbsp;&nbsp;&nbsp;threads \<\< Thread.new(link) { |myLink|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it(myLink)  
> &nbsp;&nbsp;&nbsp;}  
> else  
> &nbsp;&nbsp;&nbsp;sleep(1)  
> &nbsp;&nbsp;&nbsp;threads.each { |t|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;}  
> &nbsp;&nbsp;&nbsp;puts "total threads: " + Thread.list.size.to\_s  
> &nbsp;&nbsp;&nbsp;redo  
> end  
> end
> 
> threads.each { |t| t.join() }

I can't speak for your supposed "leaks" (pure ruby almost never has leaks, but it does have many sneaky ways to make valid object references that you can overlook), since I can't figure out what your code does or how it is supposed to work. threads is an array of thread objects that you only ever push on and you never clear or shift off of. Also, why oh why oh why are you doing a redo??? Overly complicated code may not be the source of your leak (and in this case it may very well be), but it certainly isn't doing you any favors.

I almost always structure my threads as a pool of workers pulling from a queue of tasks:

> **···**
>
> On Oct 19, 2009, at 18:10 , Rob Doug wrote:
> 
> > links = Queue.new  
> > IO.read("bases/3+4.txt").split("\n").each do |l|  
> > &nbsp;&nbsp;links \<\< l  
> > end  
> > threads =
> > 
> > 50.times do  
> > &nbsp;&nbsp;threads \<\< Thread.new do  
> > &nbsp;&nbsp;&nbsp;&nbsp;until links.empty? do  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it links.shift  
> > &nbsp;&nbsp;&nbsp;&nbsp;end  
> > &nbsp;&nbsp;end  
> > end
> > 
> > threads.each { |t| t.join() }

---

<div class="post-metadata">

**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\
**Post date:** [20 October 2009 21:05 UTC](https://rubytalk.org/t/memory-leak/55866/4 "2009-10-20T21:05:05Z")

</div>

As far as I can see you're never cleaning Array threads. You are just appending to the Array and so even terminated threads will stay in there because you also do not reuse threads.

Design criticism: usually you create threads upfront and pass work to them via a Queue then you don't need the sleep crutch and use blocking instead. Also, you can make your code better handle large volumes of URLs by using a more streamed approach along the lines Ryan presented by limiting all resources (not only threads but also size of the queue).

require 'thread'  
...

THREADS = 50  
q = SizedQueue.new(THREADS \* 2)

threads = (1..THREADS).map do  
&nbsp;&nbsp;&nbsp;Thread.new q do qq  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;until qq.equal?(url = qq.deq)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common....  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;end  
end

File.foreach("bases/3+4.txt") do |line|  
&nbsp;&nbsp;&nbsp;line.chomp!  
&nbsp;&nbsp;&nbsp;q.enq(URI.parse(line))  
end

threads.size.times {|q| q.enq q}

Kind regards

&nbsp;&nbsp;robert

> **···**
>
> On 20.10.2009 03:10, Rob Doug wrote:
> 
> > Hi all,  
> > I'm writing web crawler with threads support. And after working with  
> > some amount of link memory usage increase more and more. When program  
> > just started in use 20mb of mem. After crawling of 150-200 link, memory  
> > usage ~100. When 1000 link crawled my program may use up to 1GB of mem.  
> > Help me please find out why?
> > 
> > require 'rubygems'  
> > require 'mechanize'  
> > require 'hpricot'  
> > require 'yaml'  
> > require 'net/http'  
> > require 'uri'  
> > require 'modules/common'
> > 
> > Thread.abort\_on\_exception = true  
> > $config = YAML.load\_file "config.yml"  
> > links = IO.read("bases/3+4.txt").split("\n")  
> > threads =
> > 
> > links.each do |link|  
> > &nbsp;&nbsp;if Thread.list.size \< 50 then  
> > &nbsp;&nbsp;&nbsp;&nbsp;threads \<\< Thread.new(link) { |myLink|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it(myLink)  
> > &nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;else  
> > &nbsp;&nbsp;&nbsp;&nbsp;sleep(1)  
> > &nbsp;&nbsp;&nbsp;&nbsp;threads.each { |t|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> > &nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;&nbsp;puts "total threads: " + Thread.list.size.to\_s  
> > &nbsp;&nbsp;&nbsp;&nbsp;redo  
> > &nbsp;&nbsp;end  
> > end
> > 
> > threads.each { |t| t.join() }
> > 
> > What in "Common" module:  
> > 1. Crawler (net/http or mechanize - I tried both, results the same)  
> > 2. HTML parser (Hpricot or Nokogir - I tried both again, with same bad  
> > results)  
> > so I extract some data from page and save it to the file. Nothing  
> > special as you see.
> > 
> > When I start this program without threads I getting the same results ☹
> > 
> > Please help, is this my fault or something wrong in the libraries ?
> 
> --  
> remember.guy do |as, often| as.you\_can - without end  
> [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)

---

<div class="post-metadata">

**Author:** ![Eliza\_Sahoo](https://avatars.discourse-cdn.com/v4/letter/e/e480ec/32.png) [@Eliza\_Sahoo](https://rubytalk.org/u/Eliza_Sahoo)\
**Post date:** [15 April 2010 08:28 UTC](https://rubytalk.org/t/memory-leak/55866/5 "2010-04-15T08:28:52Z")

</div>

In a .net windows application the form might have resource leaks though  
it is running with managed codes. We can use the below procedure to  
check if a form is having resource leak.  
Open Windows Task Manager  
Click on Process tab.  
Select "View" in the menu and then select "Select Columns" menu item.  
Check the USER Objects and GDI Objects (check boxes) to make them appear  
on the process page list header.  
The code in your project that lunches the Win Form, please ensure you  
have called the Dispose method. Forms implement the IDisposable  
interface so their dispose method must be called the moment they are no  
longer needed (Test 1). We can call Dispose explicitly or even better to  
instantiate it implicitly by the help of using clause (Test 2).  
We can use the GC.Collect() after the using statement or after the call  
to dispose for troubleshooting purpose.  
Now time to launch the form. Please note the note the USER Objects and  
GDI Objects values at the task manager. Close the form after some time  
and when the form is closed, note the values again at the task manager.  
We can find the value is decreased if it is increased then there is a  
leak in the form.  
Fix the resource leak and remove the call to GC.Collect(). It is  
generally unnecessary to make an explicit call to GC.Collect().

> **[Checking resource leaks in a form - BurnIgnorance](http://burnignorance.com/c-net-programming-tips/checking-resource-leaks-in-a-form/)**
>
> In a .net windows application the form might have resource leaks though it is running with managed codes. We can use the below procedure to check if a form is having resource leak. Open  Windows Task Manager Click on Process tab. Select “View” in the...

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [20 October 2009 02:17 UTC](https://rubytalk.org/t/memory-leak/55866/6 "2009-10-20T02:17:40Z")

</div>

> One culprit could be mechanize. Mechanize keeps a history of the pages  
> you've visited, and if you don't limit that length it will grow  
> infinitely.
> 
> Try setting the max history:
> 
> &nbsp;&nbsp;[http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173](http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173)

Thanks, but I've already try this... mem leak happens even when I use  
"net/http" with Hpricot

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [20 October 2009 14:29 UTC](https://rubytalk.org/t/memory-leak/55866/7 "2009-10-20T14:29:32Z")

</div>

Ryan Davis wrote:

> > &nbsp;&nbsp;&nbsp;threads.each { |t|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> > &nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;puts "total threads: " + Thread.list.size.to\_s  
> > &nbsp;&nbsp;&nbsp;redo  
> > end  
> > end
> > 
> > threads.each { |t| t.join() }
> 
> I can't speak for your supposed "leaks" (pure ruby almost never has  
> leaks, but it does have many sneaky ways to make valid object  
> references that you can overlook), since I can't figure out what your  
> code does or how it is supposed to work. threads is an array of thread  
> objects that you only ever push on and you never clear or shift off  
> of. Also, why oh why oh why are you doing a redo??? Overly complicated  
> code may not be the source of your leak (and in this case it may very  
> well be), but it certainly isn't doing you any favors.
> 
> I almost always structure my threads as a pool of workers pulling from  
> a queue of tasks:

actually I had this kind of code at first:

links.each do |link|  
&nbsp;&nbsp;if Thread.list.size \< 50 then  
&nbsp;&nbsp;&nbsp;&nbsp;threads \<\< Thread.new(link) { |myLink|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common.post\_it(myLink,doorway)  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;sleep(1)  
&nbsp;&nbsp;&nbsp;&nbsp;threads.each { |t|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unless t.status then  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.join  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;&nbsp;threads = threads.delete\_if {|t| t.status == false}  
&nbsp;&nbsp;&nbsp;&nbsp;GC.start  
&nbsp;&nbsp;&nbsp;&nbsp;redo  
&nbsp;&nbsp;end  
end

but it does not help, and I reduce this code

> **···**
>
> > On Oct 19, 2009, at 18:10 , Rob Doug wrote:
> 
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [20 October 2009 22:47 UTC](https://rubytalk.org/t/memory-leak/55866/8 "2009-10-20T22:47:07Z")

</div>

> As far as I can see you're never cleaning Array threads. You are just  
> appending to the Array and so even terminated threads will stay in there  
> because you also do not reuse threads.
> 
> Design criticism: usually you create threads upfront and pass work to  
> them via a Queue then you don't need the sleep crutch and use blocking  
> instead. Also, you can make your code better handle large volumes of  
> URLs by using a more streamed approach along the lines Ryan presented by  
> limiting all resources (not only threads but also size of the queue).
> 
> require 'thread'  
> ...
> 
> THREADS = 50  
> q = SizedQueue.new(THREADS \* 2)
> 
> threads = (1..THREADS).map do  
> &nbsp;&nbsp;&nbsp;Thread.new q do qq  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;until qq.equal?(url = qq.deq)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common....  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;end  
> end
> 
> File.foreach("bases/3+4.txt") do |line|  
> &nbsp;&nbsp;&nbsp;line.chomp!  
> &nbsp;&nbsp;&nbsp;q.enq(URI.parse(line))  
> end
> 
> threads.size.times {|q| q.enq q}

Very thank you, your solution is great. Code become much clear and looks  
good. 🙂

But, this is not solve memory leak ☹

Any ideas or suggestions?

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Aaron\_Patterson1](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@Aaron\_Patterson1](https://rubytalk.org/u/Aaron_Patterson1)\
**Post date:** [20 October 2009 03:51 UTC](https://rubytalk.org/t/memory-leak/55866/9 "2009-10-20T03:51:20Z")

</div>

Rob Doug wrote:

> > One culprit could be mechanize. Mechanize keeps a history of the pages  
> > you've visited, and if you don't limit that length it will grow  
> > infinitely.
> > 
> > Try setting the max history:
> > 
> > &nbsp;&nbsp;[http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173](http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000173)
> 
> Thanks, but I've already try this... mem leak happens even when I use  
> "net/http" with Hpricot

You'll have to share more of your implementation then. You deduced that  
the cause wasn't threads, but you've only shared the threaded part of  
your code.

Also, have you run your program with valgrind to verify that it is in  
fact a memory leak? It could just be objects sitting around waiting for GC.

> **···**
>
> --  
> Aaron Patterson  
> [http://tenderlovemaking.com/](http://tenderlovemaking.com/)

---

<div class="post-metadata">

**Author:** ![Ryan\_Davis1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/ryan_davis1/32/1848_2.png) [@Ryan\_Davis1](https://rubytalk.org/u/Ryan_Davis1)\
**Post date:** [20 October 2009 21:31 UTC](https://rubytalk.org/t/memory-leak/55866/10 "2009-10-20T21:31:28Z")

</div>

yeah. I don't care. it is still really obfuscated. Switch to a simple pool of workers pulling tasks out of a queue. Don't do redo for no reason. Don't do crazy stuff.

> **···**
>
> On Oct 20, 2009, at 07:29 , Rob Doug wrote:
> 
> > > I almost always structure my threads as a pool of workers pulling from  
> > > a queue of tasks:
> > 
> > actually I had this kind of code at first:

---

<div class="post-metadata">

**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\
**Post date:** [21 October 2009 08:40 UTC](https://rubytalk.org/t/memory-leak/55866/11 "2009-10-21T08:40:14Z")

</div>

You could print out object statistics to get an idea about the source of the leak. Something along the lines

require 'pp'

def print\_class\_counts  
&nbsp;&nbsp;&nbsp;c = Hash.new 0  
&nbsp;&nbsp;&nbsp;ObjectSpace.each\_object(Object) do |o|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;c[o.class] += 1  
&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;pp c  
end

Cheers

&nbsp;&nbsp;robert

> **···**
>
> On 21.10.2009 00:47, Rob Doug wrote:
> 
> > > As far as I can see you're never cleaning Array threads. You are just  
> > > appending to the Array and so even terminated threads will stay in there  
> > > because you also do not reuse threads.
> > > 
> > > Design criticism: usually you create threads upfront and pass work to  
> > > them via a Queue then you don't need the sleep crutch and use blocking  
> > > instead. Also, you can make your code better handle large volumes of  
> > > URLs by using a more streamed approach along the lines Ryan presented by  
> > > limiting all resources (not only threads but also size of the queue).
> > > 
> > > require 'thread'  
> > > ...
> > > 
> > > THREADS = 50  
> > > q = SizedQueue.new(THREADS \* 2)
> > > 
> > > threads = (1..THREADS).map do  
> > > &nbsp;&nbsp;&nbsp;Thread.new q do qq  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;until qq.equal?(url = qq.deq)  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common....  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> > > &nbsp;&nbsp;&nbsp;end  
> > > end
> > > 
> > > File.foreach("bases/3+4.txt") do |line|  
> > > &nbsp;&nbsp;&nbsp;line.chomp!  
> > > &nbsp;&nbsp;&nbsp;q.enq(URI.parse(line))  
> > > end
> > > 
> > > threads.size.times {|q| q.enq q}
> > 
> > Very thank you, your solution is great. Code become much clear and looks good. 🙂
> > 
> > But, this is not solve memory leak ☹
> > 
> > Any ideas or suggestions?
> 
> --  
> remember.guy do |as, often| as.you\_can - without end  
> [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [20 October 2009 14:27 UTC](https://rubytalk.org/t/memory-leak/55866/12 "2009-10-20T14:27:48Z")

</div>

> You'll have to share more of your implementation then. You deduced that  
> the cause wasn't threads, but you've only shared the threaded part of  
> your code.
> 
> Also, have you run your program with valgrind to verify that it is in  
> fact a memory leak? It could just be objects sitting around waiting for  
> GC.

Here the rest part of program, I did not think here can be any leaks,  
this is why I posted code without this part.

module Common  
&nbsp;&nbsp;def Common.post\_it(link)  
&nbsp;&nbsp;&nbsp;&nbsp;begin  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts link  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent = Net::HTTP  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;html = agent.get(URI.parse(link))  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;doc = Hpricot(html)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;forms = doc.search "//form"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;forms.each do |form|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts form.search("input").length  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;rescue =\> err  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts err  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_write\_safe("logs/errorlog.txt", "#{err.to\_s}|#{link}\n")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false  
&nbsp;&nbsp;&nbsp;&nbsp;rescue Timeout::Error  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_write\_safe("logs/errorlog.txt", "Request Timeout|#{link}\n")  
&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;end

&nbsp;&nbsp;private

&nbsp;&nbsp;def self.file\_write\_safe(file,message)  
&nbsp;&nbsp;&nbsp;&nbsp;File.open(file, 'a') { |f|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;f.flock(File::LOCK\_EX)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;f.write(message)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;f.flock(File::LOCK\_UN)  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;end  
end

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [22 October 2009 02:52 UTC](https://rubytalk.org/t/memory-leak/55866/13 "2009-10-22T02:52:27Z")

</div>

> You could print out object statistics to get an idea about the source of  
> the leak. Something along the lines
> 
> require 'pp'
> 
> def print\_class\_counts  
> &nbsp;&nbsp;&nbsp;c = Hash.new 0  
> &nbsp;&nbsp;&nbsp;ObjectSpace.each\_object(Object) do |o|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;c[o.class] += 1  
> &nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;pp c  
> end
> 
> Cheers
> 
> &nbsp;&nbsp;robert

I retest my script again on 10k+ database and seems it stops to grow  
when reach 550mb size... I'm running 50 threads, so 10mb per thread. As  
I understand this is still not good ?

Robert, I did what you propose and here is some data. I print all  
objects that have more than 200 copies.

Here object after 100 URLs:  
"Class: 1039"  
"Thread: 395"  
"Gem::Version::Part: 343"  
"Regexp: 1463"  
"MatchData: 301"  
"Hpricot::Text: 31848"  
"Gem::Version: 424"  
"Array: 24977"  
"Nokogiri::XML::Element: 1636"  
"WWW::Mechanize::Chain: 203"  
"Gem::Requirement: 380"  
"WWW::Mechanize::Form::Field: 627"  
"String: 199615"  
"Hpricot::Elem: 21211"  
"Nokogiri::XML::SyntaxError: 1562"  
"Hash: 15884"  
"Proc: 949"  
"WWW::Mechanize::Form::Option: 458"

Here after 400 URLs:  
"Class: 890"  
"Gem::Version::Part: 343"  
"Regexp: 1089"  
"MatchData: 264"  
"Hpricot::BogusETag: 1573"  
"Hpricot::Text: 82593"  
"Gem::Version: 424"  
"Array: 53259"  
"Nokogiri::XML::Element: 2982"  
"WWW::Mechanize::Form::Button: 259"  
"WWW::Mechanize::Chain: 242"  
"Gem::Requirement: 380"  
"WWW::Mechanize::Form::Field: 1179"  
"String: 426729"  
"WWW::Mechanize::Form::RadioButton: 351"  
"Nokogiri::XML::SyntaxError: 6464"  
"Hash: 37025"  
"Hpricot::Elem: 55426"  
"Proc: 774"  
"WWW::Mechanize::Form: 338"  
"WWW::Mechanize::Form::Option: 601"

And here after 500 URLs:  
"Class: 991"  
"Thread: 354"  
"Gem::Version::Part: 343"  
"Regexp: 847"  
"MatchData: 409"  
"Hpricot::BogusETag: 915"  
"Hpricot::Text: 31707"  
"Gem::Version: 424"  
"Array: 24143"  
"Nokogiri::XML::Element: 2076"  
"WWW::Mechanize::Form::Button: 217"  
"WWW::Mechanize::Chain: 218"  
"Gem::Requirement: 380"  
"WWW::Mechanize::Form::Field: 874"  
"String: 197836"  
"Nokogiri::XML::SyntaxError: 2608"  
"Hash: 14719"  
"Hpricot::Elem: 21530"  
"Proc: 925"  
"WWW::Mechanize::Form: 266"  
"WWW::Mechanize::Form::Option: 416"

As you can see there is no stable increasing amount of some object.

What can I do more?

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![John\_W\_Higgins1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/john_w_higgins1/32/1922_2.png) [@John\_W\_Higgins1](https://rubytalk.org/u/John_W_Higgins1)\
**Post date:** [22 October 2009 03:33 UTC](https://rubytalk.org/t/memory-leak/55866/14 "2009-10-22T03:33:00Z")

</div>

Evening Rob,

May I ask why you need threads and that level of complication? Are you  
really that sensitive towards speed that it really matters if you simply  
forked processes that died after 100 downloads and just started another  
worker? It would seem to be easier to use a nice simple independant message  
queue and just fire up workers that grab messages and download away.

I know it's not the sexy option but what exactly are you gaining by using  
threads here? Much easier to get a nice single threaded worker tuned up and  
tight then what you appear to be going through here. In fact - I might argue  
that you would be better with a downloader and separater parser processes  
that worked independantly via messages.

Just a thought....

John

---

<div class="post-metadata">

**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\
**Post date:** [22 October 2009 08:40 UTC](https://rubytalk.org/t/memory-leak/55866/15 "2009-10-22T08:40:09Z")

</div>

Your number of threads is suspiciously high compared to the 50 threads that you intend to run. That may be an indication that your code still suffers the issue that you stuff Thread instances into the Array and never remove them.

Other than that I cannot see something obvious on first sight. Maybe it makes more sense to extend the memory dumping to dump only positive deltas. An alternative is to decrease the dumping interval (e.g. just after fetching just one URL).

If you want to throw more on this you could use set\_trace\_func to capture invocations of methods named "new" along with the stack trace so you get the allocation sites.

Btw, what version of Ruby is this? IIRC there was a bug with Array#shift up to 1.8.6 which could also cause these effects.

Kind regards

&nbsp;&nbsp;robert

> **···**
>
> On 10/22/2009 04:52 AM, Rob Doug wrote:
> 
> > > You could print out object statistics to get an idea about the source of  
> > > the leak. Something along the lines
> > > 
> > > require 'pp'
> > > 
> > > def print\_class\_counts  
> > > &nbsp;&nbsp;&nbsp;c = Hash.new 0  
> > > &nbsp;&nbsp;&nbsp;ObjectSpace.each\_object(Object) do |o|  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;c[o.class] += 1  
> > > &nbsp;&nbsp;&nbsp;end  
> > > &nbsp;&nbsp;&nbsp;pp c  
> > > end
> > > 
> > > Cheers
> > > 
> > > &nbsp;&nbsp;robert
> > 
> > I retest my script again on 10k+ database and seems it stops to grow when reach 550mb size... I'm running 50 threads, so 10mb per thread. As I understand this is still not good ?
> > 
> > Robert, I did what you propose and here is some data. I print all objects that have more than 200 copies.
> > 
> > Here object after 100 URLs:  
> > "Class: 1039"  
> > "Thread: 395"  
> > "Gem::Version::Part: 343"  
> > "Regexp: 1463"  
> > "MatchData: 301"  
> > "Hpricot::Text: 31848"  
> > "Gem::Version: 424"  
> > "Array: 24977"  
> > "Nokogiri::XML::Element: 1636"  
> > "WWW::Mechanize::Chain: 203"  
> > "Gem::Requirement: 380"  
> > "WWW::Mechanize::Form::Field: 627"  
> > "String: 199615"  
> > "Hpricot::Elem: 21211"  
> > "Nokogiri::XML::SyntaxError: 1562"  
> > "Hash: 15884"  
> > "Proc: 949"  
> > "WWW::Mechanize::Form::Option: 458"
> > 
> > Here after 400 URLs:  
> > "Class: 890"  
> > "Gem::Version::Part: 343"  
> > "Regexp: 1089"  
> > "MatchData: 264"  
> > "Hpricot::BogusETag: 1573"  
> > "Hpricot::Text: 82593"  
> > "Gem::Version: 424"  
> > "Array: 53259"  
> > "Nokogiri::XML::Element: 2982"  
> > "WWW::Mechanize::Form::Button: 259"  
> > "WWW::Mechanize::Chain: 242"  
> > "Gem::Requirement: 380"  
> > "WWW::Mechanize::Form::Field: 1179"  
> > "String: 426729"  
> > "WWW::Mechanize::Form::RadioButton: 351"  
> > "Nokogiri::XML::SyntaxError: 6464"  
> > "Hash: 37025"  
> > "Hpricot::Elem: 55426"  
> > "Proc: 774"  
> > "WWW::Mechanize::Form: 338"  
> > "WWW::Mechanize::Form::Option: 601"
> > 
> > And here after 500 URLs:  
> > "Class: 991"  
> > "Thread: 354"  
> > "Gem::Version::Part: 343"  
> > "Regexp: 847"  
> > "MatchData: 409"  
> > "Hpricot::BogusETag: 915"  
> > "Hpricot::Text: 31707"  
> > "Gem::Version: 424"  
> > "Array: 24143"  
> > "Nokogiri::XML::Element: 2076"  
> > "WWW::Mechanize::Form::Button: 217"  
> > "WWW::Mechanize::Chain: 218"  
> > "Gem::Requirement: 380"  
> > "WWW::Mechanize::Form::Field: 874"  
> > "String: 197836"  
> > "Nokogiri::XML::SyntaxError: 2608"  
> > "Hash: 14719"  
> > "Hpricot::Elem: 21530"  
> > "Proc: 925"  
> > "WWW::Mechanize::Form: 266"  
> > "WWW::Mechanize::Form::Option: 416"
> > 
> > As you can see there is no stable increasing amount of some object.
> > 
> > What can I do more?
> 
> --  
> remember.guy do |as, often| as.you\_can - without end  
> [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)

---

<div class="post-metadata">

**Author:** ![Brabuhr](https://avatars.discourse-cdn.com/v4/letter/b/919ad9/32.png) [@Brabuhr](https://rubytalk.org/u/Brabuhr)\
**Post date:** [23 October 2009 15:36 UTC](https://rubytalk.org/t/memory-leak/55866/16 "2009-10-23T15:36:17Z")

</div>

> **[Memory-hungry Ruby daemons | Mike Perham](http://www.mikeperham.com/2009/05/25/memory-hungry-ruby-daemons/)**
>
> Ruby, OSS and the Internet

> **···**
>
> On Wed, Oct 21, 2009 at 10:52 PM, Rob Doug \<broken.m@gmail.com\> wrote:
> 
> > I retest my script again on 10k+ database and seems it stops to grow  
> > when reach 550mb size... I'm running 50 threads, so 10mb per thread. As  
> > I understand this is still not good ?

---

<div class="post-metadata">

**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\
**Post date:** [22 October 2009 08:45 UTC](https://rubytalk.org/t/memory-leak/55866/17 "2009-10-22T08:45:06Z")

</div>

With DRb you can even work with those multiple processes like they were in a single process.

Cheers

&nbsp;&nbsp;robert

> **···**
>
> On 10/22/2009 05:33 AM, John W Higgins wrote:
> 
> > May I ask why you need threads and that level of complication? Are you  
> > really that sensitive towards speed that it really matters if you simply  
> > forked processes that died after 100 downloads and just started another  
> > worker? It would seem to be easier to use a nice simple independant message  
> > queue and just fire up workers that grab messages and download away.
> > 
> > I know it's not the sexy option but what exactly are you gaining by using  
> > threads here? Much easier to get a nice single threaded worker tuned up and  
> > tight then what you appear to be going through here. In fact - I might argue  
> > that you would be better with a downloader and separater parser processes  
> > that worked independantly via messages.
> 
> --  
> remember.guy do |as, often| as.you\_can - without end  
> [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [22 October 2009 17:12 UTC](https://rubytalk.org/t/memory-leak/55866/18 "2009-10-22T17:12:01Z")

</div>

John W Higgins wrote:

> Evening Rob,
> 
> May I ask why you need threads and that level of complication? Are you  
> really that sensitive towards speed that it really matters if you simply  
> forked processes that died after 100 downloads and just started another  
> worker? It would seem to be easier to use a nice simple independant  
> message  
> queue and just fire up workers that grab messages and download away.  
> I know it's not the sexy option but what exactly are you gaining by  
> using  
> threads here? Much easier to get a nice single threaded worker tuned up  
> and  
> tight then what you appear to be going through here.

Your solution is good and simple, but there is one problem... I'll have  
1-2 mil links in my database to crawl. With only one process it will  
take months.  
Beside URLs in my list sometimes are very slow, and response time may  
take up to 10-20-30 second.  
I know the best practice should would be multi-threads with asynchronous  
sockets, but this is too complicated for me right now, maybe in the next  
versions.

> In fact - I might  
> argue  
> that you would be better with a downloader and separater parser  
> processes  
> that worked independantly via messages.

I using mechanize, as I know it parse page right after download with  
Nokogiri, so separating threads would be good when I start to use  
net/http or sockets. Unfortunately right now I should use mechanize.

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [22 October 2009 17:26 UTC](https://rubytalk.org/t/memory-leak/55866/19 "2009-10-22T17:26:51Z")

</div>

> Your number of threads is suspiciously high compared to the 50 threads  
> that you intend to run. That may be an indication that your code still  
> suffers the issue that you stuff Thread instances into the Array and  
> never remove them.

Yeah, number of threads confusing me too, but I did as you propose:

THREADS = 50  
q = SizedQueue.new(THREADS \* 2)  
threads =   
threads = (1..THREADS).map do  
&nbsp;&nbsp;&nbsp;Thread.new q do |qq|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;until qq.equal?(myLink = qq.deq)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mutex.synchronize do  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts $n +=1 # show number of link  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ($n % 100) == 0 then  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print\_class\_counts # show classes information  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Common2.post\_it(myLink)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;end  
end

So I just don't understand how amount of threads can be more than 50

> Other than that I cannot see something obvious on first sight. Maybe it  
> makes more sense to extend the memory dumping to dump only positive  
> deltas. An alternative is to decrease the dumping interval (e.g. just  
> after fetching just one URL).
> 
> If you want to throw more on this you could use set\_trace\_func to  
> capture invocations of methods named "new" along with the stack trace so  
> you get the allocation sites.
> 
> Btw, what version of Ruby is this? IIRC there was a bug with  
> Array#shift up to 1.8.6 which could also cause these effects.

ruby 1.8.6 (2008-08-11 patchlevel 287) - it's strange because I've  
updated it week ago 🙂

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Rob\_Doug](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob\_Doug](https://rubytalk.org/u/Rob_Doug)\
**Post date:** [23 October 2009 21:28 UTC](https://rubytalk.org/t/memory-leak/55866/20 "2009-10-23T21:28:40Z")

</div>

Well, seems I found solution...  
I tried to make some test on python as well. Simple script, previously  
posted, eat memory on python too... and the only way I had it to use  
forks. I checked out forkoff, but produce some strange bugs. This is the  
working code:

threads = (1..THREADS).map do  
&nbsp;&nbsp;&nbsp;Thread.new q do |qq|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;until qq.equal?(myLink = qq.deq)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mutex.synchronize do  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts ($n +=1).to\_s # + " : " + print\_class\_counts.to\_s  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fork # \<----- You need to fork it, after exit fork will release  
memory  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;begin  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent = WWW::Mechanize.new{ |agent|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent.history.max\_size=1  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent.open\_timeout = 20  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent.read\_timeout = 40  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent.user\_agent\_alias = 'Windows IE 7'  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;agent.keep\_alive = false  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;page = agent.get(myLink)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts myLink  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts page.forms.length

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;page.forms.each do |form|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rescue  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;end  
end

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

[Next page](https://rubytalk.org/t/memory-leak/55866.md?page=2)
