# GC help

**URL:** <https://rubytalk.org/t/gc-help/45514>\
**Category:** ruby-talk\
**Created:** [30 March 2008 19:31 UTC](https://rubytalk.org/t/gc-help/45514 "2008-03-30T19:31:55Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joey\_Marino](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@Joey\_Marino](https://rubytalk.org/u/Joey_Marino)\
**Post date:** [30 March 2008 19:31 UTC](https://rubytalk.org/t/gc-help/45514/1 "2008-03-30T19:31:55Z")

</div>

I am still running out of memory with my ruby application. For some reason,  
there are objects that are not getting reclaimed long after their use is up.  
I understand the mark and sweep GC algorithm. Is there a method in Ruby to  
set an object's memory space to be collected?

> **···**
>
> --  
> Joey Marino

---

<div class="post-metadata">

**Author:** ![Tim\_Hunter4](https://avatars.discourse-cdn.com/v4/letter/t/ea5d25/32.png) [@Tim\_Hunter4](https://rubytalk.org/u/Tim_Hunter4)\
**Post date:** [30 March 2008 19:53 UTC](https://rubytalk.org/t/gc-help/45514/2 "2008-03-30T19:53:52Z")

</div>

Joey Marino wrote:

> I am still running out of memory with my ruby application. For some reason,  
> there are objects that are not getting reclaimed long after their use is up.  
> I understand the mark and sweep GC algorithm. Is there a method in Ruby to  
> set an object's memory space to be collected?

Ruby's GC will eventually collect unreferenced objects without any intervention on your part. It may not collect them on the first sweep, but it will collect them. You can force GC to run earlier than normal by calling GC.start, but that doesn't necessarily mean that every unreferenced object will be collected on that sweep.

In one of your earlier posts you said you were using a 3rd-party library? Have you determined that it's not responsible for the memory allocations?

Have you determined that no references exist to objects that you think should have been cleaned up?

> **···**
>
> --  
> RMagick: [http://rmagick.rubyforge.org/](http://rmagick.rubyforge.org/)  
> RMagick 2: [http://rmagick.rubyforge.org/rmagick2.html](http://rmagick.rubyforge.org/rmagick2.html)

---

<div class="post-metadata">

**Author:** ![Joey\_Marino](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@Joey\_Marino](https://rubytalk.org/u/Joey_Marino)\
**Post date:** [31 March 2008 00:07 UTC](https://rubytalk.org/t/gc-help/45514/3 "2008-03-31T00:07:11Z")

</div>

I found out it is not the 3rd party code. I am downloading images (~320,000  
ct.) as strings, and saving each to a file. The file handlers are getting  
closed. Using the Dike gem, I think I determined the bulk of the leaked  
memory is in these string variables. But what I don't understand is that the  
variables containing the strings are local to a method of an object and  
should be unreferenced when that method is done, right? I've invoked GC at  
the end of the method and after each iteration that the method gets called.  
I've even set the strings to nil after they've been saved to a file and  
before gc is called. The app is using 2G of ram and 4G of swap before it  
runs of out memory and crashes about 1/3rd of the way through. I'm really  
starting to doubt Ruby's ability to do memory intensive work. Any ideas?

> **···**
>
> On Sun, Mar 30, 2008 at 3:53 PM, Tim Hunter \<TimHunter@nc.rr.com\> wrote:
> 
> > Joey Marino wrote:  
> > \> I am still running out of memory with my ruby application. For some  
> > reason,  
> > \> there are objects that are not getting reclaimed long after their use is  
> > up.  
> > \> I understand the mark and sweep GC algorithm. Is there a method in Ruby  
> > to  
> > \> set an object's memory space to be collected?
> > 
> > Ruby's GC will eventually collect unreferenced objects without any  
> > intervention on your part. It may not collect them on the first sweep,  
> > but it will collect them. You can force GC to run earlier than normal by  
> > calling GC.start, but that doesn't necessarily mean that every  
> > unreferenced object will be collected on that sweep.
> > 
> > In one of your earlier posts you said you were using a 3rd-party  
> > library? Have you determined that it's not responsible for the memory  
> > allocations?
> > 
> > Have you determined that no references exist to objects that you think  
> > should have been cleaned up?
> > 
> > --  
> > RMagick: [http://rmagick.rubyforge.org/](http://rmagick.rubyforge.org/)  
> > RMagick 2: [http://rmagick.rubyforge.org/rmagick2.html](http://rmagick.rubyforge.org/rmagick2.html)
> 
> --  
> Joey Marino

---

<div class="post-metadata">

**Author:** ![M\_Edward\_Ed\_Borasky1](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@M\_Edward\_Ed\_Borasky1](https://rubytalk.org/u/M_Edward_Ed_Borasky1)\
**Post date:** [31 March 2008 00:16 UTC](https://rubytalk.org/t/gc-help/45514/4 "2008-03-31T00:16:10Z")

</div>

Joey Marino wrote:

> I found out it is not the 3rd party code. I am downloading images (~320,000  
> ct.) as strings, and saving each to a file.

Why? If the images come from the Internet and go to disk, why do you need to read them into Ruby working storage? Can't you just shell out to "wget" or "curl"?

---

<div class="post-metadata">

**Author:** ![Tim\_Hunter4](https://avatars.discourse-cdn.com/v4/letter/t/ea5d25/32.png) [@Tim\_Hunter4](https://rubytalk.org/u/Tim_Hunter4)\
**Post date:** [31 March 2008 01:45 UTC](https://rubytalk.org/t/gc-help/45514/5 "2008-03-31T01:45:00Z")

</div>

Joey Marino wrote:

> I found out it is not the 3rd party code. I am downloading images (~320,000  
> ct.) as strings, and saving each to a file. The file handlers are getting  
> closed. Using the Dike gem, I think I determined the bulk of the leaked  
> memory is in these string variables. But what I don't understand is that the  
> variables containing the strings are local to a method of an object and  
> should be unreferenced when that method is done, right? I've invoked GC at  
> the end of the method and after each iteration that the method gets called.  
> I've even set the strings to nil after they've been saved to a file and  
> before gc is called. The app is using 2G of ram and 4G of swap before it  
> runs of out memory and crashes about 1/3rd of the way through. I'm really  
> starting to doubt Ruby's ability to do memory intensive work. Any ideas?

Many people have used Ruby for long-running tasks that use a lot of memory. If Ruby was not collecting unused strings, don't you think somebody would have noticed it by now?

Actually, I think you've ruled out Ruby being the problem. What else is running? Could it be using up the memory?

> **···**
>
> --  
> RMagick: [http://rmagick.rubyforge.org/](http://rmagick.rubyforge.org/)  
> RMagick 2: [http://rmagick.rubyforge.org/rmagick2.html](http://rmagick.rubyforge.org/rmagick2.html)

---

<div class="post-metadata">

**Author:** ![Bill\_Kelly](https://avatars.discourse-cdn.com/v4/letter/b/cc9497/32.png) [@Bill\_Kelly](https://rubytalk.org/u/Bill_Kelly)\
**Post date:** [31 March 2008 02:31 UTC](https://rubytalk.org/t/gc-help/45514/6 "2008-03-31T02:31:09Z")

</div>

> I found out it is not the 3rd party code. I am downloading images (~320,000  
> ct.) as strings, and saving each to a file. The file handlers are getting  
> closed. Using the Dike gem, I think I determined the bulk of the leaked  
> memory is in these string variables. But what I don't understand is that the  
> variables containing the strings are local to a method of an object and  
> should be unreferenced when that method is done, right? I've invoked GC at  
> the end of the method and after each iteration that the method gets called.  
> I've even set the strings to nil after they've been saved to a file and  
> before gc is called. The app is using 2G of ram and 4G of swap before it  
> runs of out memory and crashes about 1/3rd of the way through. I'm really  
> starting to doubt Ruby's ability to do memory intensive work. Any ideas?

Do you have a small bit of code that reproduces the problem?

How are you downloading the images? Net::HTTP ? Or...?

If you can provide a small program that exhibits the memory  
leak I'm sure others here would be happy to try it on their  
systems as well.

Regards,

Bill

> **···**
>
> From: "Joey Marino" \<joey.da3rd@gmail.com\>

---

<div class="post-metadata">

**Author:** ![Christopher\_Dicely](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@Christopher\_Dicely](https://rubytalk.org/u/Christopher_Dicely)\
**Post date:** [2 April 2008 05:32 UTC](https://rubytalk.org/t/gc-help/45514/7 "2008-04-02T05:32:06Z")

</div>

Usually, yes. However, there are ways a reference could be kept live  
beyond the end of the method. Obviously, if you pass the string out of the  
method to something else that keeps a reference to it, that would do it, but  
also if you pass a closure (e.g., a proc object) that refers to the methods  
application's binding (I think that's the right terminology) that gets stored  
somewhere else, that would keep the method's local variables "live" even after  
the method exits.

Its hard to say if something like that might be happening without the code.

> **···**
>
> On Sun, Mar 30, 2008 at 5:07 PM, Joey Marino \<joey.da3rd@gmail.com\> wrote:
> 
> > I found out it is not the 3rd party code. I am downloading images (~320,000  
> > ct.) as strings, and saving each to a file. The file handlers are getting  
> > closed. Using the Dike gem, I think I determined the bulk of the leaked  
> > memory is in these string variables. But what I don't understand is that the  
> > variables containing the strings are local to a method of an object and  
> > should be unreferenced when that method is done, right?

---

<div class="post-metadata">

**Author:** ![Joey\_Marino](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@Joey\_Marino](https://rubytalk.org/u/Joey_Marino)\
**Post date:** [31 March 2008 00:49 UTC](https://rubytalk.org/t/gc-help/45514/8 "2008-03-31T00:49:51Z")

</div>

Good question, unfortunately this is not a simple HTTP server. It's an  
industry standard client/server communication called RETS. Real Estate  
Transaction Standard. It requires a third party library in order to  
interface with the server. I wish it were that simple!!

> **···**
>
> On Sun, Mar 30, 2008 at 8:16 PM, M. Edward (Ed) Borasky \<znmeb@cesmail.net\> wrote:
> 
> > Joey Marino wrote:  
> > \> I found out it is not the 3rd party code. I am downloading images  
> > (~320,000  
> > \> ct.) as strings, and saving each to a file.
> > 
> > Why? If the images come from the Internet and go to disk, why do you  
> > need to read them into Ruby working storage? Can't you just shell out to  
> > "wget" or "curl"?
> 
> --  
> Joey Marino

---

<div class="post-metadata">

**Author:** ![Joey\_Marino](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@Joey\_Marino](https://rubytalk.org/u/Joey_Marino)\
**Post date:** [2 April 2008 05:11 UTC](https://rubytalk.org/t/gc-help/45514/9 "2008-04-02T05:11:52Z")

</div>

I shouldn't have criticized ruby like that, it actually is my second  
favorite language right behind PHP. If I learned RoR, it might be my most  
favorite. It really does a good job, is easy to write, and has a lot of  
features. Not to mention a great community.  
I am just really frustrated about this stupid app that I can't get working  
right. I am working on a workaround since it is only going to do a major  
download once then update daily.  
Thanks for all the help though, it feels good that there is a place to turn  
to for help.

> **···**
>
> On Sun, Mar 30, 2008 at 10:31 PM, Bill Kelly \<billk@cts.com\> wrote:
> 
> > From: "Joey Marino" \<joey.da3rd@gmail.com\>  
> > \>  
> > \> I found out it is not the 3rd party code. I am downloading images  
> > (~320,000  
> > \> ct.) as strings, and saving each to a file. The file handlers are  
> > getting  
> > \> closed. Using the Dike gem, I think I determined the bulk of the leaked  
> > \> memory is in these string variables. But what I don't understand is that  
> > the  
> > \> variables containing the strings are local to a method of an object and  
> > \> should be unreferenced when that method is done, right? I've invoked GC  
> > at  
> > \> the end of the method and after each iteration that the method gets  
> > called.  
> > \> I've even set the strings to nil after they've been saved to a file and  
> > \> before gc is called. The app is using 2G of ram and 4G of swap before it  
> > \> runs of out memory and crashes about 1/3rd of the way through. I'm  
> > really  
> > \> starting to doubt Ruby's ability to do memory intensive work. Any ideas?
> > 
> > Do you have a small bit of code that reproduces the problem?
> > 
> > How are you downloading the images? Net::HTTP ? Or...?
> > 
> > If you can provide a small program that exhibits the memory  
> > leak I'm sure others here would be happy to try it on their  
> > systems as well.
> > 
> > Regards,
> > 
> > Bill
> 
> --  
> Joey Marino

---

<div class="post-metadata">

**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\
**Post date:** [2 April 2008 10:02 UTC](https://rubytalk.org/t/gc-help/45514/10 "2008-04-02T10:02:14Z")

</div>

I have noticed :-\>

Michal

> **···**
>
> On 31/03/2008, Tim Hunter \<TimHunter@nc.rr.com\> wrote:
> 
> > Joey Marino wrote:
> > 
> > \> I found out it is not the 3rd party code. I am downloading images  
> > (~320,000  
> > \> ct.) as strings, and saving each to a file. The file handlers are getting  
> > \> closed. Using the Dike gem, I think I determined the bulk of the leaked  
> > \> memory is in these string variables. But what I don't understand is that  
> > the  
> > \> variables containing the strings are local to a method of an object and  
> > \> should be unreferenced when that method is done, right? I've invoked GC at  
> > \> the end of the method and after each iteration that the method gets  
> > called.  
> > \> I've even set the strings to nil after they've been saved to a file and  
> > \> before gc is called. The app is using 2G of ram and 4G of swap before it  
> > \> runs of out memory and crashes about 1/3rd of the way through. I'm really  
> > \> starting to doubt Ruby's ability to do memory intensive work. Any ideas?  
> > \>
> > 
> > Many people have used Ruby for long-running tasks that use a lot of memory.  
> > If Ruby was not collecting unused strings, don't you think somebody would  
> > have noticed it by now?

---

<div class="post-metadata">

**Author:** ![Joey\_Marino](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@Joey\_Marino](https://rubytalk.org/u/Joey_Marino)\
**Post date:** [2 April 2008 16:52 UTC](https://rubytalk.org/t/gc-help/45514/11 "2008-04-02T16:52:53Z")

</div>

Ok, I was able to get it all into one class. The problem lies in this class:  
class Picture

def initialize(db,rets,rets\_class)  
&nbsp;&nbsp;&nbsp;@db = db  
&nbsp;&nbsp;&nbsp;@rets = rets  
&nbsp;&nbsp;&nbsp;@rets\_class = rets\_class  
&nbsp;&nbsp;&nbsp;@attempts = 0  
end

def getPic(key)  
&nbsp;&nbsp;&nbsp;begin  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request = GetObjectRequest.new(@rets\_class, "Photo")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request.add\_all\_objects(key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = @rets.session.get\_object(get\_object\_request)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type\_suffixes = { "image/jpeg" =\> "jpg"}  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;makePicDir(key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response.each\_object do |object\_descriptor|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;object\_key = object\_descriptor.object\_key  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;obj\_id = object\_descriptor.object\_id  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type = object\_descriptor.content\_type  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;description = object\_descriptor.description  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print "#{object\_key} object \##{object\_id}"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print ", description: #{description}" if !description.empty?  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#puts  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffix = content\_type\_suffixes[content\_type]  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pic = object\_descriptor.data\_as\_string  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;savePic(key,obj\_id.to\_s,suffix,description,pic)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = nil  
&nbsp;&nbsp;rescue =\> e  
&nbsp;&nbsp;&nbsp;&nbsp;puts "Error retrieving pictures for #{key}: " + e  
&nbsp;&nbsp;&nbsp;&nbsp;if @attempts \<= 5  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts += 1  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "retrying"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry  
&nbsp;&nbsp;&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "failed"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts = 0  
&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;end  
&nbsp;&nbsp;@attempts = 0  
end

def getThumb(key)  
&nbsp;&nbsp;&nbsp;begin  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request = GetObjectRequest.new(@rets\_class, "Thumbnail")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request.add\_all\_objects(key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = @rets.session.get\_object(get\_object\_request)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type\_suffixes = { "image/jpeg" =\> "jpg"}  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response.each\_object do |object\_descriptor|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;object\_key = object\_descriptor.object\_key  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;obj\_id = object\_descriptor.object\_id  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type = object\_descriptor.content\_type  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;description = object\_descriptor.description  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print "#{object\_key} object \##{object\_id}"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print ", description: #{description}" if !description.empty?  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#puts  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffix = content\_type\_suffixes[content\_type]  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pic = object\_descriptor.data\_as\_string  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;savePic(key,obj\_id.to\_s,suffix,description,pic,true)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = nil  
&nbsp;&nbsp;rescue =\> e  
&nbsp;&nbsp;&nbsp;&nbsp;puts "Error retrieving thumbs for #{key}: " + e  
&nbsp;&nbsp;&nbsp;&nbsp;if @attempts \<= 5  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts += 1  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "retrying"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry  
&nbsp;&nbsp;&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "failed"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts = 0  
&nbsp;&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;end  
&nbsp;&nbsp;@attempts = 0  
end

def makePicDir(key)  
&nbsp;&nbsp;&nbsp;FileUtils.mkpath("#{$pic\_dir}#{key}/thumb")  
end

def savePic(key,id,suffix,desc,pic,thumb\_bool=false)  
&nbsp;&nbsp;&nbsp;if thumb\_bool  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_name = $pic\_dir + key + "/thumb/" + id + "." + suffix  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;location = "/" + key + "/thumb/" + id + "." + suffix  
&nbsp;&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_name = $pic\_dir + key + "/" + id + "." + suffix  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;location = "/" + key + "/" + id + "." + suffix  
&nbsp;&nbsp;&nbsp;end  
&nbsp;&nbsp;&nbsp;self.savePicFile(file\_name,pic)  
&nbsp;&nbsp;&nbsp;size = File.size(file\_name)  
&nbsp;&nbsp;&nbsp;if thumb\_bool  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.insertThumbDB(key,id,location)  
&nbsp;&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.insertPicDB(key,id,desc,size,location)  
&nbsp;&nbsp;&nbsp;end  
end

def savePicFile(file\_name,pic)  
&nbsp;&nbsp;&nbsp;f = File.open(file\_name, "wb")  
&nbsp;&nbsp;&nbsp;f \<\< pic  
&nbsp;&nbsp;&nbsp;f.close  
end

def insertPicDB(key,id,desc,size,location)  
&nbsp;&nbsp;&nbsp;description = @db.database.escape\_string(desc)  
&nbsp;&nbsp;&nbsp;if  
@db.DBinsert("PICS","pkey,id,description,size,location","#{key},#{id},'#{description}','#{size}','#{location}'")  
&nbsp;&nbsp;&nbsp;# puts "#{key} #{id} pic added"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print ":"  
&nbsp;&nbsp;&nbsp;end  
end

def insertThumbDB(key,id,location)  
&nbsp;&nbsp;&nbsp;if @db.DBupdate("PICS","thumb = '#{location}'"," pkey = #{key} and id =  
#{id}")  
&nbsp;&nbsp;&nbsp;&nbsp;# puts "#{key} #{id} thumb added"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print "."  
&nbsp;&nbsp;&nbsp;end  
end

def deletePic(key)  
&nbsp;&nbsp;&nbsp;self.deletePicDir(key)  
&nbsp;&nbsp;&nbsp;self.deletePicDB(key)  
end

def deletePicDir(key)  
&nbsp;&nbsp;&nbsp;if File.exists?("#{$pic\_dir}#{key}")  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FileUtils.remove\_dir("#{$pic\_dir}#{key}")  
&nbsp;&nbsp;&nbsp;end  
end

def deletePicDB(key)  
&nbsp;&nbsp;if @db.DBdelete("PICS","pkey = #{key}")  
&nbsp;&nbsp;&nbsp;&nbsp;print "-"  
# puts "#{key} pics deleted from db"  
&nbsp;&nbsp;end  
end

end #end class

> **···**
>
> On Wed, Apr 2, 2008 at 6:02 AM, Michal Suchanek \<hramrach@centrum.cz\> wrote:
> 
> > On 31/03/2008, Tim Hunter \<TimHunter@nc.rr.com\> wrote:  
> > \> Joey Marino wrote:  
> > \>  
> > \> \> I found out it is not the 3rd party code. I am downloading images  
> > \> (~320,000  
> > \> \> ct.) as strings, and saving each to a file. The file handlers are  
> > getting  
> > \> \> closed. Using the Dike gem, I think I determined the bulk of the  
> > leaked  
> > \> \> memory is in these string variables. But what I don't understand is  
> > that  
> > \> the  
> > \> \> variables containing the strings are local to a method of an object  
> > and  
> > \> \> should be unreferenced when that method is done, right? I've invoked  
> > GC at  
> > \> \> the end of the method and after each iteration that the method gets  
> > \> called.  
> > \> \> I've even set the strings to nil after they've been saved to a file  
> > and  
> > \> \> before gc is called. The app is using 2G of ram and 4G of swap before  
> > it  
> > \> \> runs of out memory and crashes about 1/3rd of the way through. I'm  
> > really  
> > \> \> starting to doubt Ruby's ability to do memory intensive work. Any  
> > ideas?  
> > \> \>  
> > \>  
> > \> Many people have used Ruby for long-running tasks that use a lot of  
> > memory.  
> > \> If Ruby was not collecting unused strings, don't you think somebody  
> > would  
> > \> have noticed it by now?
> > 
> > I have noticed :-\>
> > 
> > Michal
> 
> --  
> Joey Marino

---

<div class="post-metadata">

**Author:** ![Roger\_Pack4](https://avatars.discourse-cdn.com/v4/letter/r/bb73d2/32.png) [@Roger\_Pack4](https://rubytalk.org/u/Roger_Pack4)\
**Post date:** [4 April 2008 23:26 UTC](https://rubytalk.org/t/gc-help/45514/12 "2008-04-04T23:26:55Z")

</div>

> > Many people have used Ruby for long-running tasks that use a lot of memory.  
> > If Ruby was not collecting unused strings, don't you think somebody would  
> > have noticed it by now?
> 
> I have noticed :-\>
> 
> Michal

I have too, and it drives me crazy when my mongrel instances eat up  
600MB of memory.  
I'd be willing to offer a bounty of $150 to anyone able to clear this  
up. It happens especially often when you run multiple threads, it  
seems.  
Probably a rails thing, but anyway.  
Enough ranting.  
Have a good one.  
-R

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Jan\_Svitok](https://avatars.discourse-cdn.com/v4/letter/j/cab0a1/32.png) [@Jan\_Svitok](https://rubytalk.org/u/Jan_Svitok)\
**Post date:** [2 April 2008 17:38 UTC](https://rubytalk.org/t/gc-help/45514/13 "2008-04-02T17:38:30Z")

</div>

Few suggestions, some not really related to the problem:

> Ok, I was able to get it all into one class. The problem lies in this class:  
> class Picture
> 
> &nbsp;&nbsp;def initialize(db,rets,rets\_class)  
> &nbsp;&nbsp;&nbsp;@db = db  
> &nbsp;&nbsp;&nbsp;@rets = rets  
> &nbsp;&nbsp;&nbsp;@rets\_class = rets\_class  
> &nbsp;&nbsp;&nbsp;@attempts = 0  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def getPic(key)  
> &nbsp;&nbsp;&nbsp;begin  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request = GetObjectRequest.new(@rets\_class, "Photo")  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request.add\_all\_objects(key)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = @rets.session.get\_object(get\_object\_request)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type\_suffixes = { "image/jpeg" =\> "jpg"}

Make content\_type\_suffixes a class constant, or member if you need to  
append to it.  
Now you are constructing and destructing the object on each method call.

> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;makePicDir(key)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response.each\_object do |object\_descriptor|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;object\_key = object\_descriptor.object\_key  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;obj\_id = object\_descriptor.object\_id  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type = object\_descriptor.content\_type  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;description = object\_descriptor.description  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print "#{object\_key} object \##{object\_id}"

- #print ", description: #{description}" if !description.empty?  
+ #print ", description: #{description}" unless  
description.empty? # a matter of taste/style

> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#puts  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffix = content\_type\_suffixes[content\_type]  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pic = object\_descriptor.data\_as\_string  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;savePic(key,obj\_id.to\_s,suffix,description,pic)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = nil  
> &nbsp;&nbsp;rescue =\> e  
> &nbsp;&nbsp;&nbsp;&nbsp;puts "Error retrieving pictures for #{key}: " + e  
> &nbsp;&nbsp;&nbsp;&nbsp;if @attempts \<= 5  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts += 1  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "retrying"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry  
> &nbsp;&nbsp;&nbsp;&nbsp;else  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "failed"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts = 0  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end  
> &nbsp;&nbsp;@attempts = 0  
> &nbsp;&nbsp;end

It seems that you could refactor the common code of these two methods  
in to a new one.  
The benefit would be shorter/more readable code and better  
responsibility split, the drawback slower execution.

> &nbsp;&nbsp;def getThumb(key)  
> &nbsp;&nbsp;&nbsp;begin  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request = GetObjectRequest.new(@rets\_class, "Thumbnail")  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_request.add\_all\_objects(key)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = @rets.session.get\_object(get\_object\_request)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type\_suffixes = { "image/jpeg" =\> "jpg"}  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response.each\_object do |object\_descriptor|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;object\_key = object\_descriptor.object\_key  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;obj\_id = object\_descriptor.object\_id  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content\_type = object\_descriptor.content\_type  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;description = object\_descriptor.description  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print "#{object\_key} object \##{object\_id}"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#print ", description: #{description}" if !description.empty?  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#puts  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffix = content\_type\_suffixes[content\_type]  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pic = object\_descriptor.data\_as\_string  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;savePic(key,obj\_id.to\_s,suffix,description,pic,true)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;get\_object\_response = nil  
> &nbsp;&nbsp;rescue =\> e  
> &nbsp;&nbsp;&nbsp;&nbsp;puts "Error retrieving thumbs for #{key}: " + e  
> &nbsp;&nbsp;&nbsp;&nbsp;if @attempts \<= 5  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts += 1  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "retrying"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry  
> &nbsp;&nbsp;&nbsp;&nbsp;else  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;puts "failed"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@attempts = 0  
> &nbsp;&nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end  
> &nbsp;&nbsp;@attempts = 0  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def makePicDir(key)  
> &nbsp;&nbsp;&nbsp;FileUtils.mkpath("#{$pic\_dir}#{key}/thumb")  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def savePic(key,id,suffix,desc,pic,thumb\_bool=false)  
> &nbsp;&nbsp;&nbsp;if thumb\_bool  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_name = $pic\_dir + key + "/thumb/" + id + "." + suffix  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;location = "/" + key + "/thumb/" + id + "." + suffix  
> &nbsp;&nbsp;&nbsp;else  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file\_name = $pic\_dir + key + "/" + id + "." + suffix  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;location = "/" + key + "/" + id + "." + suffix  
> &nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;&nbsp;self.savePicFile(file\_name,pic)

self is not necessary, the same below

> &nbsp;&nbsp;&nbsp;size = File.size(file\_name)  
> &nbsp;&nbsp;&nbsp;if thumb\_bool  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.insertThumbDB(key,id,location)  
> &nbsp;&nbsp;&nbsp;else  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.insertPicDB(key,id,desc,size,location)  
> &nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def savePicFile(file\_name,pic)

- f = File.open(file\_name, "wb")  
+ File.open(file\_name, "wb") do |f|

> &nbsp;&nbsp;&nbsp;f \<\< pic

- f.close  
+ end # automatic close on exceptions

> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def insertPicDB(key,id,desc,size,location)  
> &nbsp;&nbsp;&nbsp;description = @db.database.escape\_string(desc)  
> &nbsp;&nbsp;&nbsp;if  
> @db.DBinsert("PICS","pkey,id,description,size,location","#{key},#{id},'#{description}','#{size}','#{location}'")  
> &nbsp;&nbsp;&nbsp;# puts "#{key} #{id} pic added"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print ":"  
> &nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def insertThumbDB(key,id,location)  
> &nbsp;&nbsp;&nbsp;if @db.DBupdate("PICS","thumb = '#{location}'"," pkey = #{key} and id = #{id}")  
> &nbsp;&nbsp;&nbsp;&nbsp;# puts "#{key} #{id} thumb added"  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print "."  
> &nbsp;&nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def deletePic(key)  
> &nbsp;&nbsp;&nbsp;self.deletePicDir(key)  
> &nbsp;&nbsp;&nbsp;self.deletePicDB(key)  
> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def deletePicDir(key)

- if File.exists?("#{$pic\_dir}#{key}")  
- FileUtils.remove\_dir("#{$pic\_dir}#{key}")  
- end  
+ pic\_dir = $pic\_dir + key # save one allocation  
+ FileUtils.remove\_dir pic\_dir if File.directory? pic\_dir

> &nbsp;&nbsp;end
> 
> &nbsp;&nbsp;def deletePicDB(key)  
> &nbsp;&nbsp;if @db.DBdelete("PICS","pkey = #{key}")  
> &nbsp;&nbsp;&nbsp;&nbsp;print "-"  
> # puts "#{key} pics deleted from db"  
> &nbsp;&nbsp;end  
> &nbsp;&nbsp;end
> 
> end #end class

I'd suspect the database code is keeping some cache. This code seems fine.

I see you are using libRETS. Did you try to rule it out by replacing  
calls to libRETS (especially to data\_as\_string) with some stubs  
(create a random long string on the fly).  
If you are on unix, you can use IO.read("/dev/random", 100000). If you  
are on Windows, choose another long enough file to read.

The documentation says that GetData() abandons ownership to the object  
it returns. It's possible that SWIG-generated wrapper doesn't handle  
this properly.

Jano

> **···**
>
> On Wed, Apr 2, 2008 at 6:52 PM, Joey Marino \<joey.da3rd@gmail.com\> wrote:

---

<div class="post-metadata">

**Author:** ![Michael\_Fellinger1](https://avatars.discourse-cdn.com/v4/letter/m/e47774/32.png) [@Michael\_Fellinger1](https://rubytalk.org/u/Michael_Fellinger1)\
**Post date:** [5 April 2008 08:23 UTC](https://rubytalk.org/t/gc-help/45514/14 "2008-04-05T08:23:44Z")

</div>

> \>\> Many people have used Ruby for long-running tasks that use a lot of memory.  
> \>\> If Ruby was not collecting unused strings, don't you think somebody would  
> \>\> have noticed it by now?  
> \>  
> \> I have noticed :-\>  
> \>  
> \> Michal
> 
> I have too, and it drives me crazy when my mongrel instances eat up  
> 600MB of memory.  
> I'd be willing to offer a bounty of $150 to anyone able to clear this  
> up. It happens especially often when you run multiple threads, it  
> seems.

Send me the 150 and i send you a copy of ramaze 😉

> **···**
>
> On 4/5/08, Roger Pack \<rogerpack2005@gmail.com\> wrote:
> 
> > Probably a rails thing, but anyway.  
> > Enough ranting.  
> > Have a good one.  
> > -R  
> > --  
> > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Jack\_Dempsey](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jack\_Dempsey](https://rubytalk.org/u/Jack_Dempsey)\
**Post date:** [6 April 2008 02:07 UTC](https://rubytalk.org/t/gc-help/45514/15 "2008-04-06T02:07:53Z")

</div>

What version of ruby are you running? I recently saw an issue using  
1.8.6 with a low patch level. Upgrading to p114 might solve things for  
you.

> **···**
>
> On Fri, Apr 4, 2008 at 7:26 PM, Roger Pack \<rogerpack2005@gmail.com\> wrote:
> 
> > \>\> Many people have used Ruby for long-running tasks that use a lot of memory.  
> > \>\> If Ruby was not collecting unused strings, don't you think somebody would  
> > \>\> have noticed it by now?  
> > \>  
> > \> I have noticed :-\>  
> > \>  
> > \> Michal
> > 
> > I have too, and it drives me crazy when my mongrel instances eat up  
> > 600MB of memory.  
> > I'd be willing to offer a bounty of $150 to anyone able to clear this  
> > up. It happens especially often when you run multiple threads, it  
> > seems.  
> > Probably a rails thing, but anyway.  
> > Enough ranting.  
> > Have a good one.  
> > -R  
> > --  
> > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).
> 
> hat very