# Long-running daemon acquiring giant memory footprint

**URL:** <https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729>\
**Category:** ruby-talk\
**Created:** [7 November 2003 18:44 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729 "2003-11-07T18:44:59Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [7 November 2003 18:44 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/1 "2003-11-07T18:44:59Z")

</div>

I have written a long-running daemon in ruby to handle dynamic DNS updates.  
I have just recently moved it from ruby 1.6 to ruby 1.8 and updated all of  
its libraries to their latest versions (it uses dbi and dbd-postgres). The  
problem i am having now is that it appears to start out using a sane amount  
of memory (around 8mb) but then by the next day around the same time will  
be using close to 200MB for the ruby interpreter alone. The daemon code  
itself is 100% ruby so I don’t understand how this leak is happening. Are  
there any dangerous code segments I should look for that could make it do  
this? The only thing I could think of is the fact that every returned  
object from a sql query is .dup’d since ruby dbi passes a reference.  
However, these should be getting swept up automatically by the garbage  
collector. This is driving me nuts and I would love it if someone could  
point me in the right direction…

Thanks!  
-JD-

---

<div class="post-metadata">

**Author:** ![Luke\_A\_Kanies](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@Luke\_A\_Kanies](https://rubytalk.org/u/Luke_A_Kanies)\
**Post date:** [7 November 2003 21:44 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/2 "2003-11-07T21:44:00Z")

</div>

The last time this happened to me it was because I had a member of a hash  
referring to the parent. I would assume that that would reliably cause  
memory holes in just about any language. I would double check your code,  
see if you can find anything. I resolved the problem by chopping the code  
up until I found the growing part.

Good luck!

Luke

> **···**
>
> On Sat, 8 Nov 2003, Jason DiCioccio wrote:
> 
> > I have written a long-running daemon in ruby to handle dynamic DNS updates.  
> > I have just recently moved it from ruby 1.6 to ruby 1.8 and updated all of  
> > its libraries to their latest versions (it uses dbi and dbd-postgres). The  
> > problem i am having now is that it appears to start out using a sane amount  
> > of memory (around 8mb) but then by the next day around the same time will  
> > be using close to 200MB for the ruby interpreter alone. The daemon code  
> > itself is 100% ruby so I don’t understand how this leak is happening. Are  
> > there any dangerous code segments I should look for that could make it do  
> > this? The only thing I could think of is the fact that every returned  
> > object from a sql query is .dup’d since ruby dbi passes a reference.  
> > However, these should be getting swept up automatically by the garbage  
> > collector. This is driving me nuts and I would love it if someone could  
> > point me in the right direction…
> 
> –  
> “Greenspun’s Tenth Rule of Programming: any sufficiently complicated C  
> or Fortran program contains an ad hoc informally-specified bug-ridden  
> slow implementation of half of Common Lisp.” - Phil Greenspun

---

<div class="post-metadata">

**Author:** ![Joel\_VanderWerf1](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@Joel\_VanderWerf1](https://rubytalk.org/u/Joel_VanderWerf1)\
**Post date:** [7 November 2003 21:51 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/3 "2003-11-07T21:51:00Z")

</div>

Jason DiCioccio wrote:

> I have written a long-running daemon in ruby to handle dynamic DNS  
> updates. I have just recently moved it from ruby 1.6 to ruby 1.8 and  
> updated all of its libraries to their latest versions (it uses dbi and  
> dbd-postgres). The problem i am having now is that it appears to start  
> out using a sane amount of memory (around 8mb) but then by the next day  
> around the same time will be using close to 200MB for the ruby  
> interpreter alone. The daemon code itself is 100% ruby so I don’t  
> understand how this leak is happening. Are there any dangerous code  
> segments I should look for that could make it do this? The only thing I  
> could think of is the fact that every returned object from a sql query  
> is .dup’d since ruby dbi passes a reference. However, these should be  
> getting swept up automatically by the garbage collector. This is  
> driving me nuts and I would love it if someone could point me in the  
> right direction…

If the 200MB is used by objects that are still known to the interpreter  
(i.e., not garbage), then you can use ObjectSpace to find them. For  
instance, just to count objects of each class:

irb(main):001:0\> h = Hash.new(0); ObjectSpace.each\_object {|x|  
h[x.class] += 1}  
=\> 6287  
irb(main):002:0\> h  
=\> {RubyToken::TkRBRACE=\>1, IO=\>3, Regexp=\>253, IRB::WorkSpace=\>1,  
SLex::Node=\>78, RubyToken::TkRBRACK=\>1, RubyToken::TkINTEGER=\>2,  
Float=\>5, NoMemoryError=\>1, SLex=\>1, RubyToken::TkRPAREN=\>1,  
RubyToken::TkBITOR=\>2, RubyToken::TkIDENTIFIER=\>7, RubyToken::TkNL=\>1,  
RubyToken::TkCONSTANT=\>2, Proc=\>49, IRB::Context=\>1, IRB::Locale=\>1,  
RubyToken::TkSPACE=\>7, ThreadGroup=\>1, RubyToken::TkLPAREN=\>1,  
Thread=\>1, fatal=\>1, File=\>10, String=\>4413, Data=\>1,  
RubyToken::TkfLBRACE=\>1, RubyToken::TkDOT=\>3,  
IRB::ReadlineInputMethod=\>1, RubyToken::TkASSIGN=\>1, Hash=\>136,  
IRB::Irb=\>1, RubyToken::TkfLBRACK=\>1, Object=\>6, RubyLex=\>1,  
RubyToken::TkSEMICOLON=\>1, MatchData=\>111, Tempfile=\>1, Module=\>23,  
RubyToken::TkOPASGN=\>1, SystemStackError=\>1, Binding=\>2, Class=\>345,  
Array=\>806}

---

<div class="post-metadata">

**Author:** ![Mauricio\_Fernndez](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@Mauricio\_Fernndez](https://rubytalk.org/u/Mauricio_Fernndez)\
**Post date:** [8 November 2003 15:13 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/4 "2003-11-08T15:13:45Z")

</div>

Ruby does mark&sweep, not reference counting; I thus fail to see why  
such a structure would fail to be reclaimed.

Do you mean something like

a = {}  
a[:foo] = a

?

> **···**
>
> On Sat, Nov 08, 2003 at 06:44:00AM +0900, Luke A. Kanies wrote:
> 
> > The last time this happened to me it was because I had a member of a hash  
> > referring to the parent. I would assume that that would reliably cause  
> > memory holes in just about any language. I would double check your code,
> 
> –  
> \_ \_
> 
> > > \_\_ \_\_ _| |_ \_\_\_ \_ \_\_ \_\_\_ \_\_ \_ \_ \_\_  
> > > '\_ \ / _`| __/__ | '_` \_ \ / ` | ’ \   
> > > ) | (| | |_\_\_ \ | | | | | (_| | | | |  
> > > .\_\_/ \_,_|\ __|__ /| |_| |_|\__,_|_| |\_|  
> > > Running Debian GNU/Linux Sid (unstable)  
> > > batsman dot geo at yahoo dot com
> 
> Real Men don’t make backups. They upload it via ftp and let the world mirror it.  
> – Linus Torvalds

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [9 November 2003 13:22 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/5 "2003-11-09T13:22:08Z")

</div>

> Jason DiCioccio wrote:
> 
> > I have written a long-running daemon in ruby to handle dynamic DNS  
> > updates. I have just recently moved it from ruby 1.6 to ruby 1.8 and  
> > updated all of its libraries to their latest versions (it uses dbi and  
> > dbd-postgres). The problem i am having now is that it appears to start  
> > out using a sane amount of memory (around 8mb) but then by the next day  
> > around the same time will be using close to 200MB for the ruby  
> > interpreter alone. The daemon code itself is 100% ruby so I don’t  
> > understand how this leak is happening. Are there any dangerous code  
> > segments I should look for that could make it do this? The only thing I  
> > could think of is the fact that every returned object from a sql query  
> > is .dup’d since ruby dbi passes a reference. However, these should be  
> > getting swept up automatically by the garbage collector. This is  
> > driving me nuts and I would love it if someone could point me in the  
> > right direction…
> 
> If the 200MB is used by objects that are still known to the interpreter  
> (i.e., not garbage), then you can use ObjectSpace to find them. For  
> instance, just to count objects of each class:
> 
> irb(main):001:0\> h = Hash.new(0); ObjectSpace.each\_object {|x| h[x.class]  
> += 1} =\> 6287  
> irb(main):002:0\> h  
> =\> {RubyToken::TkRBRACE=\>1, IO=\>3, Regexp=\>253, IRB::WorkSpace=\>1,  
> SLex::Node=\>78, RubyToken::TkRBRACK=\>1, RubyToken::TkINTEGER=\>2,  
> Float=\>5, NoMemoryError=\>1, SLex=\>1, RubyToken::TkRPAREN=\>1,  
> RubyToken::TkBITOR=\>2, RubyToken::TkIDENTIFIER=\>7, RubyToken::TkNL=\>1,  
> RubyToken::TkCONSTANT=\>2, Proc=\>49, IRB::Context=\>1, IRB::Locale=\>1,  
> RubyToken::TkSPACE=\>7, ThreadGroup=\>1, RubyToken::TkLPAREN=\>1, Thread=\>1,  
> fatal=\>1, File=\>10, String=\>4413, Data=\>1, RubyToken::TkfLBRACE=\>1,  
> RubyToken::TkDOT=\>3, IRB::ReadlineInputMethod=\>1, RubyToken::TkASSIGN=\>1,  
> Hash=\>136, IRB::Irb=\>1, RubyToken::TkfLBRACK=\>1, Object=\>6, RubyLex=\>1,  
> RubyToken::TkSEMICOLON=\>1, MatchData=\>111, Tempfile=\>1, Module=\>23,  
> RubyToken::TkOPASGN=\>1, SystemStackError=\>1, Binding=\>2, Class=\>345,  
> Array=\>806}

I had a lot of hope for this method when I first tried it out.  
Unfortunately at the moment the process is using 162M resident memory and  
here’s the output:

Total Objects: 8499 Detail: {EOFError=\>2, DBI::Row=\>81  
, SQLPool=\>1, IO=\>3, DBI::StatementHandle=\>49, fatal=\>1,  
SystemStackError=\>1, Fl  
oat=\>18, Binding=\>1, Mutex=\>4, String=\>5218, DBI::DatabaseHandle=\>9,  
DBI::DBD:😛  
g::PgCoerce=\>9, DBI::DBD::Pg::Tuples=\>49, TCPSocket=\>7, ODS=\>1,  
DBI::DBD::Pg::Dr  
iver=\>1, NoncritError=\>3, RR=\>6, Thread=\>23, DBI::DBD::Pg::Statement=\>49,  
DBI::S  
QL::PreparedStatement=\>49, User=\>4, ConditionVariable=\>2, ThreadGroup=\>1,  
DBI:😃  
BD::Pg::Database=\>9, Event=\>22, Proc=\>16, File=\>1, Hash=\>253, Range=\>11,  
PGconn=

> 9, PGresult=\>49, CritError=\>2, Errno::ECONNABORTED=\>1, Object=\>4,  
> Bignum=\>5, IO  
> Error=\>5, Whois=\>1, TCPServer=\>2, DBI::DriverHandle=\>1, NoMemoryError=\>1,  
> Module  
> =\>34, Array=\>1922, Sql=\>5, MatchData=\>150, Class=\>232, Regexp=\>172}

At it’s peak it reaches about 20k. I’m guessing the drop occurs when the  
garbage collector steps in. However, the memory size of the process  
doesn’t seem to drop at that point… I’m running FreeBSD 4.9 with ruby  
1.8.1 (2003-10-31). The problem was also occuring with release and stable  
builds of 1.8.0 though as well. It was not, however, occurring in 1.6.x.

Any ideas? Bug?

Thanks!  
-JD-

> **···**
>
> –On Saturday, November 8, 2003 6:51 AM +0900 Joel VanderWerf [vjoel@PATH.Berkeley.EDU](mailto:vjoel@PATH.Berkeley.EDU) wrote:

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [17 November 2003 16:45 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/6 "2003-11-17T16:45:54Z")

</div>

Luke,  
After a while of debugging I found a bunch of objects that were being  
created that apparently contain one of the primary keys in one of my  
database tables. The thing is is the query should only return one result.  
This particular row is hardly ever referenced either. So now I have found  
a line in my code that is something like you might have been referring to:

```
  nsEntryId = nsEntryId[0][0]

```

That is called quite often, would that cause the object to stay around?  
Is this what you were referring to?

If not, I’ll have to dig a lot deeper and find out why these values are  
scattered all over the object space.

Thanks in advance,  
Jason DiCioccio

> **···**
>
> –On Saturday, November 8, 2003 6:44 AM +0900 “Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) wrote:
> 
> > The last time this happened to me it was because I had a member of a hash  
> > referring to the parent. I would assume that that would reliably cause  
> > memory holes in just about any language. I would double check your code,  
> > see if you can find anything. I resolved the problem by chopping the code  
> > up until I found the growing part.
> > 
> > Good luck!
> > 
> > Luke
> > 
> > –  
> > “Greenspun’s Tenth Rule of Programming: any sufficiently complicated C  
> > or Fortran program contains an ad hoc informally-specified bug-ridden  
> > slow implementation of half of Common Lisp.” - Phil Greenspun

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [11 November 2003 19:07 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/7 "2003-11-11T19:07:42Z")

</div>

Hmm… So since my memory problem does not appear to be in ruby’s object  
space and all of my code is ruby code, should this be considered a bug and  
submitted as such? If so, I can submit this as such. I wish I knew more  
about how ruby’s GC worked. One thing about this daemon is that it is used  
heavily and accepts many queries/updates per second at times. Is it  
possible that the GC is unable to ‘keep up’ ? Or does it not work that way  
(I assume it doesn’t.) I just don’t see these object counts leading to  
process sizes of over 200M after running a while.

Thanks again!  
-JD-

> **···**
>
> –On Sunday, November 9, 2003 10:22 PM +0900 Jason DiCioccio [jd@ods.org](mailto:jd@ods.org) wrote:
> 
> > > If the 200MB is used by objects that are still known to the interpreter  
> > > (i.e., not garbage), then you can use ObjectSpace to find them. For  
> > > instance, just to count objects of each class:
> > > 
> > > irb(main):001:0\> h = Hash.new(0); ObjectSpace.each\_object {|x| h[x.class]  
> > > += 1} =\> 6287  
> > > irb(main):002:0\> h  
> > > =\> {RubyToken::TkRBRACE=\>1, IO=\>3, Regexp=\>253, IRB::WorkSpace=\>1,  
> > > SLex::Node=\>78, RubyToken::TkRBRACK=\>1, RubyToken::TkINTEGER=\>2,  
> > > Float=\>5, NoMemoryError=\>1, SLex=\>1, RubyToken::TkRPAREN=\>1,  
> > > RubyToken::TkBITOR=\>2, RubyToken::TkIDENTIFIER=\>7, RubyToken::TkNL=\>1,  
> > > RubyToken::TkCONSTANT=\>2, Proc=\>49, IRB::Context=\>1, IRB::Locale=\>1,  
> > > RubyToken::TkSPACE=\>7, ThreadGroup=\>1, RubyToken::TkLPAREN=\>1, Thread=\>1,  
> > > fatal=\>1, File=\>10, String=\>4413, Data=\>1, RubyToken::TkfLBRACE=\>1,  
> > > RubyToken::TkDOT=\>3, IRB::ReadlineInputMethod=\>1, RubyToken::TkASSIGN=\>1,  
> > > Hash=\>136, IRB::Irb=\>1, RubyToken::TkfLBRACK=\>1, Object=\>6, RubyLex=\>1,  
> > > RubyToken::TkSEMICOLON=\>1, MatchData=\>111, Tempfile=\>1, Module=\>23,  
> > > RubyToken::TkOPASGN=\>1, SystemStackError=\>1, Binding=\>2, Class=\>345,  
> > > Array=\>806}
> > 
> > I had a lot of hope for this method when I first tried it out.  
> > Unfortunately at the moment the process is using 162M resident memory and  
> > here’s the output:
> > 
> > Total Objects: 8499 Detail: {EOFError=\>2, DBI::Row=\>81  
> > , SQLPool=\>1, IO=\>3, DBI::StatementHandle=\>49, fatal=\>1,  
> > SystemStackError=\>1, Fl oat=\>18, Binding=\>1, Mutex=\>4, String=\>5218,  
> > DBI::DatabaseHandle=\>9, DBI::DBD:😛 g::PgCoerce=\>9,  
> > DBI::DBD::Pg::Tuples=\>49, TCPSocket=\>7, ODS=\>1, DBI::DBD::Pg::Dr iver=\>1,  
> > NoncritError=\>3, RR=\>6, Thread=\>23, DBI::DBD::Pg::Statement=\>49, DBI::S  
> > QL::PreparedStatement=\>49, User=\>4, ConditionVariable=\>2, ThreadGroup=\>1,  
> > DBI:😃 BD::Pg::Database=\>9, Event=\>22, Proc=\>16, File=\>1, Hash=\>253,  
> > Range=\>11, PGconn=
> > 
> > > 9, PGresult=\>49, CritError=\>2, Errno::ECONNABORTED=\>1, Object=\>4,  
> > > Bignum=\>5, IO  
> > > Error=\>5, Whois=\>1, TCPServer=\>2, DBI::DriverHandle=\>1, NoMemoryError=\>1,  
> > > Module =\>34, Array=\>1922, Sql=\>5, MatchData=\>150, Class=\>232, Regexp=\>172}
> > 
> > At it’s peak it reaches about 20k. I’m guessing the drop occurs when the  
> > garbage collector steps in. However, the memory size of the process  
> > doesn’t seem to drop at that point… I’m running FreeBSD 4.9 with ruby  
> > 1.8.1 (2003-10-31). The problem was also occuring with release and  
> > stable builds of 1.8.0 though as well. It was not, however, occurring in  
> > 1.6.x.
> > 
> > Any ideas? Bug?
> > 
> > Thanks!  
> > -JD-

---

<div class="post-metadata">

**Author:** ![Luke\_A\_Kanies](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@Luke\_A\_Kanies](https://rubytalk.org/u/Luke_A_Kanies)\
**Post date:** [17 November 2003 16:49 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/8 "2003-11-17T16:49:59Z")

</div>

Yep, that's exactly what I'm referring to. If you undef your local copy  
of nsEntryId, it still maintains a pointer to itself, so it becomes a  
closed off lump of storage. I don't know much about ruby's memory  
management, but it apparently can't catch these problems (I know perl  
can't).

I expect that if you fix that self-reference, your growth will go away.

Good luck!

> **···**
>
> On Mon, 17 Nov 2003, Jason DiCioccio wrote:
> 
> > Luke,  
> > &nbsp;&nbsp;After a while of debugging I found a bunch of objects that were being  
> > created that apparently contain one of the primary keys in one of my  
> > database tables. The thing is is the query should only return one result.  
> > This particular row is hardly ever referenced either. So now I have found  
> > a line in my code that is something like you might have been referring to:
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;nsEntryId = nsEntryId[0][0]
> > 
> > &nbsp;&nbsp;That is called quite often, would that cause the object to stay around?  
> > Is this what you were referring to?
> 
> --  
> I went to a restaurant that serves "breakfast at anytime". So I  
> ordered French Toast during the Renaissance. -- Stephen Wright

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [17 November 2003 16:57 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/9 "2003-11-17T16:57:43Z")

</div>

I imagine changing it to:

nsEntryId = nsEntryId[0][0].dup would take care of the problem? Or just using another object name?

Regards,  
-JD-

> **···**
>
> --On Monday, November 17, 2003 10:50 AM -0600 "Luke A. Kanies" \<luke@madstop.com\> wrote:
> 
> > On Mon, 17 Nov 2003, Jason DiCioccio wrote:
> > 
> > > Luke,  
> > > &nbsp;&nbsp;After a while of debugging I found a bunch of objects that were being  
> > > created that apparently contain one of the primary keys in one of my  
> > > database tables. The thing is is the query should only return one  
> > > result. This particular row is hardly ever referenced either. So now I  
> > > have found a line in my code that is something like you might have been  
> > > referring to:
> > > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;nsEntryId = nsEntryId[0][0]
> > > 
> > > &nbsp;&nbsp;That is called quite often, would that cause the object to stay around?  
> > > Is this what you were referring to?
> > 
> > Yep, that's exactly what I'm referring to. If you undef your local copy  
> > of nsEntryId, it still maintains a pointer to itself, so it becomes a  
> > closed off lump of storage. I don't know much about ruby's memory  
> > management, but it apparently can't catch these problems (I know perl  
> > can't).
> > 
> > I expect that if you fix that self-reference, your growth will go away.
> > 
> > Good luck!
> > 
> > --  
> > I went to a restaurant that serves "breakfast at anytime". So I  
> > ordered French Toast during the Renaissance. -- Stephen Wright

---

<div class="post-metadata">

**Author:** ![Robert](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Robert](https://rubytalk.org/u/Robert)\
**Post date:** [17 November 2003 17:12 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/10 "2003-11-17T17:12:18Z")

</div>

“Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) schrieb im Newsbeitrag  
news:Pine.GSO.4.51.0311171048090.2656@pixie…

> > Luke,  
> > After a while of debugging I found a bunch of objects that were  
> > being  
> > created that apparently contain one of the primary keys in one of my  
> > database tables. The thing is is the query should only return one  
> > result.  
> > This particular row is hardly ever referenced either. So now I have  
> > found  
> > a line in my code that is something like you might have been referring  
> > to:
> > 
> > ```
> > nsEntryId = nsEntryId[0][0]
> > 
> > ```
> > 
> > That is called quite often, would that cause the object to stay  
> > around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

> > Is this what you were referring to?
> 
> Yep, that’s exactly what I’m referring to. If you undef your local copy  
> of nsEntryId, it still maintains a pointer to itself, so it becomes a  
> closed off lump of storage. I don’t know much about ruby’s memory  
> management, but it apparently can’t catch these problems (I know perl  
> can’t).

Hmmm, since Ruby used mark and sweep GC it should be able to catch loops -  
self references as well as loops made of several instances. It seems to  
me that this is unlikely to be the reason for the memory growth. I’d  
rather assume some references as indicated above.

Kind regards

```
robert

```

> **···**
>
> > On Mon, 17 Nov 2003, Jason DiCioccio wrote:

---

<div class="post-metadata">

**Author:** ![Luke\_A\_Kanies](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@Luke\_A\_Kanies](https://rubytalk.org/u/Luke_A_Kanies)\
**Post date:** [17 November 2003 17:02 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/11 "2003-11-17T17:02:50Z")

</div>

Um, I don’t really know. I can’t think of a generally good reason to have  
an object refer to itself. Does the object actually need a  
self-reference, or are you just reusing the name for convenience? If it’s  
just convenience, I definitely recommend using a different name.

Duplication might also solve the problem, but I’m not really sure. I’m  
still a relative ruby newbie and have not figured out all the details of  
when you get a reference vs. a copy vs. a real variable. Others can  
hopefully chime in with those answers.

Luke

> **···**
>
> On Tue, 18 Nov 2003, Jason DiCioccio wrote:
> 
> > I imagine changing it to:
> > 
> > nsEntryId = nsEntryId[0][0].dup would take care of the problem? Or just  
> > using another object name?
> 
> –  
> : We are looking at a newspaper clipping labelled “Huffington  
> : Herald 11/12/96” in which “Huffington” states, in part:  
> : “The inhabitants of Tiera del Fuego … have a single word  
> : that means ‘to look at each other hoping that either will offer to do  
> : something that both parties desire but are unwilling to do.’”  
> : Does anyone know that word?  
> Management?

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [18 November 2003 03:59 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/12 "2003-11-18T03:59:01Z")

</div>

Greetings,

> **···**
>
> –On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> 
> > “Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) schrieb im Newsbeitrag  
> > news:Pine.GSO.4.51.0311171048090.2656@pixie…
> > 
> > > On Mon, 17 Nov 2003, Jason DiCioccio wrote:
> > > 
> > > > Luke,  
> > > > After a while of debugging I found a bunch of objects that were  
> > > > being  
> > > > created that apparently contain one of the primary keys in one of my  
> > > > database tables. The thing is is the query should only return one  
> > > > result.  
> > > > This particular row is hardly ever referenced either. So now I have  
> > > > found  
> > > > a line in my code that is something like you might have been referring  
> > > > to:
> > > > 
> > > > ```
> > > > nsEntryId = nsEntryId[0][0]
> > > > 
> > > > ```
> > > > 
> > > > That is called quite often, would that cause the object to stay  
> > > > around?
> > 
> > Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.
> 
> I’ve tried removing this line and it does indeed appear to be the problem.  
> There was no reference from nsEntryId[0][0] back to nsEntryId. Is this a  
> bug? I don’t think it had this problem in ruby 1.6.x, so it appears  
> something changed in the handling of arrays between 1.6 and 1.8.
> 
> Regards,  
> -JD-

---

<div class="post-metadata">

**Author:** ![Robert](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Robert](https://rubytalk.org/u/Robert)\
**Post date:** [18 November 2003 09:17 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/13 "2003-11-18T09:17:16Z")

</div>

“Jason DiCioccio” [jd@ods.org](mailto:jd@ods.org) schrieb im Newsbeitrag  
news:2147483647.1069109940@[10.0.2.2]…

> Greetings,
> 
> > “Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) schrieb im Newsbeitrag  
> > news:Pine.GSO.4.51.0311171048090.2656@pixie…
> > 
> > > > Luke,  
> > > > After a while of debugging I found a bunch of objects that were  
> > > > being  
> > > > created that apparently contain one of the primary keys in one of  
> > > > my  
> > > > database tables. The thing is is the query should only return one  
> > > > result.  
> > > > This particular row is hardly ever referenced either. So now I  
> > > > have  
> > > > found  
> > > > a line in my code that is something like you might have been  
> > > > referring  
> > > > to:
> > > > 
> > > > ```
> > > > nsEntryId = nsEntryId[0][0]
> > > > 
> > > > ```
> > > > 
> > > > That is called quite often, would that cause the object to stay  
> > > > around?
> > 
> > Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.
> 
> I’ve tried removing this line and it does indeed appear to be the  
> problem.  
> There was no reference from nsEntryId[0][0] back to nsEntryId. Is this  
> a  
> bug? I don’t think it had this problem in ruby 1.6.x, so it appears  
> something changed in the handling of arrays between 1.6 and 1.8.

Hm. Since I don’t know the data at hand, I’m not in a position to comment  
more specific. I do wonder however, how this assignment should be  
responsible for growth of mem consumption if nsEntryId[0][0] does not  
backreference nsEntryId.

I’d do a to\_yaml of nsEntryId before and after the assignment to be able  
to analyse the object graph. Maybe there’s a backreference you  
overlooked.

The other option seems to be a bug in the garbage collector but I’m not  
very inclined in believing this.

If there is an extension involved that could hold on to the memory in ways  
not easy to determine. That’s another option that comes to my mind.

Kind regards

```
robert

```

> **···**
>
> > –On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme \> [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> > 
> > > > On Mon, 17 Nov 2003, Jason DiCioccio wrote:

---

<div class="post-metadata">

**Author:** ![ts1](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@ts1](https://rubytalk.org/u/ts1)\
**Post date:** [18 November 2003 09:52 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/14 "2003-11-18T09:52:58Z")

</div>

> I've tried removing this line and it does indeed appear to be the problem.  
> There was no reference from nsEntryId[0][0] back to nsEntryId. Is this a  
> bug? I don't think it had this problem in ruby 1.6.x, so it appears  
> something changed in the handling of arrays between 1.6 and 1.8.

Well, if you can, try to reproduce the problem with a \*small\* script and  
post it if you still have the problem

Guy Decoux

---

<div class="post-metadata">

**Author:** ![Jason\_DiCioccio1](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jason\_DiCioccio1](https://rubytalk.org/u/Jason_DiCioccio1)\
**Post date:** [18 November 2003 14:11 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/15 "2003-11-18T14:11:04Z")

</div>

Robert,  
The object in question is just data from ruby dbi object. You could be  
right about the back reference, however, the back reference was definitely  
not something that I am aware is there. It could be a pointer that ruby  
DBI puts into all of its objects?

Here’s a code snippet to illustrate what it contains:

nsEntryId, = Thread.current[:sql].query( “SELECT NSEntryId FROM NSEntry  
WHERE LOWER(Host) = LOWER(?) AND DomainId=?::bigint AND OwnerId=?::bigint  
AND UserId=?::bigint #{type} #{ target if target }”, domain[0],  
Domain::id(domain[1]), Domain::ownerId(domain[1]), @aUser.id )

The Sql#query method is below as well:

def query( sqlRequest, \*vars )  
@conn.prepare( sqlRequest ) do |sqlVar|  
begin  
sqlVar.execute( \*vars )  
rescue =\> detail  
if detail.message =~ /server closed the connection unexpectedly/  
log(“Connection to DB unexpectedly closed”)  
$sqlPool.reopen!( @conn )  
end

```
    Thread.current.raise NoncritError, "An error occurred: #{detail}, 

```

"#{sqlRequest}""  
end

```
  case sqlRequest.word( 0 )
    when /^INSERT/ then return
    when /^UPDATE/ then return
    when /^DELETE/ then return
  end

  result = []
  rcnt = 0
  while (row = sqlVar.fetch) != nil
    result.push row.dup
    rcnt = rcnt.next
  end

  return [nil, rcnt] if result.length == 0
  return [result, rcnt]
end

```

end

Any ideas where this reference might be coming from if it exists?

Regards,  
-JD-

> **···**
>
> –On Tuesday, November 18, 2003 6:17 PM +0900 Robert Klemme [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> 
> > “Jason DiCioccio” [jd@ods.org](mailto:jd@ods.org) schrieb im Newsbeitrag  
> > news:2147483647.1069109940@[10.0.2.2]…
> > 
> > > Greetings,
> > > 
> > > –On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme \>\> [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> > > 
> > > > “Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) schrieb im Newsbeitrag  
> > > > news:Pine.GSO.4.51.0311171048090.2656@pixie…
> > > > 
> > > > > On Mon, 17 Nov 2003, Jason DiCioccio wrote:
> > > > > 
> > > > > > Luke,  
> > > > > > After a while of debugging I found a bunch of objects that were  
> > > > > > being  
> > > > > > created that apparently contain one of the primary keys in one of  
> > > > > > my  
> > > > > > database tables. The thing is is the query should only return one  
> > > > > > result.  
> > > > > > This particular row is hardly ever referenced either. So now I  
> > > > > > have  
> > > > > > found  
> > > > > > a line in my code that is something like you might have been  
> > > > > > referring  
> > > > > > to:
> > > > > > 
> > > > > > ```
> > > > > > nsEntryId = nsEntryId[0][0]
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > That is called quite often, would that cause the object to stay  
> > > > > > around?
> > > > 
> > > > Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.
> > > 
> > > I’ve tried removing this line and it does indeed appear to be the  
> > > problem.  
> > > There was no reference from nsEntryId[0][0] back to nsEntryId. Is this  
> > > a  
> > > bug? I don’t think it had this problem in ruby 1.6.x, so it appears  
> > > something changed in the handling of arrays between 1.6 and 1.8.
> > 
> > Hm. Since I don’t know the data at hand, I’m not in a position to comment  
> > more specific. I do wonder however, how this assignment should be  
> > responsible for growth of mem consumption if nsEntryId[0][0] does not  
> > backreference nsEntryId.
> > 
> > I’d do a to\_yaml of nsEntryId before and after the assignment to be able  
> > to analyse the object graph. Maybe there’s a backreference you  
> > overlooked.
> > 
> > The other option seems to be a bug in the garbage collector but I’m not  
> > very inclined in believing this.
> > 
> > If there is an extension involved that could hold on to the memory in ways  
> > not easy to determine. That’s another option that comes to my mind.
> > 
> > Kind regards
> > 
> > ```
> > robert
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Robert](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Robert](https://rubytalk.org/u/Robert)\
**Post date:** [18 November 2003 15:32 UTC](https://rubytalk.org/t/long-running-daemon-acquiring-giant-memory-footprint/8729/16 "2003-11-18T15:32:16Z")

</div>

“Jason DiCioccio” [jd@ods.org](mailto:jd@ods.org) schrieb im Newsbeitrag  
[news:2147483647.1069146662@pcp04900144pcs.anapol01.md.comcast.net](mailto:news:2147483647.1069146662@pcp04900144pcs.anapol01.md.comcast.net)…

> > “Jason DiCioccio” [jd@ods.org](mailto:jd@ods.org) schrieb im Newsbeitrag  
> > news:2147483647.1069109940@[10.0.2.2]…
> > 
> > > Greetings,
> > > 
> > > > “Luke A. Kanies” [luke@madstop.com](mailto:luke@madstop.com) schrieb im Newsbeitrag  
> > > > news:Pine.GSO.4.51.0311171048090.2656@pixie…
> > > > 
> > > > > > Luke,  
> > > > > > After a while of debugging I found a bunch of objects that  
> > > > > > were  
> > > > > > being  
> > > > > > created that apparently contain one of the primary keys in one  
> > > > > > of  
> > > > > > my  
> > > > > > database tables. The thing is is the query should only return  
> > > > > > one  
> > > > > > result.  
> > > > > > This particular row is hardly ever referenced either. So now I  
> > > > > > have  
> > > > > > found  
> > > > > > a line in my code that is something like you might have been  
> > > > > > referring  
> > > > > > to:
> > > > > > 
> > > > > > ```
> > > > > > nsEntryId = nsEntryId[0][0]
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > That is called quite often, would that cause the object to  
> > > > > > stay  
> > > > > > around?
> > > > 
> > > > Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.
> > > 
> > > I’ve tried removing this line and it does indeed appear to be the  
> > > problem.  
> > > There was no reference from nsEntryId[0][0] back to nsEntryId. Is  
> > > this  
> > > a  
> > > bug? I don’t think it had this problem in ruby 1.6.x, so it appears  
> > > something changed in the handling of arrays between 1.6 and 1.8.
> > 
> > Hm. Since I don’t know the data at hand, I’m not in a position to  
> > comment  
> > more specific. I do wonder however, how this assignment should be  
> > responsible for growth of mem consumption if nsEntryId[0][0] does not  
> > backreference nsEntryId.
> > 
> > I’d do a to\_yaml of nsEntryId before and after the assignment to be  
> > able  
> > to analyse the object graph. Maybe there’s a backreference you  
> > overlooked.
> > 
> > The other option seems to be a bug in the garbage collector but I’m  
> > not  
> > very inclined in believing this.
> > 
> > If there is an extension involved that could hold on to the memory in  
> > ways  
> > not easy to determine. That’s another option that comes to my mind.
> > 
> > Kind regards
> > 
> > ```
> > robert
> > 
> > ```
> 
> Robert,  
> The object in question is just data from ruby dbi object. You could  
> be  
> right about the back reference, however, the back reference was  
> definitely  
> not something that I am aware is there. It could be a pointer that ruby  
> DBI puts into all of its objects?
> 
> Here’s a code snippet to illustrate what it contains:
> 
> nsEntryId, = Thread.current[:sql].query( “SELECT NSEntryId FROM NSEntry  
> WHERE LOWER(Host) = LOWER(?) AND DomainId=?::bigint AND  
> OwnerId=?::bigint  
> AND UserId=?::bigint #{type} #{ target if target }”, domain[0],  
> Domain::id(domain[1]), Domain::ownerId(domain[1]), @aUser.id )
> 
> The Sql#query method is below as well:
> 
> def query( sqlRequest, \*vars )  
> @conn.prepare( sqlRequest ) do |sqlVar|  
> begin  
> sqlVar.execute( \*vars )  
> rescue =\> detail  
> if detail.message =~ /server closed the connection unexpectedly/  
> log(“Connection to DB unexpectedly closed”)  
> $sqlPool.reopen!( @conn )  
> end
> 
> ```
> Thread.current.raise NoncritError, "An error occurred:
> 
> ```

#{detail},

> "#{sqlRequest}""  
> end
> 
> ```
> case sqlRequest.word( 0 )
> when /^INSERT/ then return
> when /^UPDATE/ then return
> when /^DELETE/ then return
> end
> 
> result = []
> rcnt = 0
> while (row = sqlVar.fetch) != nil
> result.push row.dup
> rcnt = rcnt.next
> end
> 
> return [nil, rcnt] if result.length == 0
> return [result, rcnt]
> end
> 
> ```
> 
> end
> 
> Any ideas where this reference might be coming from if it exists?

Only the row.dup. What type does it return? Is it an Array or some DBI  
internal class? If it’s DBI internal, I’d try row.to\_a instead to make  
sure, that only values are retained and not the row collection.

Cheers

```
robert

```

> **···**
>
> > –On Tuesday, November 18, 2003 6:17 PM +0900 Robert Klemme \> [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> > 
> > > > –On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme \> \>\> [bob.news@gmx.net](mailto:bob.news@gmx.net) wrote:
> > > > 
> > > > > > On Mon, 17 Nov 2003, Jason DiCioccio wrote: