Long-running daemon acquiring giant memory footprint

ruby-talk
1

I have written a long-running daemon in ruby to handle dynamic DNS updates.
I have just recently moved it from ruby 1.6 to ruby 1.8 and updated all of
its libraries to their latest versions (it uses dbi and dbd-postgres). The
problem i am having now is that it appears to start out using a sane amount
of memory (around 8mb) but then by the next day around the same time will
be using close to 200MB for the ruby interpreter alone. The daemon code
itself is 100% ruby so I don’t understand how this leak is happening. Are
there any dangerous code segments I should look for that could make it do
this? The only thing I could think of is the fact that every returned
object from a sql query is .dup’d since ruby dbi passes a reference.
However, these should be getting swept up automatically by the garbage
collector. This is driving me nuts and I would love it if someone could
point me in the right direction…

Thanks!
-JD-

2

The last time this happened to me it was because I had a member of a hash
referring to the parent. I would assume that that would reliably cause
memory holes in just about any language. I would double check your code,
see if you can find anything. I resolved the problem by chopping the code
up until I found the growing part.

Good luck!

Luke

···

On Sat, 8 Nov 2003, Jason DiCioccio wrote:

I have written a long-running daemon in ruby to handle dynamic DNS updates.
I have just recently moved it from ruby 1.6 to ruby 1.8 and updated all of
its libraries to their latest versions (it uses dbi and dbd-postgres). The
problem i am having now is that it appears to start out using a sane amount
of memory (around 8mb) but then by the next day around the same time will
be using close to 200MB for the ruby interpreter alone. The daemon code
itself is 100% ruby so I don’t understand how this leak is happening. Are
there any dangerous code segments I should look for that could make it do
this? The only thing I could think of is the fact that every returned
object from a sql query is .dup’d since ruby dbi passes a reference.
However, these should be getting swept up automatically by the garbage
collector. This is driving me nuts and I would love it if someone could
point me in the right direction…

–
“Greenspun’s Tenth Rule of Programming: any sufficiently complicated C
or Fortran program contains an ad hoc informally-specified bug-ridden
slow implementation of half of Common Lisp.” - Phil Greenspun

3

Jason DiCioccio wrote:

I have written a long-running daemon in ruby to handle dynamic DNS
updates. I have just recently moved it from ruby 1.6 to ruby 1.8 and
updated all of its libraries to their latest versions (it uses dbi and
dbd-postgres). The problem i am having now is that it appears to start
out using a sane amount of memory (around 8mb) but then by the next day
around the same time will be using close to 200MB for the ruby
interpreter alone. The daemon code itself is 100% ruby so I don’t
understand how this leak is happening. Are there any dangerous code
segments I should look for that could make it do this? The only thing I
could think of is the fact that every returned object from a sql query
is .dup’d since ruby dbi passes a reference. However, these should be
getting swept up automatically by the garbage collector. This is
driving me nuts and I would love it if someone could point me in the
right direction…

If the 200MB is used by objects that are still known to the interpreter
(i.e., not garbage), then you can use ObjectSpace to find them. For
instance, just to count objects of each class:

irb(main):001:0> h = Hash.new(0); ObjectSpace.each_object {|x|
h[x.class] += 1}
=> 6287
irb(main):002:0> h
=> {RubyToken::TkRBRACE=>1, IO=>3, Regexp=>253, IRB::WorkSpace=>1,
SLex::Node=>78, RubyToken::TkRBRACK=>1, RubyToken::TkINTEGER=>2,
Float=>5, NoMemoryError=>1, SLex=>1, RubyToken::TkRPAREN=>1,
RubyToken::TkBITOR=>2, RubyToken::TkIDENTIFIER=>7, RubyToken::TkNL=>1,
RubyToken::TkCONSTANT=>2, Proc=>49, IRB::Context=>1, IRB::Locale=>1,
RubyToken::TkSPACE=>7, ThreadGroup=>1, RubyToken::TkLPAREN=>1,
Thread=>1, fatal=>1, File=>10, String=>4413, Data=>1,
RubyToken::TkfLBRACE=>1, RubyToken::TkDOT=>3,
IRB::ReadlineInputMethod=>1, RubyToken::TkASSIGN=>1, Hash=>136,
IRB::Irb=>1, RubyToken::TkfLBRACK=>1, Object=>6, RubyLex=>1,
RubyToken::TkSEMICOLON=>1, MatchData=>111, Tempfile=>1, Module=>23,
RubyToken::TkOPASGN=>1, SystemStackError=>1, Binding=>2, Class=>345,
Array=>806}

4

Ruby does mark&sweep, not reference counting; I thus fail to see why
such a structure would fail to be reclaimed.

Do you mean something like

a = {}
a[:foo] = a

?

···

On Sat, Nov 08, 2003 at 06:44:00AM +0900, Luke A. Kanies wrote:

The last time this happened to me it was because I had a member of a hash
referring to the parent. I would assume that that would reliably cause
memory holes in just about any language. I would double check your code,

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
'_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Real Men don’t make backups. They upload it via ftp and let the world mirror it.
– Linus Torvalds

5

Jason DiCioccio wrote:

I have written a long-running daemon in ruby to handle dynamic DNS
updates. I have just recently moved it from ruby 1.6 to ruby 1.8 and
updated all of its libraries to their latest versions (it uses dbi and
dbd-postgres). The problem i am having now is that it appears to start
out using a sane amount of memory (around 8mb) but then by the next day
around the same time will be using close to 200MB for the ruby
interpreter alone. The daemon code itself is 100% ruby so I don’t
understand how this leak is happening. Are there any dangerous code
segments I should look for that could make it do this? The only thing I
could think of is the fact that every returned object from a sql query
is .dup’d since ruby dbi passes a reference. However, these should be
getting swept up automatically by the garbage collector. This is
driving me nuts and I would love it if someone could point me in the
right direction…

If the 200MB is used by objects that are still known to the interpreter
(i.e., not garbage), then you can use ObjectSpace to find them. For
instance, just to count objects of each class:

irb(main):001:0> h = Hash.new(0); ObjectSpace.each_object {|x| h[x.class]
+= 1} => 6287
irb(main):002:0> h
=> {RubyToken::TkRBRACE=>1, IO=>3, Regexp=>253, IRB::WorkSpace=>1,
SLex::Node=>78, RubyToken::TkRBRACK=>1, RubyToken::TkINTEGER=>2,
Float=>5, NoMemoryError=>1, SLex=>1, RubyToken::TkRPAREN=>1,
RubyToken::TkBITOR=>2, RubyToken::TkIDENTIFIER=>7, RubyToken::TkNL=>1,
RubyToken::TkCONSTANT=>2, Proc=>49, IRB::Context=>1, IRB::Locale=>1,
RubyToken::TkSPACE=>7, ThreadGroup=>1, RubyToken::TkLPAREN=>1, Thread=>1,
fatal=>1, File=>10, String=>4413, Data=>1, RubyToken::TkfLBRACE=>1,
RubyToken::TkDOT=>3, IRB::ReadlineInputMethod=>1, RubyToken::TkASSIGN=>1,
Hash=>136, IRB::Irb=>1, RubyToken::TkfLBRACK=>1, Object=>6, RubyLex=>1,
RubyToken::TkSEMICOLON=>1, MatchData=>111, Tempfile=>1, Module=>23,
RubyToken::TkOPASGN=>1, SystemStackError=>1, Binding=>2, Class=>345,
Array=>806}

I had a lot of hope for this method when I first tried it out.
Unfortunately at the moment the process is using 162M resident memory and
here’s the output:

Total Objects: 8499 Detail: {EOFError=>2, DBI::Row=>81
, SQLPool=>1, IO=>3, DBI::StatementHandle=>49, fatal=>1,
SystemStackError=>1, Fl
oat=>18, Binding=>1, Mutex=>4, String=>5218, DBI::DatabaseHandle=>9,
DBI::DBD::stuck_out_tongue:
g::PgCoerce=>9, DBI::DBD::Pg::Tuples=>49, TCPSocket=>7, ODS=>1,
DBI::DBD::Pg::Dr
iver=>1, NoncritError=>3, RR=>6, Thread=>23, DBI::DBD::Pg::Statement=>49,
DBI::S
QL::PreparedStatement=>49, User=>4, ConditionVariable=>2, ThreadGroup=>1,
DBI::smiley:
BD::Pg::Database=>9, Event=>22, Proc=>16, File=>1, Hash=>253, Range=>11,
PGconn=

9, PGresult=>49, CritError=>2, Errno::ECONNABORTED=>1, Object=>4,
Bignum=>5, IO
Error=>5, Whois=>1, TCPServer=>2, DBI::DriverHandle=>1, NoMemoryError=>1,
Module
=>34, Array=>1922, Sql=>5, MatchData=>150, Class=>232, Regexp=>172}

At it’s peak it reaches about 20k. I’m guessing the drop occurs when the
garbage collector steps in. However, the memory size of the process
doesn’t seem to drop at that point… I’m running FreeBSD 4.9 with ruby
1.8.1 (2003-10-31). The problem was also occuring with release and stable
builds of 1.8.0 though as well. It was not, however, occurring in 1.6.x.

Any ideas? Bug?

Thanks!
-JD-

···

–On Saturday, November 8, 2003 6:51 AM +0900 Joel VanderWerf vjoel@PATH.Berkeley.EDU wrote:

6

Luke,
After a while of debugging I found a bunch of objects that were being
created that apparently contain one of the primary keys in one of my
database tables. The thing is is the query should only return one result.
This particular row is hardly ever referenced either. So now I have found
a line in my code that is something like you might have been referring to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to stay around?
Is this what you were referring to?

If not, I’ll have to dig a lot deeper and find out why these values are
scattered all over the object space.

Thanks in advance,
Jason DiCioccio

···

–On Saturday, November 8, 2003 6:44 AM +0900 “Luke A. Kanies” luke@madstop.com wrote:

The last time this happened to me it was because I had a member of a hash
referring to the parent. I would assume that that would reliably cause
memory holes in just about any language. I would double check your code,
see if you can find anything. I resolved the problem by chopping the code
up until I found the growing part.

Good luck!

Luke

–
“Greenspun’s Tenth Rule of Programming: any sufficiently complicated C
or Fortran program contains an ad hoc informally-specified bug-ridden
slow implementation of half of Common Lisp.” - Phil Greenspun

7

Hmm… So since my memory problem does not appear to be in ruby’s object
space and all of my code is ruby code, should this be considered a bug and
submitted as such? If so, I can submit this as such. I wish I knew more
about how ruby’s GC worked. One thing about this daemon is that it is used
heavily and accepts many queries/updates per second at times. Is it
possible that the GC is unable to ‘keep up’ ? Or does it not work that way
(I assume it doesn’t.) I just don’t see these object counts leading to
process sizes of over 200M after running a while.

Thanks again!
-JD-

···

–On Sunday, November 9, 2003 10:22 PM +0900 Jason DiCioccio jd@ods.org wrote:

If the 200MB is used by objects that are still known to the interpreter
(i.e., not garbage), then you can use ObjectSpace to find them. For
instance, just to count objects of each class:

irb(main):001:0> h = Hash.new(0); ObjectSpace.each_object {|x| h[x.class]
+= 1} => 6287
irb(main):002:0> h
=> {RubyToken::TkRBRACE=>1, IO=>3, Regexp=>253, IRB::WorkSpace=>1,
SLex::Node=>78, RubyToken::TkRBRACK=>1, RubyToken::TkINTEGER=>2,
Float=>5, NoMemoryError=>1, SLex=>1, RubyToken::TkRPAREN=>1,
RubyToken::TkBITOR=>2, RubyToken::TkIDENTIFIER=>7, RubyToken::TkNL=>1,
RubyToken::TkCONSTANT=>2, Proc=>49, IRB::Context=>1, IRB::Locale=>1,
RubyToken::TkSPACE=>7, ThreadGroup=>1, RubyToken::TkLPAREN=>1, Thread=>1,
fatal=>1, File=>10, String=>4413, Data=>1, RubyToken::TkfLBRACE=>1,
RubyToken::TkDOT=>3, IRB::ReadlineInputMethod=>1, RubyToken::TkASSIGN=>1,
Hash=>136, IRB::Irb=>1, RubyToken::TkfLBRACK=>1, Object=>6, RubyLex=>1,
RubyToken::TkSEMICOLON=>1, MatchData=>111, Tempfile=>1, Module=>23,
RubyToken::TkOPASGN=>1, SystemStackError=>1, Binding=>2, Class=>345,
Array=>806}

I had a lot of hope for this method when I first tried it out.
Unfortunately at the moment the process is using 162M resident memory and
here’s the output:

Total Objects: 8499 Detail: {EOFError=>2, DBI::Row=>81
, SQLPool=>1, IO=>3, DBI::StatementHandle=>49, fatal=>1,
SystemStackError=>1, Fl oat=>18, Binding=>1, Mutex=>4, String=>5218,
DBI::DatabaseHandle=>9, DBI::DBD::stuck_out_tongue: g::PgCoerce=>9,
DBI::DBD::Pg::Tuples=>49, TCPSocket=>7, ODS=>1, DBI::DBD::Pg::Dr iver=>1,
NoncritError=>3, RR=>6, Thread=>23, DBI::DBD::Pg::Statement=>49, DBI::S
QL::PreparedStatement=>49, User=>4, ConditionVariable=>2, ThreadGroup=>1,
DBI::smiley: BD::Pg::Database=>9, Event=>22, Proc=>16, File=>1, Hash=>253,
Range=>11, PGconn=

9, PGresult=>49, CritError=>2, Errno::ECONNABORTED=>1, Object=>4,
Bignum=>5, IO
Error=>5, Whois=>1, TCPServer=>2, DBI::DriverHandle=>1, NoMemoryError=>1,
Module =>34, Array=>1922, Sql=>5, MatchData=>150, Class=>232, Regexp=>172}

At it’s peak it reaches about 20k. I’m guessing the drop occurs when the
garbage collector steps in. However, the memory size of the process
doesn’t seem to drop at that point… I’m running FreeBSD 4.9 with ruby
1.8.1 (2003-10-31). The problem was also occuring with release and
stable builds of 1.8.0 though as well. It was not, however, occurring in
1.6.x.

Any ideas? Bug?

Thanks!
-JD-

8

Yep, that's exactly what I'm referring to. If you undef your local copy
of nsEntryId, it still maintains a pointer to itself, so it becomes a
closed off lump of storage. I don't know much about ruby's memory
management, but it apparently can't catch these problems (I know perl
can't).

I expect that if you fix that self-reference, your growth will go away.

Good luck!

···

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

Luke,
  After a while of debugging I found a bunch of objects that were being
created that apparently contain one of the primary keys in one of my
database tables. The thing is is the query should only return one result.
This particular row is hardly ever referenced either. So now I have found
a line in my code that is something like you might have been referring to:

      nsEntryId = nsEntryId[0][0]

  That is called quite often, would that cause the object to stay around?
Is this what you were referring to?

--
I went to a restaurant that serves "breakfast at anytime". So I
ordered French Toast during the Renaissance. -- Stephen Wright

9

I imagine changing it to:

nsEntryId = nsEntryId[0][0].dup would take care of the problem? Or just using another object name?

Regards,
-JD-

···

--On Monday, November 17, 2003 10:50 AM -0600 "Luke A. Kanies" <luke@madstop.com> wrote:

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

Luke,
  After a while of debugging I found a bunch of objects that were being
created that apparently contain one of the primary keys in one of my
database tables. The thing is is the query should only return one
result. This particular row is hardly ever referenced either. So now I
have found a line in my code that is something like you might have been
referring to:

      nsEntryId = nsEntryId[0][0]

  That is called quite often, would that cause the object to stay around?
Is this what you were referring to?

Yep, that's exactly what I'm referring to. If you undef your local copy
of nsEntryId, it still maintains a pointer to itself, so it becomes a
closed off lump of storage. I don't know much about ruby's memory
management, but it apparently can't catch these problems (I know perl
can't).

I expect that if you fix that self-reference, your growth will go away.

Good luck!

--
I went to a restaurant that serves "breakfast at anytime". So I
ordered French Toast during the Renaissance. -- Stephen Wright

10

“Luke A. Kanies” luke@madstop.com schrieb im Newsbeitrag
news:Pine.GSO.4.51.0311171048090.2656@pixie…

Luke,
After a while of debugging I found a bunch of objects that were
being
created that apparently contain one of the primary keys in one of my
database tables. The thing is is the query should only return one
result.
This particular row is hardly ever referenced either. So now I have
found
a line in my code that is something like you might have been referring
to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to stay
around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

Is this what you were referring to?

Yep, that’s exactly what I’m referring to. If you undef your local copy
of nsEntryId, it still maintains a pointer to itself, so it becomes a
closed off lump of storage. I don’t know much about ruby’s memory
management, but it apparently can’t catch these problems (I know perl
can’t).

Hmmm, since Ruby used mark and sweep GC it should be able to catch loops -
self references as well as loops made of several instances. It seems to
me that this is unlikely to be the reason for the memory growth. I’d
rather assume some references as indicated above.

Kind regards

robert
···

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

11

Um, I don’t really know. I can’t think of a generally good reason to have
an object refer to itself. Does the object actually need a
self-reference, or are you just reusing the name for convenience? If it’s
just convenience, I definitely recommend using a different name.

Duplication might also solve the problem, but I’m not really sure. I’m
still a relative ruby newbie and have not figured out all the details of
when you get a reference vs. a copy vs. a real variable. Others can
hopefully chime in with those answers.

Luke

···

On Tue, 18 Nov 2003, Jason DiCioccio wrote:

I imagine changing it to:

nsEntryId = nsEntryId[0][0].dup would take care of the problem? Or just
using another object name?

–
: We are looking at a newspaper clipping labelled “Huffington
: Herald 11/12/96” in which “Huffington” states, in part:
: “The inhabitants of Tiera del Fuego … have a single word
: that means ‘to look at each other hoping that either will offer to do
: something that both parties desire but are unwilling to do.’”
: Does anyone know that word?
Management?

12

Greetings,

···

–On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme bob.news@gmx.net wrote:

“Luke A. Kanies” luke@madstop.com schrieb im Newsbeitrag
news:Pine.GSO.4.51.0311171048090.2656@pixie…

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

Luke,
After a while of debugging I found a bunch of objects that were
being
created that apparently contain one of the primary keys in one of my
database tables. The thing is is the query should only return one
result.
This particular row is hardly ever referenced either. So now I have
found
a line in my code that is something like you might have been referring
to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to stay
around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

I’ve tried removing this line and it does indeed appear to be the problem.
There was no reference from nsEntryId[0][0] back to nsEntryId. Is this a
bug? I don’t think it had this problem in ruby 1.6.x, so it appears
something changed in the handling of arrays between 1.6 and 1.8.

Regards,
-JD-

13

“Jason DiCioccio” jd@ods.org schrieb im Newsbeitrag
news:2147483647.1069109940@[10.0.2.2]…

Greetings,

“Luke A. Kanies” luke@madstop.com schrieb im Newsbeitrag
news:Pine.GSO.4.51.0311171048090.2656@pixie…

Luke,
After a while of debugging I found a bunch of objects that were
being
created that apparently contain one of the primary keys in one of
my
database tables. The thing is is the query should only return one
result.
This particular row is hardly ever referenced either. So now I
have
found
a line in my code that is something like you might have been
referring
to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to stay
around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

I’ve tried removing this line and it does indeed appear to be the
problem.
There was no reference from nsEntryId[0][0] back to nsEntryId. Is this
a
bug? I don’t think it had this problem in ruby 1.6.x, so it appears
something changed in the handling of arrays between 1.6 and 1.8.

Hm. Since I don’t know the data at hand, I’m not in a position to comment
more specific. I do wonder however, how this assignment should be
responsible for growth of mem consumption if nsEntryId[0][0] does not
backreference nsEntryId.

I’d do a to_yaml of nsEntryId before and after the assignment to be able
to analyse the object graph. Maybe there’s a backreference you
overlooked.

The other option seems to be a bug in the garbage collector but I’m not
very inclined in believing this.

If there is an extension involved that could hold on to the memory in ways
not easy to determine. That’s another option that comes to my mind.

Kind regards

robert
···

–On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme > bob.news@gmx.net wrote:

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

14

I've tried removing this line and it does indeed appear to be the problem.
There was no reference from nsEntryId[0][0] back to nsEntryId. Is this a
bug? I don't think it had this problem in ruby 1.6.x, so it appears
something changed in the handling of arrays between 1.6 and 1.8.

Well, if you can, try to reproduce the problem with a *small* script and
post it if you still have the problem

Guy Decoux

15

Robert,
The object in question is just data from ruby dbi object. You could be
right about the back reference, however, the back reference was definitely
not something that I am aware is there. It could be a pointer that ruby
DBI puts into all of its objects?

Here’s a code snippet to illustrate what it contains:

nsEntryId, = Thread.current[:sql].query( “SELECT NSEntryId FROM NSEntry
WHERE LOWER(Host) = LOWER(?) AND DomainId=?::bigint AND OwnerId=?::bigint
AND UserId=?::bigint #{type} #{ target if target }”, domain[0],
Domain::id(domain[1]), Domain::ownerId(domain[1]), @aUser.id )

The Sql#query method is below as well:

def query( sqlRequest, *vars )
@conn.prepare( sqlRequest ) do |sqlVar|
begin
sqlVar.execute( *vars )
rescue => detail
if detail.message =~ /server closed the connection unexpectedly/
log(“Connection to DB unexpectedly closed”)
$sqlPool.reopen!( @conn )
end

    Thread.current.raise NoncritError, "An error occurred: #{detail},

"#{sqlRequest}""
end

  case sqlRequest.word( 0 )
    when /^INSERT/ then return
    when /^UPDATE/ then return
    when /^DELETE/ then return
  end

  result = []
  rcnt = 0
  while (row = sqlVar.fetch) != nil
    result.push row.dup
    rcnt = rcnt.next
  end

  return [nil, rcnt] if result.length == 0
  return [result, rcnt]
end

end

Any ideas where this reference might be coming from if it exists?

Regards,
-JD-

···

–On Tuesday, November 18, 2003 6:17 PM +0900 Robert Klemme bob.news@gmx.net wrote:

“Jason DiCioccio” jd@ods.org schrieb im Newsbeitrag
news:2147483647.1069109940@[10.0.2.2]…

Greetings,

–On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme >> bob.news@gmx.net wrote:

“Luke A. Kanies” luke@madstop.com schrieb im Newsbeitrag
news:Pine.GSO.4.51.0311171048090.2656@pixie…

On Mon, 17 Nov 2003, Jason DiCioccio wrote:

Luke,
After a while of debugging I found a bunch of objects that were
being
created that apparently contain one of the primary keys in one of
my
database tables. The thing is is the query should only return one
result.
This particular row is hardly ever referenced either. So now I
have
found
a line in my code that is something like you might have been
referring
to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to stay
around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

I’ve tried removing this line and it does indeed appear to be the
problem.
There was no reference from nsEntryId[0][0] back to nsEntryId. Is this
a
bug? I don’t think it had this problem in ruby 1.6.x, so it appears
something changed in the handling of arrays between 1.6 and 1.8.

Hm. Since I don’t know the data at hand, I’m not in a position to comment
more specific. I do wonder however, how this assignment should be
responsible for growth of mem consumption if nsEntryId[0][0] does not
backreference nsEntryId.

I’d do a to_yaml of nsEntryId before and after the assignment to be able
to analyse the object graph. Maybe there’s a backreference you
overlooked.

The other option seems to be a bug in the garbage collector but I’m not
very inclined in believing this.

If there is an extension involved that could hold on to the memory in ways
not easy to determine. That’s another option that comes to my mind.

Kind regards

robert
16

“Jason DiCioccio” jd@ods.org schrieb im Newsbeitrag
news:2147483647.1069146662@pcp04900144pcs.anapol01.md.comcast.net…

“Jason DiCioccio” jd@ods.org schrieb im Newsbeitrag
news:2147483647.1069109940@[10.0.2.2]…

Greetings,

“Luke A. Kanies” luke@madstop.com schrieb im Newsbeitrag
news:Pine.GSO.4.51.0311171048090.2656@pixie…

Luke,
After a while of debugging I found a bunch of objects that
were
being
created that apparently contain one of the primary keys in one
of
my
database tables. The thing is is the query should only return
one
result.
This particular row is hardly ever referenced either. So now I
have
found
a line in my code that is something like you might have been
referring
to:

  nsEntryId = nsEntryId[0][0]

That is called quite often, would that cause the object to
stay
around?

Not the old nsEntryId unless nsEntryId[0][0] has a reference to it.

I’ve tried removing this line and it does indeed appear to be the
problem.
There was no reference from nsEntryId[0][0] back to nsEntryId. Is
this
a
bug? I don’t think it had this problem in ruby 1.6.x, so it appears
something changed in the handling of arrays between 1.6 and 1.8.

Hm. Since I don’t know the data at hand, I’m not in a position to
comment
more specific. I do wonder however, how this assignment should be
responsible for growth of mem consumption if nsEntryId[0][0] does not
backreference nsEntryId.

I’d do a to_yaml of nsEntryId before and after the assignment to be
able
to analyse the object graph. Maybe there’s a backreference you
overlooked.

The other option seems to be a bug in the garbage collector but I’m
not
very inclined in believing this.

If there is an extension involved that could hold on to the memory in
ways
not easy to determine. That’s another option that comes to my mind.

Kind regards

robert

Robert,
The object in question is just data from ruby dbi object. You could
be
right about the back reference, however, the back reference was
definitely
not something that I am aware is there. It could be a pointer that ruby
DBI puts into all of its objects?

Here’s a code snippet to illustrate what it contains:

nsEntryId, = Thread.current[:sql].query( “SELECT NSEntryId FROM NSEntry
WHERE LOWER(Host) = LOWER(?) AND DomainId=?::bigint AND
OwnerId=?::bigint
AND UserId=?::bigint #{type} #{ target if target }”, domain[0],
Domain::id(domain[1]), Domain::ownerId(domain[1]), @aUser.id )

The Sql#query method is below as well:

def query( sqlRequest, *vars )
@conn.prepare( sqlRequest ) do |sqlVar|
begin
sqlVar.execute( *vars )
rescue => detail
if detail.message =~ /server closed the connection unexpectedly/
log(“Connection to DB unexpectedly closed”)
$sqlPool.reopen!( @conn )
end

    Thread.current.raise NoncritError, "An error occurred:

#{detail},

"#{sqlRequest}""
end

  case sqlRequest.word( 0 )
    when /^INSERT/ then return
    when /^UPDATE/ then return
    when /^DELETE/ then return
  end

  result = []
  rcnt = 0
  while (row = sqlVar.fetch) != nil
    result.push row.dup
    rcnt = rcnt.next
  end

  return [nil, rcnt] if result.length == 0
  return [result, rcnt]
end

end

Any ideas where this reference might be coming from if it exists?

Only the row.dup. What type does it return? Is it an Array or some DBI
internal class? If it’s DBI internal, I’d try row.to_a instead to make
sure, that only values are retained and not the row collection.

Cheers

robert
···

–On Tuesday, November 18, 2003 6:17 PM +0900 Robert Klemme > bob.news@gmx.net wrote:

–On Tuesday, November 18, 2003 2:12 AM +0900 Robert Klemme > >> bob.news@gmx.net wrote:

On Mon, 17 Nov 2003, Jason DiCioccio wrote: