Memory leak

ruby-talk
1

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

Thanks,
Joe

2

Joe Van Dyk wrote:

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

That's a large leak. My approach is to first step back and walk through the
code on paper, finding all of the areas where I _might_ be creating
references that don't get cleaned up. Anything that I think _might_ be a
problem I then try to isolate and test individually. I'll put diagnostic
logging in to check object counts or specific contents of hashes or LRU
caches or whatever else I might suspect to be a problem.

In bad cases, you can use ObjectSpace to dump every object known to the the
Ruby interpreter at some specific point in your code, and then skim through
that output to see if you can spot references that are accumulating which
should not be.

File.open('/tmp/objects.out','a+') do |fh|
  fh.puts '________'
  ObjectSpace.each_object do |obj|
    fh.puts "#{obj.class.name} ::: #{obj.inspect}"
  end
end

If you are dealing with a C extension, walk through the code. Look for
places where memory is allocated but never freed. When one is used to
programming in a language with garbage collection, it is very easy to
allocate memory in C without making sure that it'll be freed when no longer
needed.

Kirk Haines

3

I'd probably start with the profiler.

This will at least identify the hot spots in your code. And you have
to start somewhere right?

If you suspect it's in a lib you depend on, I would write small
scripts that use the only the lib you suspect, and see if similair
behavior appears.

Basically what I try to do is eliminate the knowns, and make the
unknowns become knowns.

···

On Wed, 23 Mar 2005 10:13:22 +0900, Joe Van Dyk <joevandyk@gmail.com> wrote:

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

Thanks,
Joe

--
Thomas G. Willis
http://paperbackmusic.net

4

* Joe Van Dyk (Mar 23, 2005 02:20):

Are there any tools or strategies that I can use to track down the
leak?

http://valgrind.org/,
        nikolai

···

--
::: name: Nikolai Weibull :: aliases: pcp / lone-star / aka :::
::: born: Chicago, IL USA :: loc atm: Gothenburg, Sweden :::
::: page: minimalistic.org :: fun atm: gf,lps,ruby,lisp,war3 :::
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

5

Joe Van Dyk wrote:

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

Thanks,
Joe

It's out of date now, but I used to patch ruby to tell me why a certain object was not collectable. It's down near the end of:

This might be worth updating if you know what objects are not being collected, but can't tell why.

6

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

This might be kind of whacky, but maybe:

set_trace_func proc {|event, file, line, id, binding, classname|
  printf "%8s %s:%-2d %10s %8s\n", event, file, line, id, classname
  print `ps -u #$$`
}

This (on Linux) should print a trace of every line of
your program being executed, followed by some memory usage
information for the process. ... And slow down your app by
3 or 4 orders of magnitude... :wink:

Well, just a thought... Like, you could log the output to
a text file, then go back and see where the memory was increasing,
and see if it correlates to any particular part of the code?

Regards,

Bill

···

From: "Joe Van Dyk" <joevandyk@gmail.com>

7

Joe Van Dyk wrote:

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

250KB is pretty large... are you even *creating*
that many objects per second? If yes, review the
lifetime of those objects.

If not, is this a new problem? If so, did you just
start using a new library?

Using trace is probably helpful, too.

Thanks,
Joe

E

8

Hi ..

Hi,

My application's size in memory seems to be increasing by about 250KB
a second. This is not a good thing.

Are there any tools or strategies that I can use to track down the leak?

Can you give a bit more of an idea of the platform and the type of
application? I have had issues in the past with threaded net::telnet based
applications if I didn't do the creation / cleanup just right.

Valgrind didn't help. The only thing that did was checking the inner classes
one at a time and rewriting the driver code a number of different ways. I am
still not sure why the leak "went away" ... :frowning:

Regards,

···

On Tuesday 22 March 2005 17:13, Joe Van Dyk wrote:

--
-mark. (probertm at acm dot org)

9

Nikolai Weibull wrote:

* Joe Van Dyk (Mar 23, 2005 02:20):

Are there any tools or strategies that I can use to track down the
leak?

http://valgrind.org/,
        nikolai

But that won't help if the objects are reachable from the root references in the interpreter. In other words, it's not a true memory leak, but you've forgotten to clear out a reference somewhere.

10

Bill Kelly wrote:

This might be kind of whacky, but maybe:

set_trace_func proc {|event, file, line, id, binding, classname|
  printf "%8s %s:%-2d %10s %8s\n", event, file, line, id, classname
  print `ps -u #$$`
}

For hard to find leaks, that's helpful. Also (and it'd slow the !#$%@!!!
out of your program if you do it a lot):

count = 0; ObjectSpace.each_object {count += 1}

Gives you a count of the current number of objects. If you do this in key
places (probably don't want to do it out of set_trace_func), it might help.

Kirk Haines

11

Hello Nikolai,

* Joe Van Dyk (Mar 23, 2005 02:20):

Are there any tools or strategies that I can use to track down the
leak?

http://valgrind.org/,
        nikolai

This will not help for ruby, unless he is using a C extension that
leaks memory. And even for C it's difficult with valgrind. Or do they
now keep a log where the allocating source position of each memory block is
tracked ?

···

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

12

How can i clear a reference?

I ran into trouble before when I had a bunch of objects that I was
observing. When I overwrote those objects, the old objects stayed
around (because the other objects were still observing them). I think
a call to delete_observers fixed my problem. And I think something
like that might be the cause of my problem.

···

On Wed, 23 Mar 2005 10:44:04 +0900, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:

Nikolai Weibull wrote:
> * Joe Van Dyk (Mar 23, 2005 02:20):
>
>>Are there any tools or strategies that I can use to track down the
>>leak?
>
>
> http://valgrind.org/,
> nikolai
>

But that won't help if the objects are reachable from the root
references in the interpreter. In other words, it's not a true memory
leak, but you've forgotten to clear out a reference somewhere.

13

Maybe rather just

count = ObjectSpace.each_object {}

?

Csaba

···

On 2005-03-23, Kirk Haines <wyhaines@gmail.com> wrote:

For hard to find leaks, that's helpful. Also (and it'd slow the !#$%@!!!
out of your program if you do it a lot):

count = 0; ObjectSpace.each_object {count += 1}

Gives you a count of the current number of objects. If you do this in key
places (probably don't want to do it out of set_trace_func), it might help.

14

"Csaba Henk" <csaba@phony_for_avoiding_spam.org> schrieb im Newsbeitrag
news:slrnd41nbo.hh5.csaba@ms0.math.ucalgary.ca...

> For hard to find leaks, that's helpful. Also (and it'd slow the

!#$%@!!!

> out of your program if you do it a lot):
>
> count = 0; ObjectSpace.each_object {count += 1}
>
> Gives you a count of the current number of objects. If you do this in

key

> places (probably don't want to do it out of set_trace_func), it might

help.

Maybe rather just

count = ObjectSpace.each_object {}

Or a bit more complex

def ostats(last_stat = nil)
  stats = Hash.new(0)
  ObjectSpace.each_object {|o| stats[o.class] += 1}

  stats.sort {|(k1,v1),(k2,v2)| v2 <=> v1}.each do |k,v|
    printf "%-30s %10d", k, v
    printf " delta %10d", (v - last_stat[k]) if last_stat
    puts
  end

  stats
end

Then you can place lines like this in your program

stats = nil
....
stats = ostats stats
....
stats = ostats stats

To look at changes between lines. Of course this method does create some
instances, too - but no instances of your application classes. You can as
well add filters (by class or by count) etc.

Kind regards

    robert

···

On 2005-03-23, Kirk Haines <wyhaines@gmail.com> wrote: