WeakRef -- excessive memory usage?

ruby-talk
1

Hi everyone, I recently had a problem with memory usage in my ruby 1.8.6 program. It had ballooned from about 65mb after startup to 300mb. I tracked the problem down to my usage of WeakRefs. Storing object_ids instead got the memory usage back to where it was.

In the following example I'm checking memory usage with Windows Task Manager.

> irb

irb(main):001:0> require 'weakref'
=> true
irb(main):002:0> s = 'hello'
=> "hello"
irb(main):003:0> a = []
=> []

(memory usage is 4,171k at this point)

irb(main):004:0> 100.times { a << WeakRef.new(s) }
=> 100

(memory usage is now 54,380k)

Repeated calls to GC.start doesn't reduce the memory usage at this point. Am I missing something?

David

2

puts 'weakref'r = require 'weakref'puts rs = 'hello'a = GC.disable #<--------5.times { gets puts "100times" 100.times { WeakRef.new(s) } puts "end times"
}
In fact, GC works.
GC is enabled by default,
You may not need to use GC.start to start GC, it seems that ruby deals with GC well.
Without GC.disable, the memory usage is about 10,500K every iteration.
But when you disable GC, the memory usage is as below:
before
5,200k
after
1 77,764k2 145,976k
3 231,088k
4 277,376k

I think ruby actually get these space back,
but it keeps these space somewhere,
so it will be fast next time you make some new things without getting it from OS.

···

Date: Sat, 12 Apr 2008 23:05:05 +0900> From: "david mail"@ruby-lang.org> Subject: WeakRef -- excessive memory usage?> To: ruby-talk@ruby-lang.org> > Hi everyone, I recently had a problem with memory usage in my ruby 1.8.6 > program. It had ballooned from about 65mb after startup to 300mb. I > tracked the problem down to my usage of WeakRefs. Storing object_ids > instead got the memory usage back to where it was.> > In the following example I'm checking memory usage with Windows Task > Manager.> > > > irb> > irb(main):001:0> require 'weakref'> => true> irb(main):002:0> s = 'hello'> => "hello"> irb(main):003:0> a = > => > > (memory usage is 4,171k at this point)> > irb(main):004:0> 100.times { a << WeakRef.new(s) }> => 100> > (memory usage is now 54,380k)> > > Repeated calls to GC.start doesn't reduce the memory usage at this > point. Am I missing something?> > David>

_________________________________________________________________
用手机MSN聊天写邮件看空间，无限沟通，分享精彩！
http://mobile.msn.com.cn/

3

Thanks for your reply Derek. I can definitely see that the intepreter is
freeing memory over the course of the run.

In my original example, I made sure to add WeakRefs to an array to
ensure that they wouldn't get garbage collected. The reason I still
think WeakRef is using an unusual amount of memory is that when I assign
that array reference to nil on the completion of the run and call
GC.start, memory usage goes down from around 150mb to 20mb (for example).

So, I can see that ruby hasn't completely shrunk its heap back down to
the original size (as you say, it must reserve the memory as per its
memory management algorithm). But this also means that the WeakRef
objects were using a large amount of the heap, since the majority of the
heap was released once the references to the WeakRefs were released.

What it is about WeakRef's implementation that might cause this?

David

YuDerek wrote:

···

[Note: parts of this message were removed to make it a legal post.]

puts 'weakref'r = require 'weakref'puts rs = 'hello'a = GC.disable #<--------5.times { gets puts "100times" 100.times { WeakRef.new(s) } puts "end times"
}
In fact, GC works.
GC is enabled by default,
You may not need to use GC.start to start GC, it seems that ruby deals with GC well.
Without GC.disable, the memory usage is about 10,500K every iteration.
But when you disable GC, the memory usage is as below:
before
5,200k
after
1 77,764k2 145,976k
3 231,088k
4 277,376k

I think ruby actually get these space back,
but it keeps these space somewhere,
so it will be fast next time you make some new things without getting it from OS.

Date: Sat, 12 Apr 2008 23:05:05 +0900> From: "david mail"@ruby-lang.org> Subject: WeakRef -- excessive memory usage?> To: ruby-talk@ruby-lang.org> > Hi everyone, I recently had a problem with memory usage in my ruby 1.8.6 > program. It had ballooned from about 65mb after startup to 300mb. I > tracked the problem down to my usage of WeakRefs. Storing object_ids > instead got the memory usage back to where it was.> > In the following example I'm checking memory usage with Windows Task > Manager.> > > > irb> > irb(main):001:0> require 'weakref'> => true> irb(main):002:0> s = 'hello'> => "hello"> irb(main):003:0> a = > => > > (memory usage is 4,171k at this point)> > irb(main):004:0> 100.times { a << WeakRef.new(s) }> => 100> > (memory usage is now 54,380k)> > > Repeated calls to GC.start doesn't reduce the memory usage at this > point. Am I missing something?> > David>

_________________________________________________________________
用手机MSN聊天写邮件看空间，无限沟通，分享精彩！
http://mobile.msn.com.cn/

4

2008/4/15 mail"@ruby-lang.org David Beswick <"david>:

Thanks for your reply Derek. I can definitely see that the intepreter is
freeing memory over the course of the run.

In my original example, I made sure to add WeakRefs to an array to
ensure that they wouldn't get garbage collected. The reason I still
think WeakRef is using an unusual amount of memory is that when I assign
that array reference to nil on the completion of the run and call
GC.start, memory usage goes down from around 150mb to 20mb (for example).

So, I can see that ruby hasn't completely shrunk its heap back down to
the original size (as you say, it must reserve the memory as per its
memory management algorithm). But this also means that the WeakRef
objects were using a large amount of the heap, since the majority of the
heap was released once the references to the WeakRefs were released.

What it is about WeakRef's implementation that might cause this?

A few observations:

1) The job of the GC is to preserve any reachable objects while
reclaiming the space taken up by non-reachable objects. The first
part of this is a bit more important than the second, since freeing a
referenced object can lead to mysterious errors when a reference is
followed.

2) Irb is not a good tool for doing experiments with GC. The reason is
that it has references itself which can cause objects to hang around
which you might not expect.

3) WeakRefs themselves take up space. In the 1.8 implementation each
WeakRef is a ruby object which will have a class pointer, and an
instance variable hash to hold it's one instance var which hangs on to
the object id of the object it's referenceing. Then there are two
class variables which point to hashes. One hash has an entry for each
WeakRef object and maps it to the id of the object it references, the
other hash maps the object id of each object referenced by a WeakRef
to an array of WeakRefs referring to that object. The entries in
those hashes are set when the WeakRef is created, and manipulated by a
finalizer, which deletes the appropriate entries in each of the hashes
when a weakly referenced object is GCed. I haven't added it all up
but I suspect that each WeakRef takes more storage than your string
"Hello"

···

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

5

Thanks for your help Rick.

I didn't realise irb was a bad tool to test memory usage. I've written a standard ruby program. Comments about the memory usage I observed at each step are included. This program indicates to me that the WeakRef implementation does require more memory than I'd assumed they would.

require 'WeakRef'

s = 'Hello'
a =

GC.start

puts 'Start'
gets
   # Memory Usage: 2,652K

100.times do
   a << WeakRef.new(s)
end

GC.start

puts '100 WeakRef instances in array'
gets
   # Memory Usage: 53,360K

GC.start

puts 'Additional GC (ensure temporary data incurred by WeakRef instances is released)'
gets
   # Memory Usage: 53,360K

a = nil
GC.start

puts 'Array GCed'
gets
   # Memory Usage: 19,128K

GC.start

puts 'Additional GC (ensure release of WeakRefs)'
gets
   # Memory Usage: 18,996K

GC.start

puts '2nd Additional GC (ensure release of WeakRefs)'
gets
   # Memory Usage: 18,996K

Here's all I can think of that might be consuming memory in the WeakRef implementation:

WeakRef
1. Standard ruby object instance overhead
2. "Int" instance var (object_id of target)
3. Hash entry (WeakRef -> object_id of target)
4. Hash entry + array entry (object_id of target -> array of weakrefs)

It doesn't look like a look of data. But WeakRef also inherits from Delegator, maybe it's this class that's pushing up memory usage?

As a quick test I modified weakref.rb so that it no longer inherited from "Delegator," and I ran my test program again. The memory usage hardly moved from the value taken at startup. So I think "Delegator" is the culprit.

I can only guess at what might be causing it for now, but I've got a feeling it's the "eval" block in Delegator::initalize, which looks like it makes a new class method for each method in "obj" not belonging to an ancestor. Still, even if this is executed every time WeakRef is instantiated, I'd expect the data associated with the existing methods to be discarded every time the eval block executed again. Unless, maybe a new metaclass is created each time the methods are 'overridden' by the instantiation of the new WeakRef instance?

There's my ideas!

David

Rick DeNatale wrote:

···

2008/4/15 mail"@ruby-lang.org David Beswick <"david>:

Thanks for your reply Derek. I can definitely see that the intepreter is
freeing memory over the course of the run.

In my original example, I made sure to add WeakRefs to an array to
ensure that they wouldn't get garbage collected. The reason I still
think WeakRef is using an unusual amount of memory is that when I assign
that array reference to nil on the completion of the run and call
GC.start, memory usage goes down from around 150mb to 20mb (for example).

So, I can see that ruby hasn't completely shrunk its heap back down to
the original size (as you say, it must reserve the memory as per its
memory management algorithm). But this also means that the WeakRef
objects were using a large amount of the heap, since the majority of the
heap was released once the references to the WeakRefs were released.

What it is about WeakRef's implementation that might cause this?

A few observations:

1) The job of the GC is to preserve any reachable objects while
reclaiming the space taken up by non-reachable objects. The first
part of this is a bit more important than the second, since freeing a
referenced object can lead to mysterious errors when a reference is
followed.

2) Irb is not a good tool for doing experiments with GC. The reason is
that it has references itself which can cause objects to hang around
which you might not expect.

3) WeakRefs themselves take up space. In the 1.8 implementation each
WeakRef is a ruby object which will have a class pointer, and an
instance variable hash to hold it's one instance var which hangs on to
the object id of the object it's referenceing. Then there are two
class variables which point to hashes. One hash has an entry for each
WeakRef object and maps it to the id of the object it references, the
other hash maps the object id of each object referenced by a WeakRef
to an array of WeakRefs referring to that object. The entries in
those hashes are set when the WeakRef is created, and manipulated by a
finalizer, which deletes the appropriate entries in each of the hashes
when a weakly referenced object is GCed. I haven't added it all up
but I suspect that each WeakRef takes more storage than your string
"Hello"