Garbage collection/memory unfreed problem

It appears that the ruby garbage collector is not freeing up unused
memory on my linux machine, but IS on my windows machine... I'm baffled
and troubled. Any advice would be appreciated...

Using the attached test script, in linux I observe the VMRSS start at
around 1.5Meg, then go up to ~20Meg by iteration 3 or 4, and sit there
for the remaining iterations. Then once the t array goes out of scope, I
see it drop by a few meg. The GC.start only drops apparent memory
consumption to ~14Meg.

Meanwhile, on windows XP, I see it increase to around 20Meg, then sit
around there until the GC.free, which brings it back down to around
5Meg, close to the starting point.

Can anybody explain the difference here? Is the 14meg _really_
unavailable to the OS in linux?

···

-------------
class Grab
  def initialize
    @myhash={}
    0.upto(100000) {|i| @myhash[i]="#{i} Im a part of a hash"}
  end
end

def memuse
  #IO.read("/proc/#{Process.pid}/status") =~ /VmSize.* (\d+) k/
  "Hi"
end

puts "Mem before anything: #{memuse}"
sleep 10;
0.upto(10) do |x|
  t>>=Array.new
  t[x]=Grab.new
puts "Mem in scope #{x}: #{memuse}"
sleep 1;
end
puts "mem out of scope #{memuse}"
sleep 10;
GC.start
puts "Mem after GC: #{memuse}"
sleep 10;

Hi Mark,

I tried your code on a RHEL4 x86-64 server and I got the same problem
(the RSS size was even bigger). I also tried repeating your main loop
for several times, and the RSS size (after GC) increased each time. So
I think this is inefficiency in Ruby's GC implementation.

Since it's RSS, physical pages have been allocated for the Ruby
process's data section, I don't think Linux can use this part of memory
for other purpose, unless the Ruby process is killed by the OOM killer
:frowning:

I don't know the reason for this problem but I'll try to find out.

Lei Ming

Mark Noworolski wrote:

···

It appears that the ruby garbage collector is not freeing up unused
memory on my linux machine, but IS on my windows machine... I'm baffled
and troubled. Any advice would be appreciated...

Using the attached test script, in linux I observe the VMRSS start at
around 1.5Meg, then go up to ~20Meg by iteration 3 or 4, and sit there
for the remaining iterations. Then once the t array goes out of scope, I
see it drop by a few meg. The GC.start only drops apparent memory
consumption to ~14Meg.

Meanwhile, on windows XP, I see it increase to around 20Meg, then sit
around there until the GC.free, which brings it back down to around
5Meg, close to the starting point.

Can anybody explain the difference here? Is the 14meg _really_
unavailable to the OS in linux?

-------------
class Grab
  def initialize
    @myhash={}
    0.upto(100000) {|i| @myhash[i]="#{i} Im a part of a hash"}
  end
end

def memuse
  #IO.read("/proc/#{Process.pid}/status") =~ /VmSize.* (\d+) k/
  "Hi"
end

puts "Mem before anything: #{memuse}"
sleep 10;
0.upto(10) do |x|
  t>>=Array.new
  t=Grab.new
puts "Mem in scope #{x}: #{memuse}"
sleep 1;
end
puts "mem out of scope #{memuse}"
sleep 10;
GC.start
puts "Mem after GC: #{memuse}"
sleep 10;

It appears that the ruby garbage collector is not freeing up unused
memory on my linux machine, but IS on my windows machine... I'm baffled
and troubled. Any advice would be appreciated...

Using the attached test script, in linux I observe the VMRSS start at
around 1.5Meg, then go up to ~20Meg by iteration 3 or 4, and sit there
for the remaining iterations. Then once the t array goes out of scope, I
see it drop by a few meg. The GC.start only drops apparent memory
consumption to ~14Meg.

Meanwhile, on windows XP, I see it increase to around 20Meg, then sit
around there until the GC.free, which brings it back down to around
5Meg, close to the starting point.

Can anybody explain the difference here? Is the 14meg _really_
unavailable to the OS in linux?

Ruby's garbage collector is conservative and walks the stack looking for pointers into the heap. If it finds a possible pointer to the heap the referenced memory will not be reclaimed. This can happen even if you've returned from a method that referenced a large object.

···

On Nov 2, 2006, at 9:55 PM, Mark Noworolski wrote:

-------------
class Grab
  def initialize
    @myhash={}
    0.upto(100000) {|i| @myhash[i]="#{i} Im a part of a hash"}
  end
end

def memuse
  #IO.read("/proc/#{Process.pid}/status") =~ /VmSize.* (\d+) k/
  "Hi"
end

puts "Mem before anything: #{memuse}"
sleep 10;
0.upto(10) do |x|
  t>>=Array.new
  t=Grab.new
puts "Mem in scope #{x}: #{memuse}"
sleep 1;
end
puts "mem out of scope #{memuse}"
sleep 10;
GC.start
puts "Mem after GC: #{memuse}"
sleep 10;

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com