GC error?

ruby-talk
1

Run this program and observe memory usage.

<code>

def leaker
    a = ""
    10000.times do
         a = a + "buble"
    end
    a = ""
end

loop do
    leaker
end

</code>

···

--
Posted via http://www.ruby-forum.com/.

2

Your point being?

  robert

···

On 16.02.2008 13:12, Piotr Sawicki wrote:

Run this program and observe memory usage.

<code>

def leaker
    a = ""
    10000.times do
         a = a + "buble"
    end
    a = ""
end

loop do
    leaker
end

</code>

3

Piotr Sawicki wrote:

Run this program and observe memory usage.

Slightly modified program, with option to GC each cycle:

def leaker
     a = ""
     10000.times do
          a = a + "buble"
     end
     a = ""
end

gc = ARGV[0] == "gc"

300.times do |i|
     printf "%6d %10s", i, `ps h -o rss #{$$}`
     leaker
     GC.start if gc
end

With ruby-1.8.6p111 (linux), memory usage (RSS) seems to grow without bound. With the extra GC, it varies between a low of about 4Mb and a high point that's on the same curve as without the extra GC.

Grain of salt and all...

···

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

4

Not a bug in any way. It's simply that the garbage collector has not
been trigger. It runs only when it needs to. As you realized you, you
can force it to run by doing 'GC.start'. If you run 'GC.start' at the
end of your program, you'll see the memory reclaimed.

This is a typical behavior for a mark/sweep garbage collector. Perl
and Python are reference count, freeing memory the instant it goes out
of scope. Ruby waits to free memory until it needs memory to free some.

···

On Feb 16, 4:12 am, Piotr Sawicki <piotr.sawi...@gmail.com> wrote:

Run this program and observe memory usage.

<code>

def leaker
a = ""
10000.times do
a = a + "buble"
end
a = ""
end

loop do
leaker
end

</code>
--
Posted viahttp://www.ruby-forum.com/.

5

Robert Klemme wrote:

Your point being?

  robert

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I wrote the same progam on python and perl and I have discover that that
this problem occur only in ruby.

···

--
Posted via http://www.ruby-forum.com/\.

6

Guys, it's worth to send it to ruby-core and ask there. I checked
ruby1.9 and it has the same issue :/.

···

--
Radosław Bułat

http://radarek.jogger.pl - mój blog

7

evanwebb@gmail.com wrote:

Not a bug in any way. It's simply that the garbage collector has not
been trigger. It runs only when it needs to. As you realized you, you
can force it to run by doing 'GC.start'. If you run 'GC.start' at the
end of your program, you'll see the memory reclaimed.

This is a typical behavior for a mark/sweep garbage collector. Perl
and Python are reference count, freeing memory the instant it goes out
of scope. Ruby waits to free memory until it needs memory to free some.

Yes, I agree. I analyzed source code of gc.c. GC was triggered each time
when malloc_increase reach malloc_limit. Variable malloc_limit was
increased by factor proportional to (malloc_increase - malloc_limit). In
each loop cycle malloc_increase > malloc_limit because malloc_increase
is sum of arithmetic series of string length between GC calls.

···

--
Posted via http://www.ruby-forum.com/\.

8

Basically it's true. But notice that this program should use constant
of maxium memory (maybe I'm wrong). When method 'leaker' is finished
all objects created in it are candidates to free (but GC isn't called
yet) because there is no reference to them (in particular binding or
scope reference). When method 'leaker' is called again, Ruby tries
create new objects and instead of increasing heap capacity it should
run GC first and frees all unnecessary objects.

Calling manually GC is not solution because... it isn't work in this
case. Calling GC.start after 'leaker' frees some memory but heap
capacity is constanly increased so it leads to using big amount of
memory. Joel VanderWerf showed it in his graph.

Second, GC.start shouldn't be used in typical case. It could affect
whole program (specially slow down program ;)).

<code>
def leaker
  a = Array.new(1_000_000) {|i| "blah"}
end

loop do
  printf "%10s", `ps h -o rss #{$$}`
  leaker
end
</code>

Why this program doesn't cause memory leaks?

btw, great story about GC from why the lucky stiff:
http://whytheluckystiff.net/articles/theFullyUpturnedBin.html

···

On Feb 19, 2008 8:59 AM, evanwebb@gmail.com <evanwebb@gmail.com> wrote:

Not a bug in any way. It's simply that the garbage collector has not
been trigger. It runs only when it needs to. As you realized you, you
can force it to run by doing 'GC.start'. If you run 'GC.start' at the
end of your program, you'll see the memory reclaimed.

This is a typical behavior for a mark/sweep garbage collector. Perl
and Python are reference count, freeing memory the instant it goes out
of scope. Ruby waits to free memory until it needs memory to free some.

--
Radosław Bułat

http://radarek.jogger.pl - mój blog

9

I ran that code via irb and compared the memory usage to running 'loop {}' in irb.

Your code used more memory but it oscillated around a fixed point that was about 10 megabytes larger than the empty loop process, it didn't grow.

This was on Mac OS X 10.4, ruby 1.8.6.

···

On Feb 16, 2008, at 12:39 PM, Piotr Sawicki wrote:

Robert Klemme wrote:

Your point being?

  robert

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

10

Well, concluding from the postings not all versions of Ruby on all
platforms suffer this phenomenon. As far as I can see so far you have
proven that

(1) on cygwin on Windows XP and on Linux
(2) Ruby version 1.8.6
(3) will allocate and free memory in a way that
(4) the OS shows continuous increasing size of used memory
(5) for a particular program, namely one that allocates memory at a high rate.

Whether that's a bug or not is probably really a question for ruby
core. There might be multiple causes for this observed behavior
including but not limited to Ruby's GC.

Kind regards

robert

···

2008/2/16, Piotr Sawicki <piotr.sawicki@gmail.com>:

Robert Klemme wrote:
> Your point being?

I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
Why memory used by ruby process grow?
Is that normal behaviour ?

I wrote the same progam on python and perl and I have discover that that
this problem occur only in ruby.

--
use.inject do |as, often| as.you_can - without end

11

Radosław Bułat wrote:

<code>
def leaker
  a = Array.new(1_000_000) {|i| "blah"}
end

loop do
  printf "%10s", `ps h -o rss #{$$}`
  leaker
end
</code>

Why this program doesn't cause memory leaks?

Beacuse in this program

<code>
a = ""
10000.times do
   a = a + "buble"
end
</code>

in each cycle ruby creates a new string with length:
<No. of cycle> * "buble".length + 1 ('\0')

No. of cycle Heap
------------ ----------
1) a = "buble"
2) a = "bublebuble" , a' = "buble"
3) a = "bublebublebuble", a' = "bublebuble", a'' = "buble"
etc.

so after n-cycles we have
    S = n * (("buble".length + 1) + (("buble".length * n + 1)) / 2
in my program:
    n = 10000
    "buble".size = 5
    S = 10000 * (5 + 1 + 50000 + 1)/2 = 5000 * 50007 = 250_035_000 bytes

In your program the strings occupy only
  1_000_000 * ("blah".length + 1) = 6_000_000 bytes

···

--
Posted via http://www.ruby-forum.com/\.

12

I have simple, but ugly solution for this problem.

class String
    alias plus +
    def +(a)
        res = self.plus(a)
        GC.start
        return res
    end
end

···

--
Posted via http://www.ruby-forum.com/.

13

To perhaps underscore Robert's point, I did this in IRB on the OCI
version on WinXP, and its behavior appeared (just watching memory
usage in the Windows task manager) very similar to the with-gc variant
on cygwin, even though it was run without explicit GC calls.
(Interestingly, about 27 MB of the excess over what IRB was using
before running this remained in use afterward until an explicit
GC.start, and the explicit call to GC only dropped that by about 21
MB.)

···

On Feb 18, 2008 12:40 AM, Robert Klemme <shortcutter@googlemail.com> wrote:

2008/2/16, Piotr Sawicki <piotr.sawicki@gmail.com>:
> Robert Klemme wrote:
> > Your point being?
>
> I run this on ruby-1.8.6 (Linux and cygwin [WinXP]).
> Why memory used by ruby process grow?
> Is that normal behaviour ?
>
> I wrote the same progam on python and perl and I have discover that that
> this problem occur only in ruby.

Well, concluding from the postings not all versions of Ruby on all
platforms suffer this phenomenon. As far as I can see so far you have
proven that

(1) on cygwin on Windows XP and on Linux
(2) Ruby version 1.8.6
(3) will allocate and free memory in a way that
(4) the OS shows continuous increasing size of used memory
(5) for a particular program, namely one that allocates memory at a high rate.

Whether that's a bug or not is probably really a question for ruby
core. There might be multiple causes for this observed behavior
including but not limited to Ruby's GC.

Kind regards

robert

--
use.inject do |as, often| as.you_can - without end

14

Beacuse in this program

<code>
a = ""
10000.times do
   a = a + "buble"
end
</code>

in each cycle ruby creates a new string with length:
<No. of cycle> * "buble".length + 1 ('\0')

No. of cycle Heap
------------ ----------
1) a = "buble"
2) a = "bublebuble" , a' = "buble"
3) a = "bublebublebuble", a' = "bublebuble", a'' = "buble"
etc.

so after n-cycles we have
    S = n * (("buble".length + 1) + (("buble".length * n + 1)) / 2
in my program:
    n = 10000
    "buble".size = 5
    S = 10000 * (5 + 1 + 50000 + 1)/2 = 5000 * 50007 = 250_035_000 bytes

In your program the strings occupy only
  1_000_000 * ("blah".length + 1) = 6_000_000 bytes

You refer to cycles inside method "leaker". But this method always
creates the same amount of objects so I don't see reason why ruby eats
more and more memory for each "leaker" call.

loop do
  leaker # call leaker, it creates some amount of objects
  # here all objects are possibly to free (there is no reference to them)
  # so when "leaker" is called one more time and it tries to
  # increase heap capacity it should first frees all these objects
end

My point is that this simple program should use constant amount of
_maximum_ (like mine simple program). memory but it doesn't (even if
we call GC manually it frees memory but heap capacity increases so
each time maximum used memory is greater than before).

···

--
Radosław Bułat

http://radarek.jogger.pl - mój blog

15

That's not a solution. That's a bad workaround which might degrade
performance of other applications.

Cheers

robert

···

2008/2/18, Piotr Sawicki <piotr.sawicki@gmail.com>:

I have simple, but ugly solution for this problem.

class String
    alias plus +
    def +(a)
        res = self.plus(a)
        GC.start
        return res
    end
end

--
use.inject do |as, often| as.you_can - without end

16

Piotr Sawicki wrote:

I have simple, but ugly solution for this problem.

class String
    alias plus +
    def +(a)
        res = self.plus(a)
        GC.start
        return res
    end
end
  
Obviously this is a just an example to illustrate the issue, but

def leaker
    a = ""
    10000.times do
         a << "buble"
    end
    a = ""
end

loop do
    leaker
end

does not exhibit the same memory usage as the OP.

-Justin

17

My point is that this simple program should use constant amount of
_maximum_ (like mine simple program). memory but it doesn't (even if
we call GC manually it frees memory but heap capacity increases so
each time maximum used memory is greater than before).

Write it like this

   def leaker
      GC.disable
      a = ""
      10000.times do
         a = a + "buble"
      end
      GC.enable
      a = ""
   end

Guy Decoux

18

Radosław Bułat wrote:

You refer to cycles inside method "leaker". But this method always
creates the same amount of objects so I don't see reason why ruby eats
more and more memory for each "leaker" call.

Haw gc works ? Here is pseudocode:

malloc_limit = 8_000_000
malloc_increase = 0;

malloc(size)
{
   malloc_increase += size;
   if (malloc_increase > malloc_limit)
     garbage_collect()
}

garbage_collect()
{
   malloc_limit = malloc_limit + (malloc_increase - malloc_limit) *
some_factor
   malloc_increase = 0
}

···

--
Posted via http://www.ruby-forum.com/\.

19

Piotr Sawicki wrote:

Radosław Bułat wrote:

You refer to cycles inside method "leaker". But this method always
creates the same amount of objects so I don't see reason why ruby eats
more and more memory for each "leaker" call.

Haw gc works ? Here is pseudocode:

malloc_limit = 8_000_000
malloc_increase = 0;

malloc(size)
{
   malloc_increase += size;
   if (malloc_increase > malloc_limit)
     garbage_collect()
}

garbage_collect()
{
   malloc_limit = malloc_limit + (malloc_increase - malloc_limit) * some_factor
   malloc_increase = 0
}

In my day they were called libraries. Now its called "wikipedia".

But efore you read up, ask yourself how you *think* it might work. One hint. Its paradoxical. To first collect garbage you could identify everything you don't wan to throw away.

HTH

···

--
The surest sign that intelligent life exists elsewhere in Calvin &
the universe is that none of it has tried to contact us. Hobbes.
--
Eliot ,^..^, Smalltalk - scene not herd

20

In my day they were called libraries. Now its called "wikipedia".

Garbage collection (computer science) - Wikipedia

I hadn't seen this article before, a quick skim and I'd say it looks
like a decent survey, although I'd like to see a little more credit
given to some of the GC innovators:

   Henry Baker,
   Dave Ungar
   Eliot Miranda ?

But efore you read up, ask yourself how you *think* it might work. One
hint. Its paradoxical. To first collect garbage you could identify
everything you don't wan to throw away.

Yes indeed, unless you want a potentially unpleasant surprise! Far
worse for a GC to throw away non-garbage than not to throw away all
garbage.

···

On 2/20/08, Eliot Miranda <eliotm@pacbell.net> wrote:
--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/