Writing long-running daemons without memory leaks?

ruby-talk
1

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I've heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven't confirmed that) for the same reason. I understand that Ruby's GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I've read the Pickaxe's lone page on the subject (in the Duck
Typing chapter) and I've seen Why's article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html).

P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC...

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/.

2

i'm running dozens of ruby daemons, some of which are extremely (months at a
time) long lived and have not seen any gc issues. can you post a specific
(minimal) example that you find leaks memory?

regards.

-a

···

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I've heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven't confirmed that) for the same reason. I understand that Ruby's GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I've read the Pickaxe's lone page on the subject (in the Duck
Typing chapter) and I've seen Why's article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html\).

P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC...

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

3

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,

[snip]

source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

Maybe you already knew this stuff, but, here's some things that might help you:

Set things to nil after you're finished using them.

Avoid continuations. If I recall correctly, the GC doesn't handle them right.

Be careful with blocks (and bindings); they keep a reference to all
local variables that are (potentially) visible in the block, even if
those variables aren't actually used in the block. Any values attached
to those local vars not be garbage collected until the whole block is.
Here's an example:

  def foo
     a1=%w[foo bar baz]
     a2=%w[shish boom bah]
     a3=%w[this one's actually used]
     return bar { p a3 }
  end

  def bar(&block)
    return block
  end

a1 and a2 are leaked (temporarily). Even tho they're unused, they
don't actually get GC'd until block is also GC'd.

You can also use ObjectSpace.each_object to iterate over currently
live objects; this might help you determine which objects are being
leaked.

···

On 3/17/06, Toby DiPasquale <toby@cbcg.net> wrote:

4

In article <7d20c3f1c9ff08ff73faab5feefc8eb2@ruby-forum.com>,

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

_ If your process is not using the memory, it may stay swapped
out, ( look at the real and virtual sizes ). If this is really
an issue, the standard way around it is to do the memory
intensive part in a forked subprocess that exits after it's
done.

_ Booker C. Bense

···

Toby DiPasquale <toby@cbcg.net> wrote:

5

unknown wrote:

i'm running dozens of ruby daemons, some of which are extremely (months
at a
time) long lived and have not seen any gc issues. can you post a
specific
(minimal) example that you find leaks memory?

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

puma:> cat leak.rb
# vim:set ts=4 sw=4 ai:
require 'socket'

class Buffer
    def initialize size
        @size = size
        @buffer =
        @length = 0
    end
    attr_reader :length, :size
    def full? ; @length == @size end
    def empty? ; @length.zero? end
    # Lets you fill up to capacity and then returns
    # what's left over
    def fill data
        if full?
            data
        elsif @length + data.length < @size
            @buffer << data
            @length += data.length
            nil
        else
            l = @size - @length
            @buffer << data[0, l]
            data[l..-1]
        end
    end
    def expunge
        buf = @buffer.join
        @buffer.clear
        @length = 0
        buf
    end
end # class Buffer

def read_long s
    s.read( 4).unpack( "N")[0]
end

# main client handling thread logic
client_handler = lambda do |s|
    begin
        len = read_long s
        buf = Buffer.new len
        until buf.full?
            x = s.read 4096
            break if x.nil?
            buf.fill x
        end
        puts "writing #{buf.length} bytes"
        f = File.open "/dev/null", "w"
        f.write buf.expunge
        f.close
    ensure
        s.close
    end
end

# main server loop
ss = TCPServer.new '127.0.0.1', 10001
begin
    while true
        s = ss.accept
        puts "got connection"
        Thread.start s, &client_handler
    end
ensure
    ss.close
end

puma:~> cat leak_client.rb
# vim:set ts=4 sw=4 ai:
require 'socket'

def write_long s, l
        s.write( [l].pack( "N"))
end

str = "a" * 65536

t = TCPSocket.new '127.0.0.1', 10001
write_long t, 1024 * str.length
1024.times { t.write str }
t.close

puma:~>

And here's the output from a simple test to show the issue:

puma:~> for i in 1 2 3 4 5 6 7 8 9 10; do ps aux | grep [l]eak.rb ; ruby
leak_client.rb ; done
toby 8452 0.0 0.1 3116 1684 pts/3 S+ 11:23 0:00 ruby
leak.rb
toby 8452 3.8 12.8 143032 133432 pts/3 Sl+ 11:23 0:01 ruby
leak.rb
toby 8452 6.4 15.8 173664 164020 pts/3 Sl+ 11:23 0:02 ruby
leak.rb
toby 8452 7.8 15.2 167988 158472 pts/3 Sl+ 11:23 0:03 ruby
leak.rb
toby 8452 9.2 15.8 173532 164004 pts/3 Sl+ 11:23 0:03 ruby
leak.rb
toby 8452 10.7 15.2 168024 158508 pts/3 Sl+ 11:23 0:04 ruby
leak.rb
toby 8452 12.0 15.8 173568 164000 pts/3 Sl+ 11:23 0:04 ruby
leak.rb
toby 8452 13.2 15.8 173568 164008 pts/3 Sl+ 11:23 0:05 ruby
leak.rb
toby 8452 14.3 15.8 173568 164012 pts/3 Sl+ 11:23 0:06 ruby
leak.rb
toby 8452 15.7 15.0 165012 155496 pts/3 Sl+ 11:23 0:06 ruby
leak.rb
puma:~> ps aux | grep [l]eak.rb
toby 8452 13.0 22.1 239132 229544 pts/3 Sl+ 11:23 0:07 ruby
leak.rb
puma:~>

This memory never really goes away. Also notice that the client is
sending 64MB, but the first time the leak.rb image jumps to 143MB. Any
clues? Am I just doing something really stupid? TIA.

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.

6

Booker C.Bense wrote:

_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

I thought they did return freed memory. Am I misreading the following?

irb(main):003:0> puts `ps`
  PID TTY TIME CMD
7791 pts/7 00:00:00 zsh
13495 pts/7 00:00:00 irb
13501 pts/7 00:00:00 ps
=> nil
irb(main):004:0> puts `ps -Ovsz`
  PID VSZ S TTY TIME COMMAND
7791 6068 S pts/7 00:00:00 /bin/zsh
13495 8404 S pts/7 00:00:00 irb
13502 3664 R pts/7 00:00:00 ps -Ovsz
=> nil
irb(main):005:0> puts `ps -Ovsz,rss`
  PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 8404 5100 S pts/7 00:00:00 irb
13503 3668 888 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):006:0> def usemem(sz); @x = Array.new(sz); nil; end
=> nil
irb(main):007:0> usemem 10_000_000
=> nil
irb(main):008:0> puts `ps -Ovsz,rss`
  PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 47468 44192 S pts/7 00:00:00 irb
13504 3668 892 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):009:0> usemem 10_000
=> nil
irb(main):010:0> puts `ps -Ovsz,rss`
  PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 47468 44232 S pts/7 00:00:00 irb
13505 3664 884 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):011:0> GC.start
=> nil
irb(main):012:0> puts `ps -Ovsz,rss`
  PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 8404 5168 S pts/7 00:00:00 irb
13506 3664 888 R pts/7 00:00:00 ps -Ovsz,rss
=> nil

···

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

7

i cannot reproduce it:

     harp:~ > cat a.rb
     require 'socket'

     class Buffer
        def initialize size
            @size = size
            @buffer =
            @length = 0
        end
        attr_reader :length, :size
        def full? ; @length == @size end
        def empty? ; @length.zero? end
        # Lets you fill up to capacity and then returns
        # what's left over
        def fill data
            if full?
                data
            elsif @length + data.length < @size
                @buffer << data
                @length += data.length
                nil
            else
                l = @size - @length
                @buffer << data[0, l]
                data[l..-1]
            end
        end
        def expunge
            buf = @buffer.join
            @buffer.clear
            @length = 0
            buf
        end
     end # class Buffer

     def read_long s
        s.read( 4).unpack( "N")[0]
     end

     # main client handling thread logic
     client_handler = lambda do |s|
        begin
            len = read_long s
            buf = Buffer.new len
            until buf.full?
                x = s.read 4096
                break if x.nil?
                buf.fill x
            end
            puts "writing #{buf.length} bytes"
            f = File.open "/dev/null", "w"
            f.write buf.expunge
            f.close
        ensure
            s.close
        end
     end

     # main server loop
     ss = TCPServer.new '127.0.0.1', 10001
     begin
        while true
            s = ss.accept
            puts "got connection"
            Thread.start s, &client_handler
        end
     ensure
        ss.close
     end

     harp:~ > cat b.rb
     require 'socket'

     def write_long s, l
            s.write( [l].pack( "N"))
     end

     str = "a" * 65536

     t = TCPSocket.new '127.0.0.1', 10001
     write_long t, 1024 * str.length
     1024.times { t.write str }
     t.close

     harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep 'ruby a.rb'|grep -v grep ;done
     USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
     ahoward 13671 12.7 23.1 239328 237956 pts/26 S 10:14 0:44 ruby a.rb
     ahoward 13671 12.7 21.2 219404 218032 pts/26 S 10:14 0:45 ruby a.rb
     ahoward 13671 12.8 22.3 230784 229396 pts/26 S 10:14 0:45 ruby a.rb
     ahoward 13671 13.0 22.3 230784 229412 pts/26 S 10:14 0:45 ruby a.rb
     ahoward 13671 13.1 22.3 230916 229416 pts/26 S 10:14 0:46 ruby a.rb
     ahoward 13671 13.2 15.9 165364 163876 pts/26 S 10:14 0:46 ruby a.rb
     ahoward 13671 13.3 22.3 230904 229424 pts/26 S 10:14 0:47 ruby a.rb
     ahoward 13671 13.4 21.5 225116 221504 pts/26 R 10:14 0:47 ruby a.rb
     ahoward 13671 13.5 28.7 296328 294940 pts/26 S 10:14 0:48 ruby a.rb
     ahoward 13671 13.6 21.5 222480 221108 pts/26 S 10:14 0:48 ruby a.rb
     ahoward 13671 13.7 22.3 230796 229400 pts/26 S 10:14 0:49 ruby a.rb
     ahoward 13671 13.8 21.8 230804 224296 pts/26 R 10:14 0:49 ruby a.rb
     ahoward 13671 13.9 27.3 296344 281256 pts/26 R 10:14 0:50 ruby a.rb
     ahoward 13671 14.0 25.4 265672 261500 pts/26 R 10:14 0:50 ruby a.rb
     ahoward 13671 14.1 22.3 230880 229384 pts/26 S 10:14 0:51 ruby a.rb
     ahoward 13671 14.2 15.9 165340 163864 pts/26 S 10:14 0:51 ruby a.rb
     ahoward 13671 14.3 20.8 230888 214376 pts/26 R 10:14 0:52 ruby a.rb
     ahoward 13671 14.4 19.3 200220 198844 pts/26 S 10:14 0:52 ruby a.rb
     ahoward 13671 14.5 21.2 230852 218484 pts/26 R 10:14 0:53 ruby a.rb
     ahoward 13671 14.6 15.9 165312 163872 pts/26 R 10:14 0:53 ruby a.rb
     ahoward 13671 14.8 28.1 290632 289256 pts/26 S 10:14 0:54 ruby a.rb
     ahoward 13671 14.8 16.0 230776 164724 pts/26 R 10:14 0:54 ruby a.rb
     ahoward 13671 14.9 21.6 230908 222412 pts/26 R 10:14 0:54 ruby a.rb
     ahoward 13671 15.0 15.0 219356 154356 pts/26 R 10:14 0:55 ruby a.rb
     ahoward 13671 15.1 16.2 230864 166464 pts/26 R 10:14 0:55 ruby a.rb
     ahoward 13671 15.2 25.7 265764 264320 pts/26 S 10:14 0:56 ruby a.rb
     ahoward 13671 15.4 22.3 230852 229388 pts/26 S 10:14 0:56 ruby a.rb
     ahoward 13671 15.4 15.9 165312 163864 pts/26 S 10:14 0:57 ruby a.rb
     ahoward 13671 15.5 15.9 165320 163880 pts/26 R 10:14 0:57 ruby a.rb
     ahoward 13671 15.6 28.5 294368 292992 pts/26 S 10:14 0:58 ruby a.rb
     ahoward 13671 15.7 22.3 230808 229388 pts/26 S 10:14 0:58 ruby a.rb
     ahoward 13671 15.8 21.7 230816 223624 pts/26 R 10:14 0:59 ruby a.rb
     ahoward 13671 15.9 15.9 165276 163872 pts/26 R 10:14 0:59 ruby a.rb
     ahoward 13671 16.0 20.6 213384 212008 pts/26 S 10:14 0:59 ruby a.rb
     ahoward 13671 16.1 22.3 230812 229396 pts/26 S 10:14 1:00 ruby a.rb
     ahoward 13671 16.2 22.3 230812 229408 pts/26 S 10:14 1:00 ruby a.rb
     ahoward 13671 16.2 15.9 165280 163872 pts/26 R 10:14 1:01 ruby a.rb
     ahoward 13671 16.3 22.3 230812 229408 pts/26 S 10:14 1:01 ruby a.rb
     ahoward 13671 16.4 19.3 200212 198836 pts/26 S 10:14 1:02 ruby a.rb
     ahoward 13671 16.5 22.3 230840 229400 pts/26 S 10:14 1:02 ruby a.rb
     ahoward 13671 16.5 15.4 159584 158204 pts/26 R 10:14 1:02 ruby a.rb
     ahoward 13671 16.7 27.4 296424 281408 pts/26 R 10:14 1:03 ruby a.rb
     ahoward 13671 16.8 19.3 203424 199044 pts/26 R 10:14 1:03 ruby a.rb
     ahoward 13671 16.9 22.1 230888 227576 pts/26 R 10:14 1:04 ruby a.rb
     ahoward 13671 17.0 25.7 265720 264108 pts/26 S 10:14 1:04 ruby a.rb
     ahoward 13671 17.0 15.9 230812 163952 pts/26 R 10:14 1:05 ruby a.rb
     ahoward 13671 17.1 27.1 296352 278484 pts/26 R 10:14 1:05 ruby a.rb
     ahoward 13671 17.2 28.1 290640 289264 pts/26 S 10:14 1:06 ruby a.rb
     ahoward 13671 17.3 15.9 165248 163860 pts/26 R 10:14 1:06 ruby a.rb
     ahoward 13671 17.4 21.8 230920 224092 pts/26 R 10:14 1:07 ruby a.rb
     ahoward 13671 17.5 22.0 228288 226912 pts/26 S 10:14 1:07 ruby a.rb
     ahoward 13671 17.5 15.9 165264 163872 pts/26 R 10:14 1:07 ruby a.rb
     ahoward 13671 17.6 22.0 228228 226852 pts/26 S 10:14 1:08 ruby a.rb
     ahoward 13671 17.7 22.3 230868 229400 pts/26 S 10:14 1:08 ruby a.rb
     ahoward 13671 17.8 14.8 153684 152304 pts/26 R 10:14 1:09 ruby a.rb
     ahoward 13671 17.9 22.3 230812 229400 pts/26 S 10:14 1:09 ruby a.rb
     ahoward 13671 18.0 21.7 225108 223732 pts/26 S 10:14 1:10 ruby a.rb
     ahoward 13671 18.1 20.6 230924 212128 pts/26 R 10:14 1:10 ruby a.rb
     ahoward 13671 18.1 14.4 149944 148564 pts/26 R 10:14 1:11 ruby a.rb
     ahoward 13671 18.3 28.7 296332 294936 pts/26 S 10:14 1:11 ruby a.rb
     ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby a.rb
     ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby a.rb
     ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby a.rb

so memory builds to about 25 and, presumably when the gc kicks in, drops to 15
- but it certainly stays in this range.

what os and ruby version?

-a

···

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

8

unknown wrote:

     harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep

[...]

     ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby
a.rb
     ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby
a.rb
     ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby
a.rb

so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15
- but it certainly stays in this range.

what os and ruby version?

ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB
RAM. The other systems having this problem are all also running Linux
2.6.x with ruby 1.8.4.

One thing about the above: you do realize that's ~250MB and ~150MB, not
25 and 15, right?

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.

9

yes - but it's definitely not leaking. i'm running

   harp:~ > ruby -v
   ruby 1.8.4 (2006-01-12) [i686-linux]

   harp:~ > cat /etc/redhat-release
   Red Hat Enterprise Linux WS release 3 (Taroon Update 6)

   harp:~ > uname -srm
   Linux 2.4.21-37.0.1.EL i686

btw.

it's clear to me that your code is using tons of memory - it's not clear that
it's leaking, however.

if you looking for why your code is so big look to you handler: changing it
to

   Thread.start(s){|s| loop{ s.read(8192) or break }}

will result in runs that look like this:

   harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep 'ruby a.rb'|grep -v grep ;done
   USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
   ahoward 15535 4.1 0.4 6004 4556 pts/26 S 10:53 0:00 ruby a.rb
   ahoward 15535 7.1 0.7 8940 7520 pts/26 S 10:53 0:00 ruby a.rb
   ahoward 15535 9.4 0.2 4080 2704 pts/26 S 10:53 0:00 ruby a.rb
   ahoward 15535 10.7 0.5 6820 5432 pts/26 S 10:53 0:01 ruby a.rb
   ahoward 15535 13.0 0.7 9676 8212 pts/26 S 10:53 0:01 ruby a.rb
   ahoward 15535 13.2 0.3 4552 3176 pts/26 S 10:53 0:01 ruby a.rb
   ahoward 15535 15.0 0.5 7156 5712 pts/26 S 10:53 0:01 ruby a.rb
   ahoward 15535 15.9 0.8 9684 8300 pts/26 S 10:53 0:01 ruby a.rb
   ahoward 15535 17.1 0.2 4432 3056 pts/26 S 10:53 0:02 ruby a.rb
   ahoward 15535 19.2 0.5 6892 5392 pts/26 S 10:53 0:02 ruby a.rb
   ahoward 15535 19.5 0.7 9228 7796 pts/26 S 10:53 0:02 ruby a.rb
   ahoward 15535 21.1 0.2 3712 2332 pts/26 S 10:53 0:02 ruby a.rb
   ahoward 15535 21.6 0.4 5908 4484 pts/26 S 10:53 0:03 ruby a.rb

so it's the handler/Buffer class that consumes (but does't leak) so much
memory.

in any case, i think you are making the code too hard : ruby already does
massive internal buffering - you can do what you want with less code i think:

   harp:~ > cat netstring.rb
   class NetString < ::String
     def initialize arg = ""
       arg.respond_to?("read") ? read(arg) : super(arg)
     end
     def clear
       self.replace ""
     end
     def read port
       clear
       self.replace(port.read(port.read(4).unpack("N").first))
     end
     def write port
       port << [size].pack("N")
       port << self
     end
   end

   harp:~ > cat a.rb
   require 'socket'
   require 'netstring'

   STDOUT.sync = true

   ss = TCPServer.new '127.0.0.1', 10001

   loop do
     s = ss.accept
     puts "got connection"
     Thread.start(s) do |s|
       File.open("/dev/null", "w") do |f|
         ns = NetString.new s
         puts "received #{ ns.size } bytes"
         written = f.write ns
         puts "wrote #{ written } bytes"
       end
     end
   end

   harp:~ > cat b.rb
   require 'socket'
   require 'netstring'

   loop do
     TCPSocket.open('127.0.0.1', 10001) do |t|
       NetString::new("a" * 65536).write t
     end
   end

   harp:~ > ruby a.rb # terminal one

   harp:~ > ruby b.rb # terminal two

   harp:~ > ps aux|head -1;{ while true;do ps aux|grep 'ruby a.rb'|grep -v grep;sleep 1;done; }|head -42
   USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
   ahoward 17990 10.7 1.1 13480 11868 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 1.1 13164 11704 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 1.1 13660 12144 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 1.1 13424 11904 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 1.1 13328 11792 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 1.1 13784 12184 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.7 0.9 11040 9496 pts/26 S 11:30 0:27 ruby a.rb
   ahoward 17990 10.8 0.9 10904 9428 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.7 0.9 11968 10220 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.7 1.1 13464 12056 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.7 0.9 11000 9476 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.7 0.9 10992 9472 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.8 1.1 13364 11912 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.8 0.9 10976 9464 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.8 1.1 13356 11908 pts/26 S 11:30 0:28 ruby a.rb
   ahoward 17990 10.8 1.1 13796 12228 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.1 13572 12116 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.1 13836 12244 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.1 13872 12284 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.2 14112 12436 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.2 14224 12428 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.1 13888 12292 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.1 13896 12296 pts/26 S 11:30 0:29 ruby a.rb
   ahoward 17990 10.8 1.2 14200 12416 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.1 13856 12276 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.2 13960 12328 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.1 13624 12164 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.2 14048 12384 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.1 13984 12308 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.2 14088 12424 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.1 13744 12220 pts/26 S 11:30 0:30 ruby a.rb
   ahoward 17990 10.8 1.2 14040 12396 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.1 13496 12100 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.2 14056 12472 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.2 14080 12452 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.1 13520 12108 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.1 13656 12276 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.1 13624 12160 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.2 13824 12392 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.2 13912 12432 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.2 14152 12520 pts/26 S 11:30 0:31 ruby a.rb
   ahoward 17990 10.8 1.1 13768 12232 pts/26 S 11:30 0:32 ruby a.rb

kind regards.

-a

···

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

unknown wrote:

     harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep

[...]

     ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby
a.rb
     ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby
a.rb
     ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby
a.rb

so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15
- but it certainly stays in this range.

what os and ruby version?

ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB
RAM. The other systems having this problem are all also running Linux
2.6.x with ruby 1.8.4.

One thing about the above: you do realize that's ~250MB and ~150MB, not
25 and 15, right?

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

10

ahoward wrote:

if you looking for why your code is so big look to you handler:
changing it
to

   Thread.start(s){|s| loop{ s.read(8192) or break }}

will result in runs that look like this:

<snip>

Interesting.

so it's the handler/Buffer class that consumes (but does't leak) so much
memory.

in any case, i think you are making the code too hard : ruby already
does
massive internal buffering - you can do what you want with less code i
think:

   harp:~ > cat netstring.rb
   class NetString < ::String
     def initialize arg = ""
       arg.respond_to?("read") ? read(arg) : super(arg)
     end
     def clear
       self.replace ""
     end
     def read port
       clear
       self.replace(port.read(port.read(4).unpack("N").first))
     end
     def write port
       port << [size].pack("N")
       port << self
     end
   end

   harp:~ > cat a.rb
   require 'socket'
   require 'netstring'

   STDOUT.sync = true

   ss = TCPServer.new '127.0.0.1', 10001

   loop do
     s = ss.accept
     puts "got connection"
     Thread.start(s) do |s|
       File.open("/dev/null", "w") do |f|
         ns = NetString.new s
         puts "received #{ ns.size } bytes"
         written = f.write ns
         puts "wrote #{ written } bytes"
       end
     end
   end

   harp:~ > cat b.rb
   require 'socket'
   require 'netstring'

   loop do
     TCPSocket.open('127.0.0.1', 10001) do |t|
       NetString::new("a" * 65536).write t
     end
   end

<snip>

Thanks, a. I will try to incorporate your suggestions when I get it
working again.

···

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/\.