Ruby on Unix vs. Windows

ruby-talk
1

I suspect mostly the latter, with a little bit of the former mixed in
for good measure. I'm pretty sure the GC algorithm is different on
Windows, though I can't remember the details now.

Any chance you could show us the relevant code? Maybe we could help you
with the memory footprint in general.

Regards,

Dan

···

-----Original Message-----
From: Rick Nooner [mailto:rick@nooner.net]
Sent: Tuesday, August 16, 2005 3:01 PM
To: ruby-talk ML
Subject: Ruby on Unix vs. Windows

Yesterday at work we took an analysis program written in ruby
that we had been running on a Solaris box (Sunblade 1500, 1
Gig RAM, 1.5 Ghz Sparc) and moved it to a windows box (HP
D530, 1 Gig RAM, 2.8 Ghz Pentium) to do performance comparisons.

The analysis builds a profile in memory of over 3.6 GB of
data on disk. On the Solaris box, it takes about 35 mins and
uses about 700 MB of RAM. It would not complete on the
windows box using the full data set, bombing with "failed to
allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.

Is windows that inefficient with memory allocation or is this
a ruby implementation issue on windows?

2

I suspect mostly the latter, with a little bit of the former mixed in
for good measure. I'm pretty sure the GC algorithm is different on
Windows, though I can't remember the details now.

That's what I suspect as well.

Any chance you could show us the relevant code? Maybe we could help you
with the memory footprint in general.

I'm afraid that I can't. That's the problem when you work for a big company. Even
though most of my work is in Ruby, I can share very little of it. I can tell you what
it is though. It's a SIP protocol analyzer and it builds SIP call flows in memory
from snoop files so there is alot of state that has to be kept resident in memory for
each call (and we are talking about 1000's of simultanous calls and millions of
messages).

Thanks,
Rick

···

On Wed, Aug 17, 2005 at 06:38:30AM +0900, Berger, Daniel wrote:

--
Rick Nooner
rick@nooner.net

3

Hello Daniel,

The analysis builds a profile in memory of over 3.6 GB of
data on disk. On the Solaris box, it takes about 35 mins and
uses about 700 MB of RAM. It would not complete on the
windows box using the full data set, bombing with "failed to
allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.

Is windows that inefficient with memory allocation or is this
a ruby implementation issue on windows?

I suspect mostly the latter, with a little bit of the former mixed in
for good measure. I'm pretty sure the GC algorithm is different on
Windows, though I can't remember the details now.

No its exactly the same. Remember that the ruby GC has an extremely
dumb implementation.

The problem is MSVC runtime. It is well known to be fragmenting. There
are many drop in replacements. Programs with large datasets should
never use the default malloc/free.

···

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

4

I looked at Ruby's allocator and Lothar is right. It's just a wrapper
around malloc/free.

Also, if it can't allocate memory, it then runs the GC and tries again.
If it fails a second time, it gives up and throws an error.

This means that calling the GC explicitly won't do any good since it'll
have the same effect as just letting the allocator take care of it.

Rick

···

On Wed, Aug 17, 2005 at 08:17:07AM +0900, Lothar Scholz wrote:

> for good measure. I'm pretty sure the GC algorithm is different on
> Windows, though I can't remember the details now.

No its exactly the same. Remember that the ruby GC has an extremely
dumb implementation.

The problem is MSVC runtime. It is well known to be fragmenting. There
are many drop in replacements. Programs with large datasets should
never use the default malloc/free.

--
Rick Nooner
rick@nooner.net

5

Lothar Scholz wrote:

Hello Daniel,

The analysis builds a profile in memory of over 3.6 GB of
data on disk. On the Solaris box, it takes about 35 mins and
uses about 700 MB of RAM. It would not complete on the
windows box using the full data set, bombing with "failed to
allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap
available.

Is windows that inefficient with memory allocation or is this
a ruby implementation issue on windows?

I suspect mostly the latter, with a little bit of the former mixed in
for good measure. I'm pretty sure the GC algorithm is different on
Windows, though I can't remember the details now.

No its exactly the same. Remember that the ruby GC has an extremely
dumb implementation.

The problem is MSVC runtime. It is well known to be fragmenting. There
are many drop in replacements. Programs with large datasets should
never use the default malloc/free.

Also I believe the limit for user process memory is lower on 32 bit
windows boxes than on 32 bit Solaris systems due to the way windows
relocates DLL's etc.

Kind regards

    robert

6

Lothar Scholz wrote:

Hello Daniel,

The analysis builds a profile in memory of over 3.6 GB of data on disk. On the Solaris box, it takes about 35 mins and uses about 700 MB of RAM. It would not complete on the windows box using the full data set, bombing with "failed to allocate memory (NoMemoryError)". There was nearly 800 MB of RAM free on the windows box as well as having a 4 Gig swap available.

Is windows that inefficient with memory allocation or is this a ruby implementation issue on windows?

> I suspect mostly the latter, with a little bit of the former mixed in
> for good measure. I'm pretty sure the GC algorithm is different on
> Windows, though I can't remember the details now.

No its exactly the same. Remember that the ruby GC has an extremely
dumb implementation.

Well, consider the following script. On Solaris the memory peaks at about 8k, but never drops. On Windows it climbs to about 11k, then drops, climbs again, drops, etc.

I could have sworn I read somewhere that it had to do with the way Ruby was allocating memory. Or is it just the operating system?

Regards

Dan

# blocktest.rb
class BlockTest
    def test
       yield "hello" * 1000
    end
end

if $0 == __FILE__
    require "sys/proctable"
    include Sys

    p Process.pid

    # Windows and Solaris
    if File::ALT_SEPARATOR
       mem = ProcTable.ps(Process.pid).working_set_size
       vmem = ProcTable.ps(Process.pid).page_file_usage
    else
       mem = ProcTable.ps(Process.pid).size
       vmem = ProcTable.ps(Process.pid).rss
    end

    puts "Starting Memory: #{mem}"
    puts "Starting VMemory: #{vmem}"

    b = BlockTest.new

    100000.times{ |n|
       b.test{ }
       sleep 0.5

       if File::ALT_SEPARATOR
          cur_mem = ProcTable.ps(Process.pid).working_set_size
          cur_vmem = ProcTable.ps(Process.pid).page_file_usage
       else
          cur_mem = ProcTable.ps(Process.pid).size
          cur_vmem = ProcTable.ps(Process.pid).rss
       end

       if cur_mem < mem
          puts "Memory dropped to #{cur_mem} from #{mem} on iter #{n}"
          puts "Current VMemory: #{cur_vmem}"
          puts
          mem = cur_mem
       else
          mem = cur_mem
          vmem = cur_vmem
          if n % 100 == 0
             puts "Current Memory: #{mem}"
             puts "Current VMemory: #{vmem}"
             puts
          end
       end

       if cur_vmem < vmem
          puts "VMemory dropped to #{cur_vmem} from #{vmem} on iter #{n}"
          puts "Current Memory: #{cur_mem}"
          puts
          vmem = cur_vmem
       else
          mem = cur_mem
          vmem = cur_vmem

          # Print every 100th iteration
          if n % 100 == 0
             puts "Current Memory: #{mem}"
             puts "Current VMemory: #{vmem}"
             puts
          end
       end
    }
end