Yesterday at work we took an analysis program written in ruby that we had been
running on a Solaris box (Sunblade 1500, 1 Gig RAM, 1.5 Ghz Sparc) and moved
it to a windows box (HP D530, 1 Gig RAM, 2.8 Ghz Pentium) to do performance
comparisons.
The analysis builds a profile in memory of over 3.6 GB of data on disk. On
the Solaris box, it takes about 35 mins and uses about 700 MB of RAM. It
would not complete on the windows box using the full data set, bombing with
"failed to allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.
Is windows that inefficient with memory allocation or is this a ruby
implementation issue on windows?
Before anyone asks, I would have done the comparison on a linux box if I
had one available since what I really wanted to know was the speed boost
that a Intel processor would give me. I didn't have one readly available,
though.
Yesterday at work we took an analysis program written in ruby that we had been
running on a Solaris box (Sunblade 1500, 1 Gig RAM, 1.5 Ghz Sparc) and moved
it to a windows box (HP D530, 1 Gig RAM, 2.8 Ghz Pentium) to do performance
comparisons.
The analysis builds a profile in memory of over 3.6 GB of data on disk. On
the Solaris box, it takes about 35 mins and uses about 700 MB of RAM. It
would not complete on the windows box using the full data set, bombing with
"failed to allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.
Is windows that inefficient with memory allocation or is this a ruby
implementation issue on windows?
Any chance the program does any large block allocations?
Like more than 250 MB in one chunk? I've noticed windows
performs poorly when a program churns lots of small
memory allocations, then occasionally wants to grab a big
block of memory. I have a test program (two, actually -
one in ruby, the other in C using malloc) that doesn't
fail on Linux and Darwin, and does fail on Windows. Even
though the Linux and Darwin systems had less physical
RAM and less swap than the Windows system with its 2GB
ram and 4GB swap. When it fails on Windows, there's
plenty of system memory still available - but the windows
heap management has allowed the process virtual memory
space to become so fragmented that there's no room to map
the large block allocation into the process' virtual
address space.
Look around for a Live CD distribution like Knoppix. But find one
without ruby. Set aside a partition for Linux ( it can be Fat32 )
mount the partition somewhere and install ruby ( you DON'T want
ruby running of the CD for benchmarks even hand waiving ones ).
The hard part will be finding a Live CD without ruby.
It's been a while, but I seem to remember that Intel CPUs make memory
management for things like GC much harder, so it would be a good idea
to compare OSs running on the same CPU. If not maybe you can scrounge
an Alpha running NT and see if you get any different results on it.
The reply-to email address is olczyk2002@yahoo.com.
This is an address I ignore.
To reply via email, remove 2002 and change yahoo to
interaccess,
···
On Wed, 17 Aug 2005 06:00:33 +0900, Rick Nooner <rick@nooner.net> wrote:
Yesterday at work we took an analysis program written in ruby that we had been
running on a Solaris box (Sunblade 1500, 1 Gig RAM, 1.5 Ghz Sparc) and moved
it to a windows box (HP D530, 1 Gig RAM, 2.8 Ghz Pentium) to do performance
comparisons.
The analysis builds a profile in memory of over 3.6 GB of data on disk. On
the Solaris box, it takes about 35 mins and uses about 700 MB of RAM. It
would not complete on the windows box using the full data set, bombing with
"failed to allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.
Is windows that inefficient with memory allocation or is this a ruby
implementation issue on windows?
Before anyone asks, I would have done the comparison on a linux box if I
had one available since what I really wanted to know was the speed boost
that a Intel processor would give me. I didn't have one readly available,
though.
**
Thaddeus L. Olczyk, PhD
There is a difference between
*thinking* you know something,
and *knowing* you know something.
Yesterday at work we took an analysis program written in ruby that we had been
running on a Solaris box (Sunblade 1500, 1 Gig RAM, 1.5 Ghz Sparc) and moved
it to a windows box (HP D530, 1 Gig RAM, 2.8 Ghz Pentium) to do performance
comparisons.
The analysis builds a profile in memory of over 3.6 GB of data on disk. On
the Solaris box, it takes about 35 mins and uses about 700 MB of RAM. It
would not complete on the windows box using the full data set, bombing with
"failed to allocate memory (NoMemoryError)". There was nearly 800 MB of
RAM free on the windows box as well as having a 4 Gig swap available.
Is windows that inefficient with memory allocation or is this a ruby
implementation issue on windows?
Before anyone asks, I would have done the comparison on a linux box if I
had one available since what I really wanted to know was the speed boost
that a Intel processor would give me. I didn't have one readly available,
though.
Thanks,
Rick
My only experience with this is through SQL Server 2000. Large processes were not completing even though the machince had plenty of memory. Our theory was that it was a combination of NTFS and deframentation of the swap file. It seemed as though NTFS security was trying to keep track of everything and using even more memory. The swap file also could not respond fast enough to the calls because SQL Server was busy on the same disks. What we did was take and old 9gb scsi disk with FAT32 and set the swap file on to it. Not a perfect solution but it worked.
You are describing exactly the behavior that I'm seeing.
My program does a lot of small block allocations. How that's handled by Ruby under the
hood, I don't know. If the allocator see's lots of small allocations so starts asking
the OS for larger and larger chunks of memory, what you are describing could happen.
If this has happened to you using malloc, it's probably beyond Ruby's control, unless
Ruby replaces the default memory allocation mechanism.
I'll write a test program to test this theory.
Thanks,
Rick
···
On Wed, Aug 17, 2005 at 06:53:15AM +0900, Bill Kelly wrote:
Any chance the program does any large block allocations?
Like more than 250 MB in one chunk? I've noticed windows
performs poorly when a program churns lots of small
memory allocations, then occasionally wants to grab a big
block of memory. I have a test program (two, actually -
one in ruby, the other in C using malloc) that doesn't
fail on Linux and Darwin, and does fail on Windows. Even
though the Linux and Darwin systems had less physical
RAM and less swap than the Windows system with its 2GB
ram and 4GB swap. When it fails on Windows, there's
plenty of system memory still available - but the windows
heap management has allowed the process virtual memory
space to become so fragmented that there's no room to map
the large block allocation into the process' virtual
address space.
Another option is SFU(www.interix.com). I believe there is a Ruby 1.8.2 binary package. I don't even have to worry about / or \, : or ; once I'm in the SFU environment.
Joel VanderWerf wrote:
···
Rick Nooner wrote:
> If this has happened to you using malloc, it's probably beyond Ruby's control, unless
> Ruby replaces the default memory allocation mechanism.
What about testing with cygwin or mingw32 ruby? I'm guessing those use
different allocators than win32 ruby.
--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
If this has happened to you using malloc, it's probably beyond Ruby's control, unless
Ruby replaces the default memory allocation mechanism.
What about testing with cygwin or mingw32 ruby? I'm guessing those use
different allocators than win32 ruby.
I would be curious to see if mingw32 has the same problems (remembering the big mingw vs msvc debates of yesteryear). As a Windows user this is a worrisome thing to me. If it is a "known" problem then is it going to be fixed at some point in time?
As a Windows user this is
a worrisome thing to me. If it is a "known" problem then is it going to
be fixed at some point in time?
I encountered this problem a year or so ago working
on my employer's application which is all C / C++.
Our customers sometimes browse very large images,
upwards of 200 or 300 megs, which our software will
attempt to load entirely into memory if requested to
do so.
Oh my! That's ptmalloc on Windows, isn't it?! That little thing seriously rocks in multi-threaded systems with a lot of dynamic memory allocation. It can resolve nasty scalability bottlenecks there by getting rid of a mutex in malloc/free.
Not sure if it will do much good for Ruby though - Ruby threads are green, not native.
If you get some results, please share with the list.
Oh my! That's ptmalloc on Windows, isn't it?! That little thing
seriously rocks in multi-threaded systems with a lot of dynamic memory
allocation. It can resolve nasty scalability bottlenecks there by
getting rid of a mutex in malloc/free.
Not sure if it will do much good for Ruby though - Ruby threads are
green, not native.
Yes it will improve the runtime speed but it also has lower
fragmentation then the original msvcrt.dll implementation.
···
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's