I’m using a lot of Tempfile’s and I have been made aware that they
consume a lot of memory.
I’ve examined the size of the ruby process right before it exits in
the example below, where I make 100 Tempfiles (and for comparison 100
ordinary Files in another run):
require ‘tempfile’
temps =
100.times {
i>
t = Tempfile.new(“blah#{i}.txt”, “.”)
#t = File.new(“blah#{i}.txt”, “w”)
t.puts “hello world”
t.close
temps << t
}
Uses 38852KB of memory with Tempfile and only 2820KB of memory with
File (commented out above).
What’s up with that?
Cheers,
Thomas
Hi,
I’m using a lot of Tempfile’s and I have been made aware that they
consume a lot of memory.
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
require ‘tempfile’
temps =
100.times {
i>
t = Tempfile.new(“blah#{i}.txt”, “.”)
#t = File.new(“blah#{i}.txt”, “w”)
t.puts “hello world”
t.close
temps << t # < this line
}
matz.
···
In message “Tempfile consuming a lot of memory” on 04/03/24, Thomas thomass@deltadata.dk writes:
Hi,
At Wed, 24 Mar 2004 04:10:41 +0900,
Yukihiro Matsumoto wrote in [ruby-talk:95653]:
I’m using a lot of Tempfile’s and I have been made aware that they
consume a lot of memory.
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
lib/tempfile.rb:1.20 looks like a commit miss, doesn’t it?
[ruby-core:02684]
···
–
Nobu Nakada
I’m using a lot of Tempfile’s and I have been made aware that they
consume a lot of memory.
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
No, I need to hang on to them for a while. All in all I may easily
need a thousand tempfiles or more, but they wont all be open at the
same time. I don’t understand why the Tempfile’s need to be so
expensive to keep around if I close them when I’m not using them.
Cheers,
Thomas
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
I can understand that, but this fragment uses 320MB of memory, ie. about
300KB per closed Tempfile. That seems wildly unreasonable to me. With
regular files the total size of the ruby process is 3.2MB!
temps =
1000.times {
i>
t = Tempfile.new(“blah#{i}.txt”, “.”)
t.puts “hello world”
t.close
temps << t
}
puts “Done”
gets
I use ruby 1.8.0 (pragprog build) on Windows XP.
Cheers,
Thomas
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
lib/tempfile.rb:1.20 looks like a commit miss, doesn’t it?
[ruby-core:02684]
What do you mean? Is the latest CVS version better?
Cheers,
Thomas
probably bevcause:
close(unlink_now=false)
Closes the file. If the optional flag is true, unlinks the file after
closing.
If you dont explicitly unlink the temporary file, the removal will be
delayed until the object is finalized.
you have references to tempfile so it will nevere be finalized. Maybe
you could use close!
···
il 24 Mar 2004 01:34:19 -0800, thomass@deltadata.dk (Thomas) ha scritto::
I’m using a lot of Tempfile’s and I have been made aware that they
consume a lot of memory.
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove “temps” variable from
your example?
No, I need to hang on to them for a while. All in all I may easily
need a thousand tempfiles or more, but they wont all be open at the
same time. I don’t understand why the Tempfile’s need to be so
expensive to keep around if I close them when I’m not using them.
Hello Thomas,
Wednesday, March 24, 2004, 4:14:29 PM, you wrote:
Tempfile allocates far lot more objects internally than plain File
objects. If you allow garbage collection for these tempfile objects,
its consumption stays moderate. Can you remove "temps" variable from
your example?
I can understand that, but this fragment uses 320MB of memory, ie. about
300KB per closed Tempfile. That seems wildly unreasonable to me. With
regular files the total size of the ruby process is 3.2MB!
temps =
1000.times {
>i>
t = Tempfile.new("blah#{i}.txt", ".")
t.puts "hello world"
t.close
temps << t
}
puts "Done"
gets
I use ruby 1.8.0 (pragprog build) on Windows XP.
Have you tried the same under Linux ?
Maybe this can give us a clue what is wrong.
···
--
Best regards,
Lothar mailto:mailinglists@scriptolutions.com
Hi,
At Wed, 24 Mar 2004 10:16:30 +0900,
Yukihiro Matsumoto wrote in [ruby-talk:95676]:
lib/tempfile.rb:1.20 looks like a commit miss, doesn’t it?
[ruby-core:02684]
Fixed, I think.
Re-opening tempfile fails. @data should not be nil, I guess.
$ ruby -v lib/tempfile.rb
ruby 1.9.0 (2004-03-24) [i686-linux]
lib/tempfile.rb:73:in open': undefined method =’ for nil:NilClass (NoMethodError)
from lib/tempfile.rb:183
$ cvs st lib/tempfile.rb
···
===================================================================
File: tempfile.rb Status: Up-to-date
Working revision: 1.21 Wed Mar 24 02:06:07 2004
Repository revision: 1.21 /cvs/ruby/src/ruby/lib/tempfile.rb,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
–
Nobu Nakada
It takes 330MB on my system.
ruby 1.8.1 (2003-12-25) [i686-linux]
···
On Thu, Mar 25, 2004 at 12:43:15AM +0900, Lothar Scholz wrote:
temps =
1000.times {
i>
t = Tempfile.new(“blah#{i}.txt”, “.”)
t.puts “hello world”
t.close
temps << t
}
puts “Done”
gets
Have you tried the same under Linux ?
Maybe this can give us a clue what is wrong.
–
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com
All the existing 2.0.x kernels are to buggy for 2.1.x to be the
main goal.
– Alan Cox
Have you tried the same under Linux ?
Maybe this can give us a clue what is wrong.
I have now:
18:47:02 up 33 days, 23:19, 2 users, load average: 1.38, 0.97, 0.43
58 processes: 57 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 11.3% user 4.7% system 0.0% nice 0.0% iowait 83.8% idle
Mem: 223140k av, 220044k used, 3096k free, 0k shrd, 752k
buff
141028k actv, 29580k in_d, 3632k in_c
Swap: 457844k av, 231440k used, 226404k free 4788k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
26968 ts 15 0 272M 160M 500 D 11.1 73.6 1:16 0 ruby
26966 ts 15 0 4116 1052 500 S 0.0 0.4 0:03 0 emacs
27036 ts 15 0 976 840 684 R 0.5 0.3 0:00 0 top
I grew tired of waiting on it (my linux box only has 256 megs of memory)
when it had about 780 temporary files at which point the process was using
272MB of memory.
So it is pretty much the same on linux - not surprisingly.
I have made a few other experiments, including implementing my own TempFile2
class. If I use the same strategy as Tempfile and use a SimpleDelegator to
delegate to an instance of File I get the same memory consumption. I haven’t
figured out why it is so…
Cheers,
Thomas
I have made a few other experiments, including implementing my own
TempFile2
class. If I use the same strategy as Tempfile and use a SimpleDelegator to
delegate to an instance of File I get the same memory consumption. I
haven’t
figured out why it is so…
Having just written this, I had to do a SimpleDelegator experiment. Here’s
the result.
Size of the ruby interpreter after having created 1000 instances of:
ADelegatingClass: 315MB
ANondelegatingClass: 2.6MB
Well?
Cheers,
Thomas
Here’s the code:
require ‘delegate’
class ADelegatingClass < SimpleDelegator
def initialize
super()
end
end
class ANondelegatingClass < Array
end
temps =
1000.times {
i>
t = ADelegatingClass.new
t.size
temps << t
}
puts “Done”
gets
Hello Thomas,
Wednesday, March 24, 2004, 6:54:31 PM, you wrote:
Have you tried the same under Linux ?
Maybe this can give us a clue what is wrong.
I have now:
18:47:02 up 33 days, 23:19, 2 users, load average: 1.38, 0.97, 0.43
58 processes: 57 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 11.3% user 4.7% system 0.0% nice 0.0% iowait 83.8% idle
Mem: 223140k av, 220044k used, 3096k free, 0k shrd, 752k
buff
141028k actv, 29580k in_d, 3632k in_c
Swap: 457844k av, 231440k used, 226404k free 4788k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
26968 ts 15 0 272M 160M 500 D 11.1 73.6 1:16 0 ruby
26966 ts 15 0 4116 1052 500 S 0.0 0.4 0:03 0 emacs
27036 ts 15 0 976 840 684 R 0.5 0.3 0:00 0 top
I grew tired of waiting on it (my linux box only has 256 megs of memory)
when it had about 780 temporary files at which point the process was using
272MB of memory.
So it is pretty much the same on linux - not surprisingly.
I have made a few other experiments, including implementing my own TempFile2
class. If I use the same strategy as Tempfile and use a SimpleDelegator to
delegate to an instance of File I get the same memory consumption. I haven't
figured out why it is so..
I also had a quick look at TempFile and couldn't find something in the
data modell that would justify this memory overhead. So i guess you
are right it has something to do with the code model und the internal
representation of the parse tree. Have you looked at the object space
which objects are available at the end of the run ?
···
--
Best regards,
Lothar mailto:mailinglists@scriptolutions.com
Hi,
Having just written this, I had to do a SimpleDelegator experiment. Here’s
the result.
Size of the ruby interpreter after having created 1000 instances of:
ADelegatingClass: 315MB
ANondelegatingClass: 2.6MB
Well?
This is a good hint. Thomas, can you try this patch?
matz.
— lib/tempfile.rb 24 Mar 2004 09:27:15 -0000 1.22
+++ lib/tempfile.rb 25 Mar 2004 01:27:09 -0000
@@ -11,3 +11,3 @@ require ‘tmpdir’
thread safe.
-class Tempfile < SimpleDelegator
+class Tempfile < DelegateClass(File)
MAX_TRY = 10
— lib/delegate.rb 27 Jan 2004 06:04:59 -0000 1.15
+++ lib/delegate.rb 25 Mar 2004 01:27:09 -0000
@@ -113,2 +113,5 @@ def DelegateClass(superclass)
end
- def setobj(obj)
- @_dc_obj = obj
- end
return klass;
···
In message “Re: Tempfile consuming a lot of memory” on 04/03/25, “Thomas Sondergaard” thomas@FirstNameGoesHereSondergaard.com writes:
This is a good hint. Thomas, can you try this patch?
Yes, it is much better. The footprint of my original application just went
from 167MB to 11MB. A nice improvement :o)
Will this fix be committed to CVS and be available from ruby 1.8.2?
Cheers,
Thomas