cfp2:~ > cat a.rb
#! /usr/bin/env ruby
require 'net/http'
3.times do
Net::HTTP.start('www.google.com') do |http|
http_response = http.get '/'
end
p ObjectSpace.each_object(Net::HTTPResponse){}
end
cfp2:~ > ruby a.rb
1
2
3
why is this leaking Net::HTTPResponse objects?
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama
why is this leaking Net::HTTPResponse objects?
I don't know, but jruby has the exact same behaviour. Maybe that helps?
It does not leak, objects are just not immediately released by the GC. Try it
running 1000 times (on local server preferably).
Jan
On Wednesday 09 January 2008 04:23:23 ara howard wrote:
why is this leaking Net::HTTPResponse objects?
Why are you assuming it's leaking without giving the GC a chance to run?
require 'net/http'
3.times do
Net::HTTP.start('www.google.com') do |http|
http_response = http.get '/'
end
p ObjectSpace.each_object(Net::HTTPResponse){}
end
p "Force GC"
ObjectSpace.garbage_collect
p ObjectSpace.each_object(Net::HTTPResponse){}
C:\>ruby test.rb
1
2
3
"Force GC"
0
MBL
On Jan 8, 2008 10:23 PM, ara howard <ara.t.howard@gmail.com> wrote:
cfp2:~ > cat a.rb
#! /usr/bin/env rubyrequire 'net/http'
3.times do
Net::HTTP.start('www.google.com') do |http|
http_response = http.get '/'
endp ObjectSpace.each_object(Net::HTTPResponse){}
endcfp2:~ > ruby a.rb
1
2
3why is this leaking Net::HTTPResponse objects?
a @ http://codeforpeople.com/
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama
because i'm an idiot and pasted the wrong buffer. here is the code:
On Jan 8, 2008, at 8:53 PM, Michael Bevilacqua-Linn wrote:
Why are you assuming it's leaking without giving the GC a chance to run?
#
# distilled behaviour from dike.rb
#
class Object
Methods = instance_methods.inject(Hash.new){|h, m| h.update m => instance_method(m)}
end
class Class
Methods = instance_methods.inject(Hash.new){|h, m| h.update m => instance_method(m)}
def new *a, &b
object = Methods["new"].bind(self).call *a, &b
ensure
ObjectSpace.define_finalizer object, finalizer
end
def finalizer
lambda{}
end
end
#
# the above makes this code leaks, but *only* Net::HTTPOK objects
#
require "net/http"
def leak!
Net::HTTP.start("localhost") do |http|
puts http.get('/').code
end
end
3.times {
puts "---"
p ObjectSpace.each_object(Net::HTTPResponse){}
leak!
GC.start
}
a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama
as i already posted - i posted the wrong code. here is an even more distilled version:
cfp2:~ > cat a.rb
On Jan 8, 2008, at 8:45 PM, Jan Dvorak wrote:
It does not leak, objects are just not immediately released by the GC. Try it
running 1000 times (on local server preferably).
#
# distilled behaviour from dike.rb
#
class Class
Finalizer = lambda {}
def leak_free_finalizer
Finalizer
end
def leaky_finalizer
lambda{}
end
def finalizer
%r/leak/ =~ ARGV.first ? leaky_finalizer : leak_free_finalizer
end
def new *a, &b
object = allocate
object.send :initialize, *a, &b
object
ensure
ObjectSpace.define_finalizer object, finalizer
end
end
#
# the above makes this code leak iff ARGV has "leak" in it
#
require "yaml"
7.times {
GC.start
y Array.name => ObjectSpace.each_object(Array){}
Array.new
}
cfp2:~ > ruby a.rb
---
Array: 21
---
Array: 49
---
Array: 58
---
Array: 58
---
Array: 58
---
Array: 58
---
Array: 58
cfp2:~ > ruby a.rb leak
---
Array: 21
---
Array: 38
---
Array: 54
---
Array: 67
---
Array: 79
---
Array: 91
---
Array: 103
so.... why does installing a static finalizer work ok, but a dynamic one leaks memory!?
i'm nice and confused now.
a @ http://codeforpeople.com/
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama
I'd say you picked the wrong class for your tests - apparently a
lambda uses an array somehow. This is what I see from the modified
script (attached):
10:38:52 /cygdrive/c/Temp
$ ./a.rb leak
a.rb (696 Bytes)
2008/1/9, ara howard <ara.t.howard@gmail.com>:
so.... why does installing a static finalizer work ok, but a dynamic
one leaks memory!?i'm nice and confused now.
---
Foo: 0
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
10:39:04 /cygdrive/c/Temp
$ ./a.rb noleak
---
Foo: 0
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
---
Foo: 1
10:39:08 /cygdrive/c/Temp
$ ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin]
Kind regards
robert
--
use.inject do |as, often| as.you_can - without end
so.... why does installing a static finalizer work ok, but a dynamic
one leaks memory!?
My guess is that the lambda is keeping the scope of the new invocation
around, which includes a reference to the newly created array.
Would it be a better test not to use Array (instances of which seems
to be created also by ObjectSpace / YAML)? Ex:
class Foo ; end
7.times {
GC.start
count = ObjectSpace.each_object(Foo)
p "Count: #{count}"
Foo.new
}
-Dan
Here's an even simpler example. You don't need anything but the
following to demonstrate the problem.
class Foo
Finalizer = lambda{}
def initialize
ObjectSpace.define_finalizer(self,Finalizer)
end
end
def test
10.times do
GC.start
count = ObjectSpace.each_object(Foo) {}
p "Count: #{count}"
Foo.new
end
end
test # -> no leak
"Count: 0"
"Count: 1"
"Count: 2"
"Count: 2"
"Count: 2"
etc.
Now, re-open Foo and add an inline finalizer.
class Foo
def initialize
ObjectSpace.define_finalizer(self,lambda{})
end
end
test # -> now it leaks
"Count: 1"
"Count: 2"
"Count: 3"
"Count: 4"
etc.
I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.
Dan
---
dev.zeraweb.com
hmmm. doing this in irb suggests not:
list =
GC.start; p ObjectSpace.each_object(Array){}; list << lambda{}
the number of Array's will remain static. hrrrmmm....
On Jan 9, 2008, at 2:41 AM, Robert Klemme wrote:
'd say you picked the wrong class for your tests - apparently a
lambda uses an array somehow. This is what I see from the modified
script (attached):
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama
i think it's actually some strange interaction with yaml. check this out:
cfp2:~ > cat a.rb
class Class
def finalizer
lambda{}
end
def new *a, &b
object = allocate
object.send :initialize, *a, &b
object
ensure
ObjectSpace.define_finalizer object, finalizer
end
end
class Foo; end
class Bar < Foo; end
c = Array
if ARGV.detect{|arg| arg["leak"]}
require "yaml"
7.times {
GC.start
y c.name => ObjectSpace.each_object(c){}
c.new
}
else
7.times {
GC.start
puts "---"
puts "#{ c.name }: #{ ObjectSpace.each_object(c){} }"
c.new
}
end
cfp2:~ > ruby a.rb
On Jan 9, 2008, at 9:39 AM, dan yoder wrote:
I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.
---
Array: 6
---
Array: 11
---
Array: 14
---
Array: 18
---
Array: 20
---
Array: 20
---
Array: 20
cfp2:~ > ruby a.rb leak
---
Array: 21
---
Array: 38
---
Array: 54
---
Array: 67
---
Array: 79
---
Array: 91
---
Array: 103
a @ http://codeforpeople.com/
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama
> I realize the scope of the lambda invocation is different in this
> example, but since the behavior is so similar, I thought it likely
> pointed to the same underlying issue.i think it's actually some strange interaction with yaml. check this
out:
...
if ARGV.detect{|arg| arg["leak"]}
require "yaml"
7.times {
GC.start
y c.name => ObjectSpace.each_object(c){}
c.new
}
else
7.times {
GC.start
puts "---"
puts "#{ c.name }: #{ ObjectSpace.each_object(c){} }"
c.new
}
endcfp2:~ > ruby a.rb
---
Array: 6
cfp2:~ > ruby a.rb leak
---
Array: 21
Not sure how you got there Ara, I don't see where the OP ever mentioned YAML.
I think the key is where the lambda is created. The lambda is
capturing the binding.
In the first case the lambda is being created in the bindig context of
the class, and in particular self is the class.
In the second case, the lambda is being created in the binding context
of the new instance, and self is that new instance, so the lambda in
the finalizer is hanging on to it.
That's my theory anyway.
On Jan 9, 2008 11:45 AM, ara howard <ara.t.howard@gmail.com> wrote:
On Jan 9, 2008, at 9:39 AM, dan yoder wrote:
--
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.i think it's actually some strange interaction with yaml. check this
out:..
if ARGV.detect{|arg| arg["leak"]}
require "yaml"
7.times {
GC.start
y c.name => ObjectSpace.each_object(c){}
c.new
}
else
7.times {
GC.start
puts "---"
puts "#{ c.name }: #{ ObjectSpace.each_object(c){} }"
c.new
}
endcfp2:~ > ruby a.rb
---
Array: 6cfp2:~ > ruby a.rb leak
---
Array: 21Not sure how you got there Ara, I don't see where the OP ever mentioned YAML.
Rick, OP == ara => true.
I think the key is where the lambda is created. The lambda is
capturing the binding.
In his revisited example (msg id F36A72A4-6216-4251-BF98-D231597C7B0D@gmail.com) the non leaky finalizer had self bound to Class and the non leaky to the particular class instance. I believe Ara's confusion stems from the question how a class instance can keep instances in memory.
In the first case the lambda is being created in the bindig context of
the class, and in particular self is the class.In the second case, the lambda is being created in the binding context
of the new instance, and self is that new instance, so the lambda in
the finalizer is hanging on to it.
That's not true for the posting I mentioned above and also not for the last one. There was only one finalizer but one branch used yaml while the other did not.
Kind regards
robert
On 09.01.2008 19:26, Rick DeNatale wrote:
On Jan 9, 2008 11:45 AM, ara howard <ara.t.howard@gmail.com> wrote:
On Jan 9, 2008, at 9:39 AM, dan yoder wrote:
Ah, missed that.
Anyway ara re-asked the question on ruby-core and Matz answered him
pretty much the same way I did.
On Jan 9, 2008 1:39 PM, Robert Klemme <shortcutter@googlemail.com> wrote:
On 09.01.2008 19:26, Rick DeNatale wrote:
> On Jan 9, 2008 11:45 AM, ara howard <ara.t.howard@gmail.com> wrote:
>> On Jan 9, 2008, at 9:39 AM, dan yoder wrote:
>>
>>> I realize the scope of the lambda invocation is different in this
>>> example, but since the behavior is so similar, I thought it likely
>>> pointed to the same underlying issue.
>> i think it's actually some strange interaction with yaml. check this
>> out:
> ..
>> if ARGV.detect{|arg| arg["leak"]}
>> require "yaml"
>> 7.times {
>> GC.start
>> y c.name => ObjectSpace.each_object(c){}
>> c.new
>> }
>> else
>> 7.times {
>> GC.start
>> puts "---"
>> puts "#{ c.name }: #{ ObjectSpace.each_object(c){} }"
>> c.new
>> }
>> end
>>
>> cfp2:~ > ruby a.rb
>> ---
>> Array: 6
>
>> cfp2:~ > ruby a.rb leak
>> ---
>> Array: 21
>
> Not sure how you got there Ara, I don't see where the OP ever mentioned YAML.Rick, OP == ara => true.
> I think the key is where the lambda is created. The lambda is
> capturing the binding.In his revisited example (msg id
F36A72A4-6216-4251-BF98-D231597C7B0D@gmail.com) the non leaky finalizer
had self bound to Class and the non leaky to the particular class
instance. I believe Ara's confusion stems from the question how a class
instance can keep instances in memory.> In the first case the lambda is being created in the bindig context of
> the class, and in particular self is the class.
>
> In the second case, the lambda is being created in the binding context
> of the new instance, and self is that new instance, so the lambda in
> the finalizer is hanging on to it.That's not true for the posting I mentioned above and also not for the
last one. There was only one finalizer but one branch used yaml while
the other did not.Kind regards
robert
--
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
sort of - i still don't see how
1) the 'object' ended up being bound. it was a local var of another function. makes no sense.
2) why this happened for only say, 1 of 100000 objects
i still think it's a bug, but a have a work around - see dike announcement.
On Jan 9, 2008, at 1:42 PM, Rick DeNatale wrote:
Ah, missed that.
Anyway ara re-asked the question on ruby-core and Matz answered him
pretty much the same way I did.
--
it is not enough to be compassionate. you must act.
h.h. the 14th dalai lama
I think that this is the code you are talking about right?
class Class
Finalizer = lambda { }
def leak_free_finalizer
Finalizer
end
def leaky_finalizer
# What's self here? It's the same as for the finalizer method
# which
lambda{}
end
def finalizer
%r/leak/ =~ ARGV.first ? leaky_finalizer : leak_free_finalizer
end
def new *a, &b
object = allocate
object.send :initialize, *a, &b
object
ensure
ObjectSpace.define_finalizer object, finalizer
end
end
Looking at eval.c, it looks like lambda actually copies information
from the invocation stack, not just from the current frame. In the
leaky finalizer case we have the following on the stack when
leaky_finalizer is called, with the binding represented in hash
notation.
leaky_finalizer :self => Class
finalizer :self => Class
new :self=> Class, :object => the new Array instance
code which called Array.new
The leak free finalizer lambda was created once at a time when no
instance to be finalized was on the stack.
On Jan 9, 2008 3:46 PM, ara howard <ara.t.howard@gmail.com> wrote:
On Jan 9, 2008, at 1:42 PM, Rick DeNatale wrote:
> Ah, missed that.
>
> Anyway ara re-asked the question on ruby-core and Matz answered him
> pretty much the same way I did.sort of - i still don't see how
1) the 'object' ended up being bound. it was a local var of another
function. makes no sense.
--
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
yeah i think that may be true - but it doesn't make sense. the
def leaky_finalizer
lambda{}
end
is the current paradigm for preventing lambdas from enclosing a reference to an object. what you are saying is that this call encloses a local variable from another (the calling in the this case) function.
how would that not be a bug? why enclose a variable that cannot possible be reached in the code ran? this seems, to me just like this code
void * leak (){ return(&malloc(42)) }
i just cannot image why lambda would crawl up the stack outside the current function. if that is true then they are useless and *any* invocation means every object in memory at the time of creation can never be freed while that lambda exists. doesn't that seem excessive? also i use tons of lambdas in code that does not leak so this just seems impossible.
nevertheless you may be right!
cheers.
On Jan 9, 2008, at 2:52 PM, Rick DeNatale wrote:
Looking at eval.c, it looks like lambda actually copies information
from the invocation stack, not just from the current frame. In the
leaky finalizer case we have the following on the stack when
leaky_finalizer is called, with the binding represented in hash
notation.leaky_finalizer :self => Class
finalizer :self => Class
new :self=> Class, :object => the new Array instance
code which called Array.newThe leak free finalizer lambda was created once at a time when no
instance to be finalized was on the stack.
--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama
> Looking at eval.c, it looks like lambda actually copies information
> from the invocation stack, not just from the current frame. In the
> leaky finalizer case we have the following on the stack when
> leaky_finalizer is called, with the binding represented in hash
> notation.
>
> leaky_finalizer :self => Class
> finalizer :self => Class
> new :self=> Class, :object => the new Array
> instance
> code which called Array.new
>
> The leak free finalizer lambda was created once at a time when no
> instance to be finalized was on the stack.yeah i think that may be true - but it doesn't make sense. the
def leaky_finalizer
lambda{}
endis the current paradigm for preventing lambdas from enclosing a
reference to an object. what you are saying is that this call
encloses a local variable from another (the calling in the this case)
function.
Well I'm positing that based on a rather superficial read of eval.c.
how would that not be a bug? why enclose a variable that cannot
possible be reached in the code ran?
Well I suspect that it's because no real analysis is done of what's
inside the block when a proc is created, so the assumption is that the
entire binding is needed. The Smalltalk compilers I recall would
produce different types of block objects depending on whether or not
the block contained references to variables outside the block, and/or
contained a return.
Now why it goes back down the stack, if it indeed does, I'm not sure.
Perhaps it has something to do with the lambda vs. Proc.new
differences. I think that Proc.new and lambda/proc both use the
proc_alloc function where this seems to be happening.
In any event this is probably more a topic for ruby-core so I'm
cross-posting this reply there.
this seems, to me just like this
codevoid * leak (){ return(&malloc(42)) }
i just cannot image why lambda would crawl up the stack outside the
current function. if that is true then they are useless and *any*
invocation means every object in memory at the time of creation can
never be freed while that lambda exists. doesn't that seem
excessive? also i use tons of lambdas in code that does not leak so
this just seems impossible.nevertheless you may be right!
Or not <G>
On Jan 9, 2008 5:18 PM, ara howard <ara.t.howard@gmail.com> wrote:
On Jan 9, 2008, at 2:52 PM, Rick DeNatale wrote:
--
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
Just trying to answer the question if it is a bug by making a minimum
version of the leaking version (and correcting Ara's terrible bug how
to write 7 
and running it with 1.8 and 1.9
591/92 > cat leak.rb
# vim: sw=2 ts=2 ft=ruby expandtab tw=0 nu syn:
#
class Foo
def initialize
ObjectSpace.define_finalizer self, lambda{}
end
end
(42/6).times {
GC.start
p "Foo" => ObjectSpace.each_object(Foo){}
Foo.new
}
robert@roma:~/log/ruby/theory 13:19:00
592/93 > ruby leak.rb
{"Foo"=>0}
{"Foo"=>1}
{"Foo"=>2}
{"Foo"=>3}
{"Foo"=>4}
{"Foo"=>5}
{"Foo"=>6}
robert@roma:~/log/ruby/theory 13:19:06
593/94 > ruby1.9 leak.rb
{"Foo"=>0}
{"Foo"=>1}
{"Foo"=>1}
{"Foo"=>1}
{"Foo"=>1}
{"Foo"=>1}
{"Foo"=>1}
I do not know if this is good enough to say it is a bug in 1.8, but I
would somehow suspect so.
Cheers
Robert
--
http://ruby-smalltalk.blogspot.com/
---
Whereof one cannot speak, thereof one must be silent.
Ludwig Wittgenstein
<snip>
nice! 
so i posted something like this over on ruby-core, which i'll add for posterity:
"
to add a note to the end of the thread the fix to my problem was essentially this
class Class
New = instance_method :new
Objects = Hash.new
Destroy = lambda{|object_id| Objects.delete object_id}
def new *a, &b
object = allocate
Objects[object.object_id] = caller
object.send :initialize *a, &b
object
ensure
ObjectSpace.define_finalizer object, Destroy
end
end
and that
class Class
def destroy
lambda{}
end
....
ObjectSpace.define_finalizer object, destroy
....
end
perhaps i have not explained this adequately, but i still feel this is a bug. the 'self' that is enclosed is *never* 'object' and that self has no reference, save a local variable in another function, that refers to 'object'.
in any case i have a workaround and dike.rb is better than ever (see [ANN] on ruby-talk) so thanks all for the input!
"
On Jan 10, 2008, at 5:26 AM, Robert Dober wrote:
Just trying to answer the question if it is a bug by making a minimum
version of the leaking version (and correcting Ara's terrible bug how
to write 7
591/92 > cat leak.rb
# vim: sw=2 ts=2 ft=ruby expandtab tw=0 nu syn:
#
class Foo
def initialize
ObjectSpace.define_finalizer self, lambda{}
end
end(42/6).times {
GC.start
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama