Using lambda/Proc can prevent a lot of garbage collection

ruby-talk
1

Does anybody else think it is a serious issue that a Proc holds
references to all variables (including self) where it was
created (through a block)? These references are held through
Proc#binding. Although this does add some useful capability on
occasion, this causes excess memory to be used (worst case
could be a leaky program). Is this worth it? Personally, I
think not. I'd propose that Proc#binding return something that
only have access to variables referenced in the Proc. Even the
self from the defining context would be prohibited if not
referenced (possibly implicitly) in the Proc. Another option
would be to make these references weak so that they won't
prevent GC and those references would disappear if the
referenced object (including the variables/variable-table) is
GCed.

Comments?

···

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

2

Does anybody else think it is a serious issue that a Proc holds
references to all variables (including self) where it was
created (through a block)? These references are held through
Proc#binding. Although this does add some useful capability on
occasion, this causes excess memory to be used (worst case
could be a leaky program). Is this worth it? Personally, I
think not. I'd propose that Proc#binding return something that
only have access to variables referenced in the Proc. Even the
self from the defining context would be prohibited if not
referenced (possibly implicitly) in the Proc. Another option
would be to make these references weak so that they won't
prevent GC and those references would disappear if the
referenced object (including the variables/variable-table) is
GCed.

Which variables does a proc reference, when it uses eval, looks things
up via symbols, etc, etc?

You can't know in advance.

For an editor (or irb/completion) a best effort approach is OK, for a
binding, it is not OK.

+--- Kero ------------------------- kero@chello@nl ---+

all the meaningless and empty words I spoke |
                      Promises -- The Cranberries |

+--- M38c --- http://members.chello.nl/k.vangelder ---+

3

Here is an example where a Proc can cause a memory leak:

ruby -e '
n=2**13;squares=(1..n).map{|i|a=(1..i).to_a;lambda{i*i}};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 169312 kB

You wouldn't expect each of these lambda's to need the local
"a", but the binding holds it. Right now, the programmer needs
to specifically think about this and clear these unexpected
object references by hand:

ruby -e ' n=2**13;squares =
(1..n).map{|i|a=(1..i).to_a;f=lambda{i*i};a=nil;f};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 10644 kB

To me, it seems kind of silly for the programmer to have to
worry about this level of detail. I'd expect most Proc's to
not need access to all variables in the defining context.

Here are the solutions I see:

1. Let the ruby programmer worry about it. They should assign
a variable to nil when a Proc has access to it and they are
done needing the variable.

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

4. Provide an additional facility for generating a Proc-like
object that doesn't have full variable access to the
surrounding context like a Proc has. This could be an Proc
method that generates a new Proc-like object, a Proc method
modifies self in-place, a different block syntax, or something
else.

Personally, I think #2 or #3 should be done, because it is
automatic, but this will could cause code that does eval within
the block or from Proc#binding to break - as Kero said.

···

--- Kero <kero@chello.single-dot.nl> wrote:

> Does anybody else think it is a serious issue that a Proc
holds
> references to all variables (including self) where it was
> created (through a block)? These references are held
through
> Proc#binding. Although this does add some useful
capability on
> occasion, this causes excess memory to be used (worst case
> could be a leaky program). Is this worth it? Personally,
I
> think not. I'd propose that Proc#binding return something
that
> only have access to variables referenced in the Proc. Even
the
> self from the defining context would be prohibited if not
> referenced (possibly implicitly) in the Proc. Another
option
> would be to make these references weak so that they won't
> prevent GC and those references would disappear if the
> referenced object (including the variables/variable-table)
is
> GCed.

Which variables does a proc reference, when it uses eval,
looks things
up via symbols, etc, etc?

You can't know in advance.

For an editor (or irb/completion) a best effort approach is
OK, for a
binding, it is not OK.

__________________________________
Yahoo! Music Unlimited
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/

4

I believe that detecting references at compile time is tricky in a dynamic language. Is there not a way in Ruby to retrieve a local variable whose name is stored in another string? (Perhaps there isn't.

···

On Oct 9, 2005, at 8:26 AM, Eric Mahurin wrote:

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

5

Hi --

Here is an example where a Proc can cause a memory leak:

ruby -e '
n=2**13;squares=(1..n).map{|i|a=(1..i).to_a;lambda{i*i}};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 169312 kB

You wouldn't expect each of these lambda's to need the local
"a", but the binding holds it. Right now, the programmer needs
to specifically think about this and clear these unexpected
object references by hand:

ruby -e ' n=2**13;squares =
(1..n).map{|i|a=(1..i).to_a;f=lambda{i*i};a=nil;f};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 10644 kB

That's an interesting illustration, but I don't think I'd call it a
memory leak, since it's working as advertised. (At least, "memory
leak" to me implies something going wrong under the hood, so to
speak.)

To me, it seems kind of silly for the programmer to have to
worry about this level of detail. I'd expect most Proc's to
not need access to all variables in the defining context.

Here are the solutions I see:

1. Let the ruby programmer worry about it. They should assign
a variable to nil when a Proc has access to it and they are
done needing the variable.

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

4. Provide an additional facility for generating a Proc-like
object that doesn't have full variable access to the
surrounding context like a Proc has. This could be an Proc
method that generates a new Proc-like object, a Proc method
modifies self in-place, a different block syntax, or something
else.

Nooooo...please... not *another* Proc/proc/lambda/block/method-like
object :slight_smile:

Personally, I think #2 or #3 should be done, because it is
automatic, but this will could cause code that does eval within
the block or from Proc#binding to break - as Kero said.

It would be great if such things could be optimized away, but I just
can't reconcile doing it by "stingy" binding at the expense of the
dynamic techniques normally available. Maybe we all just have to be
aware of the fact that returning a closure means, in a sense, not
returning.

David

···

On Sun, 9 Oct 2005, Eric Mahurin wrote:

--
David A. Black
dblack@wobblini.net

6

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

  [1,2,3].each{|x| p x |}

The final '|' making it a closed block.

T.

7

David A. Black wrote:

Nooooo...please... not *another* Proc/proc/lambda/block/method-like
object :slight_smile:

Why not? More the merrier! :wink: Still think they could be reduced as a
matter of difference in internal state of the same class rather than
wholly separate classes.

T.

8

"Trans" <transfire@gmail.com> writes:

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

  [1,2,3].each{|x| p x |}

The final '|' making it a closed block.

*This* will be really fun to parse. :stuck_out_tongue:

···

T.

--
Christian Neukirchen <chneukirchen@gmail.com> http://chneukirchen.org

9

A closure should only enclose the variables needed for its execution. This is actually not very difficult but unfortunately eval('a') prevents Ruby from excluding unbound variables in closures.

This is a (unfortunate IMO) feature of Ruby.

···

On Oct 9, 2005, at 8:20 AM, David A. Black wrote:

Hi --

On Sun, 9 Oct 2005, Eric Mahurin wrote:

Here is an example where a Proc can cause a memory leak:

ruby -e '
n=2**13;squares=(1..n).map{|i|a=(1..i).to_a;lambda{i*i}};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 169312 kB

You wouldn't expect each of these lambda's to need the local
"a", but the binding holds it. Right now, the programmer needs
to specifically think about this and clear these unexpected
object references by hand:

ruby -e ' n=2**13;squares =
(1..n).map{|i|a=(1..i).to_a;f=lambda{i*i};a=nil;f};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 10644 kB

That's an interesting illustration, but I don't think I'd call it a
memory leak, since it's working as advertised. (At least, "memory
leak" to me implies something going wrong under the hood, so to
speak.)

10

That won't make eval work in a closure like it should.

···

On Oct 9, 2005, at 9:06 AM, Trans wrote:

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

  [1,2,3].each{|x| p x |}

The final '|' making it a closed block.

11

Compile-time meaning compile-time of the block - when the block
is created. I think of this as at compile-time, but with the
ruby interpreter it may be at run-time. I see no reason why it
shouldn't be able to determine what variables a block accesses
when it is created - with the exception of eval in the block
and eval using Proc#binding.

···

--- Gavin Kistner <gavin@refinery.com> wrote:

On Oct 9, 2005, at 8:26 AM, Eric Mahurin wrote:
> 2. Proc#binding should give a binding that only has
variables
> that the block accesses (determined when the compiled).
>
> 3. Proc#binding should hold weak references to variables
that
> the block doesn't access (at compile-time).

I believe that detecting references at compile time is tricky
in a
dynamic language. Is there not a way in Ruby to retrieve a
local
variable whose name is stored in another string? (Perhaps
there isn't.

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around

12

Hi --

> Here is an example where a Proc can cause a memory leak:
>
> ruby -e '
> n=2**13;squares=(1..n).map{|i|a=(1..i).to_a;lambda{i*i}};
>

IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'

> VmSize: 169312 kB
>
> You wouldn't expect each of these lambda's to need the
local
> "a", but the binding holds it. Right now, the programmer
needs
> to specifically think about this and clear these unexpected
> object references by hand:
>
> ruby -e ' n=2**13;squares =
> (1..n).map{|i|a=(1..i).to_a;f=lambda{i*i};a=nil;f};
>

IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'

> VmSize: 10644 kB

That's an interesting illustration, but I don't think I'd
call it a
memory leak, since it's working as advertised. (At least,
"memory
leak" to me implies something going wrong under the hood, so
to
speak.)

Well, C's malloc and free also work as advertised. And you can
easily make a C program that has a memory leak if you don't
pair these properly. You can blame it on the C
language/library or you can blame it on the C programmer, but
either way there is a memory leak when you don't pair these
properly. I see the above case as the same. Here is the
definition of "memory leak" I found on
http://www.webopedia.com/TERM/M/memory_leak.html :

"A bug in a program that prevents it from freeing up memory
that it no longer needs"

In the first example above that is what is going on. The
program is not freeing memory it no longer needs. Again, you
can blame this on the ruby programmer or the ruby langauge.
Since this is usually unwanted and/or unexpected, I choose to
blame the language.

···

--- "David A. Black" <dblack@wobblini.net> wrote:

On Sun, 9 Oct 2005, Eric Mahurin wrote:

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around

13

Eric Hodel wrote:

···

On Oct 9, 2005, at 9:06 AM, Trans wrote:

> This seems essentially the same as the discussion on closed blocks.
> Probably a bad idea to get rif of current behaivor, but there are good
> reasons to add an additional for what you suggest. Of course, off the
> bat there is a notation problem. My latest thought:
>
> [1,2,3].each{|x| p x |}
>
> The final '|' making it a closed block.

That won't make eval work in a closure like it should.

Like it *should*? I think eval works like it should, it's just that
having eval means you can't be sure of what you can exclude or not. (Or
is there something else?) The above explicity excludes all, essentially
no closure --that's the idea anyway. Once again a good notation proves
a problem.

T.

14

Christian Neukirchen wrote:

"Trans" <transfire@gmail.com> writes:

> This seems essentially the same as the discussion on closed blocks.
> Probably a bad idea to get rif of current behaivor, but there are good
> reasons to add an additional for what you suggest. Of course, off the
> bat there is a notation problem. My latest thought:
>
> [1,2,3].each{|x| p x |}
>
> The final '|' making it a closed block.

*This* will be really fun to parse. :stuck_out_tongue:

Ugh, yea. Well, it sucked for other reasons anyway. Just tryng to churn
the pot.

T.

15

Eric Hodel wrote:

A closure should only enclose the variables needed for its execution.
This is actually not very difficult but unfortunately eval('a') prevents
Ruby from excluding unbound variables in closures.

This is a (unfortunate IMO) feature of Ruby.

Aren't there some useful hacks that depend on being able to get at vars
in the caller's scope (which need not be referenced in the block), using
the

  eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's context,
using something like this:

      def when_#{var} pattern=Object, &block
        observer_map = @#{var}__observer_map ||= ObserverMap.new
        if block
          observer = eval "self", block

to find out what object is observing a variable. But that's not as
objectionable as getting access to arbitrary local vars in that context.

Does anyone remember--is there is anything that uses the latter kind of
access (arb. local vars) that is more than just a cute hack? Is there
anything really useful we would lose without that behavior?

···

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

16

Hi --

Well, C's malloc and free also work as advertised. And you can
easily make a C program that has a memory leak if you don't
pair these properly. You can blame it on the C
language/library or you can blame it on the C programmer, but
either way there is a memory leak when you don't pair these
properly. I see the above case as the same. Here is the
definition of "memory leak" I found on
http://www.webopedia.com/TERM/M/memory_leak.html :

"A bug in a program that prevents it from freeing up memory
that it no longer needs"

In the first example above that is what is going on. The
program is not freeing memory it no longer needs. Again, you
can blame this on the ruby programmer or the ruby langauge.
Since this is usually unwanted and/or unexpected, I choose to
blame the language.

The problem, though, is that the need (or lack thereof) can't be
determined, isn't it? It's not even safe to assume that the presence
of "eval" means "don't free the memory", nor that its absence means
not to. I mean, it's unlikely that someone would do:

   alias :blah :eval
   x = <some huge thing>
   lambda { blah("x") }

but it's possible.

David

···

On Mon, 10 Oct 2005, Eric Mahurin wrote:

--
David A. Black
dblack@wobblini.net

17

Eric Mahurin wrote:

"A bug in a program that prevents it from freeing up memory
that it no longer needs"

Some have a narrower definition of memory leak: a block of memory that
has been allocated by a program, but which is no longer reachable from
any variables and is therefore unfreeable.

···

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

18

Speaking of eval, I've always wondered why ruby of all languages needed a string based eval. What can you do with a string eval that you can't do with something like:

eval(some_binding) { code }

some_binding of course would be optional just like currently.

with instance_variable_get() and instance_variable_get you dynamically get at ivars. with send and/or instance eval you can dynamically send messages. define_method blah blah... works just as well as eval "def blah blah". This is something thats always kind of irked me. And in my brave new world of evalless ruby, worse case scenario you write a string to a temporary file and load it. But I really don't believe thats necessary. Can anyone come up with a use for an eval taking a string that CAN'T be acheieved by an eval that takes a block? (well, irb I suppose). If nothing else I would appreciate it greatly if eval could take a block in addition to a string. I think this may have come off kind of like, so if I did I apologize in advance :wink:

···

On Oct 9, 2005, at 5:31 PM, Joel VanderWerf wrote:

Eric Hodel wrote:

A closure should only enclose the variables needed for its execution.
This is actually not very difficult but unfortunately eval('a') prevents
Ruby from excluding unbound variables in closures.

This is a (unfortunate IMO) feature of Ruby.

Aren't there some useful hacks that depend on being able to get at vars
in the caller's scope (which need not be referenced in the block), using
the

  eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's context,
using something like this:

      def when_#{var} pattern=Object, &block
        observer_map = @#{var}__observer_map ||= ObserverMap.new
        if block
          observer = eval "self", block

to find out what object is observing a variable. But that's not as
objectionable as getting access to arbitrary local vars in that context.

Does anyone remember--is there is anything that uses the latter kind of
access (arb. local vars) that is more than just a cute hack? Is there
anything really useful we would lose without that behavior?

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

19

I assume you mean "eval str, some_binding" and 'eval "self",
block.binding'.

I've used this before too. This is kind of a way to get the
Binding#of_caller. I was thinking of this case when I made
this suggestion:

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

So in the above case, "self" would still be available
regardless of whether the block referenced self in the code
because there is no way it would be freed yet (it is in the
context of the caller) and Proc#binding would still have this
weak reference. "self" is a special case "variable", but it
should apply to more generic variables too.

···

--- Joel VanderWerf <vjoel@path.berkeley.edu> wrote:

Eric Hodel wrote:

> A closure should only enclose the variables needed for its
execution.
> This is actually not very difficult but unfortunately
eval('a') prevents
> Ruby from excluding unbound variables in closures.
>
> This is a (unfortunate IMO) feature of Ruby.
>

Aren't there some useful hacks that depend on being able to
get at vars
in the caller's scope (which need not be referenced in the
block), using
the

  eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's
context,
using something like this:

      def when_#{var} pattern=Object, &block
        observer_map = @#{var}__observer_map ||=
ObserverMap.new
        if block
          observer = eval "self", block

to find out what object is observing a variable. But that's
not as
objectionable as getting access to arbitrary local vars in
that context.

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005

20

Unfortunately, eval can do things that non-eval-code can't. Little
things like methods that take blocks, easier class definitions, quick
and dirty singleton definitions, etc... These _can_ be solved w/o eval
(well most of them) but it requires more code, more trickery, and/or
more discipline.

With that said, I am all for a Ruby w/o a direct need for eval besides
a REPL. Even a keyword-less way to do everything in Ruby would be nice
(singletons, I'm looking at you).

Brian.

···

On 10/9/05, Logan Capaldo <logancapaldo@gmail.com> wrote:

On Oct 9, 2005, at 5:31 PM, Joel VanderWerf wrote:

> Eric Hodel wrote:
>
>
>> A closure should only enclose the variables needed for its execution.
>> This is actually not very difficult but unfortunately eval('a')
>> prevents
>> Ruby from excluding unbound variables in closures.
>>
>> This is a (unfortunate IMO) feature of Ruby.
>>
>>
>
> Aren't there some useful hacks that depend on being able to get at
> vars
> in the caller's scope (which need not be referenced in the block),
> using
> the
>
> eval str, some_proc
>
> construct?
>
> I use it in my observable lib to get the self of the block's context,
> using something like this:
>
> def when_#{var} pattern=Object, &block
> observer_map = @#{var}__observer_map ||= ObserverMap.new
> if block
> observer = eval "self", block
>
> to find out what object is observing a variable. But that's not as
> objectionable as getting access to arbitrary local vars in that
> context.
>
> Does anyone remember--is there is anything that uses the latter
> kind of
> access (arb. local vars) that is more than just a cute hack? Is there
> anything really useful we would lose without that behavior?
>
> --
> vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
>
>

Speaking of eval, I've always wondered why ruby of all languages
needed a string based eval. What can you do with a string eval that
you can't do with something like:

eval(some_binding) { code }

some_binding of course would be optional just like currently.

with instance_variable_get() and instance_variable_get you
dynamically get at ivars. with send and/or instance eval you can
dynamically send messages. define_method blah blah... works just as
well as eval "def blah blah". This is something thats always kind of
irked me. And in my brave new world of evalless ruby, worse case
scenario you write a string to a temporary file and load it. But I
really don't believe thats necessary. Can anyone come up with a use
for an eval taking a string that CAN'T be acheieved by an eval that
takes a block? (well, irb I suppose). If nothing else I would
appreciate it greatly if eval could take a block in addition to a
string. I think this may have come off kind of like, so if I did I
apologize in advance :wink: