[ANN] lazy.rb 0.9.5 -- transparent futures!

I'd like to announce a new version of lazy.rb -- this one offering
thread safety and transparent futures!

Here's the web site, complete with gem and tarball downloads, plus a bit
of documentation:

http://moonbase.rydia.net/software/lazy.rb/

Bug reports would be very, very welcome.

== What is lazy.rb?

lazy.rb is a library providing transparent lazy evaluation and futures
for Ruby. It provides a bag of clever tricks to help you avoid doing
expensive computations up front.

= Lazy Evaluation

Lazy evaluation simply refers to computations which are run on an
as-needed basis. For example:

x = promise { 3 + 5 }

Means that the block -- 3 + 5 -- won't actually be evaluated until
something tries to use x's value.

p x # => #<Lazy::Promise computation=#<Proc:...>>

# forces evaluation
p x * 3 # => 24

p x # => 8

You can also force evaluation using demand:

x = promise { 3 + 5 }

p x # => #<Lazy::Promise computation=#<Proc:...>>

# forces evaluation
p demand( x ) # => 8

p x # => 8

It's a bit silly for 3 + 5, but it's handy for more intensive
calculations. You can unconditionally promise a computation, yet only
pay for it if and when its result is actually used.

= Futures

Futures are blocks of code that are evaluated immediately, but in a
background thread.

x = future { 3 + 5 }

p x # => #<Lazy::Future computation=#<Proc:...>>

# You could do other stuff here while
# the computation ran in the background

# blocks until the background thread completes
p x * 3 # => 24

p x # => 8

Again, silly for 3 + 5 perhaps, but I'm sure you can see how this might
come in handy for more involved computations.

= Other stuff

lazy.rb also includes support for circular programming, where a
computation is passed its own result:

matryoshka = demand( promise { |result| [result] } )

p matryoshka # => [[...]]

p matryoshka.object_id # => -605506544

p matryoshka.first.object_id # => -605506544

p matryoshka.first.first.object_id # => -605506544

This works for both promises and futures, although it has the usual
limitations: if a computation tries to call methods on its own result,
it will diverge.

== What's new in 0.9.5?

- Optional support for multithreaded programs:

    require 'lazy/threadsafe' and you can safely use lazy evaluation in
    multithreaded programs.

- Futures:

    With thread support, it turned out that futures were really
    easy to implement atop promises -- just fire off a thread with the
    computation and return a promise to join the thread and grab its
    result. So I implemented that.

== What happened to lazy streams from 0.2?

I ditched the lazy streams API for now. It just wasn't working out.

== What next?

Except perhaps for lazy streams (which might end up becoming a separate
library), I think we're nearly feature-complete. Ideas and suggestions
are very welcome, though.

-mental

= Futures

Futures are blocks of code that are evaluated immediately, but in a
background thread.

x = future { 3 + 5 }

p x # => #<Lazy::Future computation=#<Proc:...>>

# You could do other stuff here while
# the computation ran in the background

# blocks until the background thread completes
p x * 3 # => 24

p x # => 8

Again, silly for 3 + 5 perhaps, but I'm sure you can see how this might
come in handy for more involved computations.

Wow, that is just super studly!

== What happened to lazy streams from 0.2?

I ditched the lazy streams API for now. It just wasn't working out.

Can you expand on what you mean by this? I'm working on a huge article about infinite streams for my blog and lazy.rb 0.2 was a big inspiration to me.

James Edward Gray II

···

On Feb 18, 2006, at 7:07 PM, MenTaLguY wrote:

MenTaLguY wrote:

I'd like to announce a new version of lazy.rb -- this one offering
thread safety and transparent futures!

Nice.

# forces evaluation
p x * 3 # => 24

What will it do for:
  p 3 * x

Thanks for sharing.

When I toyed with a proof-of-concept of Io's asynchronous messages in Ruby

  http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/Async.rdoc

Jim Weirich pointed out that methods later added to Object or Kernel will
no longer be "missing" in the Async (or Future) class. One work around is
to trap "method_added" up the chain, for example:

  http://www.siaris.net/cgi-bin/siwiki.pl?FeedBack/AsyncMessages

But then I discovered that methods added by including a module weren't
trapped by method_added -- so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async's parent to keep the proxy clean.

regards,
andrew

···

On Sun, 19 Feb 2006 10:07:12 +0900, MenTaLguY <mental@rydia.net> wrote:

--=-Nhoy1n60Sqvzvq/VUinT
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I'd like to announce a new version of lazy.rb -- this one offering
thread safety and transparent futures!

Here's the web site, complete with gem and tarball downloads, plus a bit
of documentation:

http://moonbase.rydia.net/software/lazy.rb/

Bug reports would be very, very welcome.

--
Andrew L. Johnson http://www.siaris.net/
      What have you done to the cat? It looks half-dead.
          -- Schroedinger's wife

Sorry for newbie question. I tried to install lazy only to fail:

[root@poseidon tmp]# ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[root@poseidon tmp]# ls -l lazy-0.9.5.gem
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...

lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]#

Any idea?

Minkoo Seo

Hi,
   
  I have a feature request (along with a quick-hack solution).
   
  Please ignore if this is already done or sounds crazy. :slight_smile:
   
  Problem: All the promises I make, cannot have any arguments ! !
   
  So something like this is what I have in mind:
   
  x = promise("CUSTOMERS") { |table_name| long_running_query(table_name)}
   
  Admitedly, in this one-liner it does not make much sense. But if the block had many many lines of code and referred to the block variable at multiple places, suddenly it would be mean more to me.
   
  I have attached my hackish solution without really thinking all the way thru.
  Thanks,
   
  -- Shashank

my_lazy.rb (4.21 KB)

···

MenTaLguY <mental@rydia.net> wrote:
  I'd like to announce a new version of lazy.rb -- this one offering
thread safety and transparent futures!

Here's the web site, complete with gem and tarball downloads, plus a bit
of documentation:

http://moonbase.rydia.net/software/lazy.rb/

Bug reports would be very, very welcome.

---------------------------------
Yahoo! Mail
Use Photomail to share photos without annoying attachments.

Hi mental. Thank you for the really nice library.

I'm sorry if this bothers you, but I have no idea what "circular
programming" is. Could you give me an example? Especially, what does
"matryoshka = demand( promise { |result| [result] } )" mean?

I also suggest that you should upload the description posted here to
your own website(http://moonbase.rydia.net/software/lazy.rb/), so that
people can find out what lazy.rb really is. IMHO, RDoc itself is not
sufficient.

- Minkoo Seo

If you suspected "the right thing" you'd probably be right:

irb(main):001:0> require 'lazy'
irb(main):002:0> x = promise { 5 + 3 }
=> #<Lazy::Promise computation=#<Proc:0x0035788c@(irb):2>>
irb(main):003:0> p 3 * x
24
=> nil

Quoting James Edward Gray II <james@grayproductions.net>:

> I ditched the lazy streams API for now. It just wasn't working
> out.

Can you expand on what you mean by this? I'm working on a huge
article about infinite streams for my blog and lazy.rb 0.2 was a
big inspiration to me.

Oh, mainly it was an aesthetic thing. It _worked_ fine.

I'd been fighting with the streams API to make it more Ruby-esque
and easier to use properly, but I finally punted on it to get this
release out the door quickly.

In retrospect, I probably should have just kept the 0.2 API for now.
I definitely want something better for 1.0, though.

-mental

Minkoo Seo wrote:

Sorry for newbie question. I tried to install lazy only to fail:

[root@poseidon tmp]# ruby -v
ruby 1.8.4 (2005-12-24) [i686-linux]
[root@poseidon tmp]# ls -l lazy-0.9.5.gem
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5

  ^^^^^^^^^^^^

The installatiof of the lazy software is OK ...

Installing RDoc documentation for lazy-0.9.5...

lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module

Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.

···

--
-- Jim Weirich

--
Posted via http://www.ruby-forum.com/\.

Hmm, good catch. Thanks!

I wonder if it's worth introducing a dependency on evil.rb?

-mental

···

On Tue, 2006-02-21 at 17:33 +0900, Andrew Johnson wrote:

When I toyed with a proof-of-concept of Io's asynchronous messages in Ruby

  http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/Async.rdoc

Jim Weirich pointed out that methods later added to Object or Kernel will
no longer be "missing" in the Async (or Future) class. One work around is
to trap "method_added" up the chain, for example:

  http://www.siaris.net/cgi-bin/siwiki.pl?FeedBack/AsyncMessages

But then I discovered that methods added by including a module weren't
trapped by method_added -- so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async's parent to keep the proxy clean.

Hmm. Would this work?

table_name = "CUSTOMERS"
x = promise { long_running_query(table_name) }

-mental

···

On Wed, 2006-02-22 at 14:14 -0800, Shashank Date wrote:

Problem: All the promises I make, cannot have any arguments ! !

So something like this is what I have in mind:

x = promise("CUSTOMERS") { |table_name|
long_running_query(table_name)}

Quoting "matt.smillie@gmail.com" <matt.smillie@gmail.com>:

If you suspected "the right thing" you'd probably be right:

irb(main):001:0> require 'lazy'
irb(main):002:0> x = promise { 5 + 3 }
=> #<Lazy::Promise computation=#<Proc:0x0035788c@(irb):2>>
irb(main):003:0> p 3 * x
24
=> nil

It's worth noting that this only works so well because of the very
nice coercion machinery Ruby has built around its numeric classes.

Ruby's NUM2INT (for example) will call .to_int on the promise when
FIXNUM_P fails, so everything Just Works(tm).

For other cases (e.g. promises returning file handles), you may find
you need to unwrap them explicitly with demand() to pass them to
certain methods.

Of course this is an issue only when passing promises to methods
implemented in C; lazy.rb does a very good job of faking out Ruby
otherwise (the boolean issue notwithstanding).

-mental

Hah! Awesome mental, I was going to be implementing something like
this for an Actor library that I'm going to start work on Real Soon
(tm). And, of course, you did it way better than I would have.

One question: How hard would it be to modify the way that exceptions
are handled to hold off raising the exception until the result was
requested? I'm thinking specifically of some non-deterministic
situations where you may request a value but never end up needing to
use it.

···

On 2/20/06, mental@rydia.net <mental@rydia.net> wrote:

Quoting James Edward Gray II <james@grayproductions.net>:

> > I ditched the lazy streams API for now. It just wasn't working
> > out.
>
> Can you expand on what you mean by this? I'm working on a huge
> article about infinite streams for my blog and lazy.rb 0.2 was a
> big inspiration to me.

Oh, mainly it was an aesthetic thing. It _worked_ fine.

I'd been fighting with the streams API to make it more Ruby-esque
and easier to use properly, but I finally punted on it to get this
release out the door quickly.

In retrospect, I probably should have just kept the 0.2 API for now.
I definitely want something better for 1.0, though.

-mental

--
-Dan Nugent

Still, no luck.

[root@poseidon tmp]# ll
합계 12
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...

lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]# irb
irb(main):001:0> require 'lazy'
LoadError: no such file to load -- lazy
        from (irb):1:in `require'
        from (irb):1
irb(main):002:0> require 'lazy/future'
LoadError: no such file to load -- lazy/future
        from (irb):2:in `require'
        from (irb):2
irb(main):003:0>

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/
합계 72
drwxr-xr-x 4 root root 4096 2월 12 07:32 actionmailer-1.1.5/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionpack-1.11.2/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionwebservice-1.0.0/
drwxr-xr-x 5 root root 4096 2월 12 07:31 activerecord-1.13.2/
drwxr-xr-x 3 root root 4096 2월 12 07:31 activesupport-1.2.5/
drwxr-xr-x 3 root root 4096 2월 22 00:42 lazy-0.9.5/
drwxr-xr-x 11 root root 4096 2월 12 07:32 rails-1.0.0/
drwxr-xr-x 6 root root 4096 2월 12 07:31 rake-0.7.0/
drwxr-xr-x 3 root root 4096 2월 12 07:30 sources-0.0.1/

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/1.8/ | grep lazy
[root@poseidon tmp]#

I've taken the liberty of posting installation problem to this post,
because this is the first time for me to install local *.gem file.
Can anybody tell me why this is happening? Isn't the lazy.rb file
supposed to be installed in /usr/local/lib/ruby/1.8 ?

- Minkoo Seo

DIVERGES is a nodoc'd constant which is used internally. If anyone can
find a way to avoid the RDoc warning, I'd really appreciate it...

-mental

···

On Tue, 2006-02-21 at 21:39 +0900, Jim Weirich wrote:

Its just that RDoc is complaining about something. I got the same error
on my system, but the RDoc looks ok, even with the error.

Oh, I don't think so. It would cut off some users, like me. :wink:

James Edward Gray II

···

On Feb 21, 2006, at 12:29 PM, MenTaLguY wrote:

On Tue, 2006-02-21 at 17:33 +0900, Andrew Johnson wrote:

When I toyed with a proof-of-concept of Io's asynchronous messages in Ruby

  http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/Async.rdoc

Jim Weirich pointed out that methods later added to Object or Kernel will
no longer be "missing" in the Async (or Future) class. One work around is
to trap "method_added" up the chain, for example:

  http://www.siaris.net/cgi-bin/siwiki.pl?FeedBack/AsyncMessages

But then I discovered that methods added by including a module weren't
trapped by method_added -- so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async's parent to keep the proxy clean.

Hmm, good catch. Thanks!

I wonder if it's worth introducing a dependency on evil.rb?

MenTaLguY wrote:

···

On Tue, 2006-02-21 at 17:33 +0900, Andrew Johnson wrote:

But then I discovered that methods added by including a module weren't
trapped by method_added -- so in the end, a KernellessObject (from evil.rb)
was needed (same link as above) as Async's parent to keep the proxy clean.

Hmm, good catch. Thanks!

I wonder if it's worth introducing a dependency on evil.rb?

The BlankSlate class in Builder handles this without resorting to the
"evil" that lies in the heart of evil.rb. And the CVS head version of
BlankSlate also handles the module hole Andrew mentioned earlier (I
think ... I just now updated it).4

--
-- Jim Weirich

--
Posted via http://www.ruby-forum.com/\.

Haven't followed this thread all the way, but I'm assuming you must have
already tried:

  $ irb
  require 'rubygems'
  require 'lazy'

Or alternatively passing -rubygems as an option to IRB.

···

On Wed, 2006-02-22 at 00:58 +0900, Minkoo Seo wrote:

Still, no luck.

[root@poseidon tmp]# ll
합계 12
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...

lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]# irb
irb(main):001:0> require 'lazy'
LoadError: no such file to load -- lazy
        from (irb):1:in `require'
        from (irb):1
irb(main):002:0> require 'lazy/future'
LoadError: no such file to load -- lazy/future
        from (irb):2:in `require'
        from (irb):2
irb(main):003:0>

--
Ross Bamford - rosco@roscopeco.REMOVE.co.uk

require 'rubygems'
require 'lazy'

···

On Feb 21, 2006, at 10:58 AM, Minkoo Seo wrote:

Still, no luck.

[root@poseidon tmp]# ll
합계 12
-rw-r--r-- 1 mkseo users 6656 2월 19 09:31 lazy-0.9.5.gem
[root@poseidon tmp]# gem install lazy-0.9.5.gem
Attempting local installation of 'lazy-0.9.5.gem'
Successfully installed lazy, version 0.9.5
Installing RDoc documentation for lazy-0.9.5...

lazy.rb:60:22: Couldn't find DIVERGES. Assuming it's a module
[root@poseidon tmp]# irb
irb(main):001:0> require 'lazy'
LoadError: no such file to load -- lazy
        from (irb):1:in `require'
        from (irb):1
irb(main):002:0> require 'lazy/future'
LoadError: no such file to load -- lazy/future
        from (irb):2:in `require'
        from (irb):2
irb(main):003:0>

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/gems/1.8/gems/
합계 72
drwxr-xr-x 4 root root 4096 2월 12 07:32 actionmailer-1.1.5/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionpack-1.11.2/
drwxr-xr-x 5 root root 4096 2월 12 07:32 actionwebservice-1.0.0/
drwxr-xr-x 5 root root 4096 2월 12 07:31 activerecord-1.13.2/
drwxr-xr-x 3 root root 4096 2월 12 07:31 activesupport-1.2.5/
drwxr-xr-x 3 root root 4096 2월 22 00:42 lazy-0.9.5/
drwxr-xr-x 11 root root 4096 2월 12 07:32 rails-1.0.0/
drwxr-xr-x 6 root root 4096 2월 12 07:31 rake-0.7.0/
drwxr-xr-x 3 root root 4096 2월 12 07:30 sources-0.0.1/

[root@poseidon tmp]# ls -l /usr/local/lib/ruby/1.8/ | grep lazy
[root@poseidon tmp]#

I've taken the liberty of posting installation problem to this post,
because this is the first time for me to install local *.gem file.
Can anybody tell me why this is happening? Isn't the lazy.rb file
supposed to be installed in /usr/local/lib/ruby/1.8 ?

- Minkoo Seo