[ANN] slave-1.1.0

SYNOPSIS

   the Slave class forks a process and starts a drb server in the child using
   any object as the server. the process is detached so it is not required
   (nor possible) to wait on the child pid. a Heartbeat is set up between the
   parent and child processes so that the child will exit of the parent exits
   for any reason - preventing orphaned slaves from running indefinitely. the
   purpose of Slaves is to be able to easily set up a collection of objects
   communicating via drb protocols instead of having to use IPC.

   typical usage:

     slave = Slave::new{ AnyObject.new }

     slave.object #=> handle on drb object
     slave.uri #=> uri of the drb object
     slave.socket #=> unix domain socket path for drb object
     slave.psname #=> title shown in ps/top

     object = slave.object

     value = object.any_method #=> use the object normally

   slaves may be configured via the environment, the Slave class, or via the
   ctor for object itself. attributes which may be configured include

     * object : specify the slave object. otherwise block value is used.
     * socket_creation_attempts : specify how many attempts to create a unix domain socket will be made
     * debug : turn on some logging to STDERR
     * psname : specify the name that will appear in 'top' ($0)
     * at_exit : specify a lambda to be called in the *parent* when the child dies
     * dumped : specify that the slave object should *not* be DRbUndumped (default is DRbUndumped)
     * threadsafe : wrap the slave object with ThreadSafe to implement gross thread safety

URIS

   http://rubyforge.org/projects/codeforpeople/
   http://codeforpeople.com/lib/ruby/slave

HISTORY

   1.1.0:
     - replaced HeartBeat class with LifeLine.

     - __HUGE__ cleanup of file descriptor/fork management with tons of help
       from skaar and ezra. thanks guys!

     - introduced Slave.object method used to return any object directory from
       a child process. see samples/g.rb.

     - indroduced keyword to automatically make slave objects threadsafe.
       remember that your slave object must be threadsafe because they are
       being served via DRb!!!

SAMPLES

   <========< samples/a.rb >========>

   ~ > cat samples/a.rb

     require 'slave'

···

#
     # simple usage is simply to stand up a server object as a slave. you do not
     # need to wait for the server, join it, etc. it will die when the parent
     # process dies - even under 'kill -9' conditions
     #
       class Server
         def add_two n
           n + 2
         end
       end

       slave = Slave.new :object => Server.new

       server = slave.object
       p server.add_two(40) #=> 42

       slave.shutdown

   ~ > ruby samples/a.rb

     42

   <========< samples/b.rb >========>

   ~ > cat samples/b.rb

     require 'slave'
     #
     # if certain operations need to take place in the child only a block can be
     # used
     #
       class Server
         def connect_to_db
           "we only want to do this in the child process!"
           @connection = :postgresql
         end
         attr :connection
       end

       slave = Slave.new('object' => Server.new){|s| s.connect_to_db}

       server = slave.object

       p server.connection #=> :postgresql
     #
     # errors in the child are detected and raised in the parent
     #
       slave = Slave.new('object' => Server.new){|s| s.typo} #=> raises an error!

   ~ > ruby samples/b.rb

     :postgresql
     ./lib/slave.rb:460:in `initialize': undefined method `typo' for #<Server:0xb7565694> (NoMethodError)
       from samples/b.rb:22:in `new'
       from samples/b.rb:22

   <========< samples/c.rb >========>

   ~ > cat samples/c.rb

     require 'slave'
     #
     # if no slave object is given the block itself is used to contruct it
     #
       class Server
         def initialize
           "this is run only in the child"
           @pid = Process.pid
         end
         attr 'pid'
       end

       slave = Slave.new{ Server.new }
       server = slave.object

       p Process.pid
       p server.pid # not going to be the same as parents!
     #
     # errors are still detected though
     #
       slave = Slave.new{ fubar } # raises error in parent

   ~ > ruby samples/c.rb

     14387
     14388
     ./lib/slave.rb:460:in `initialize': undefined local variable or method `fubar' for main:Object (NameError)
       from samples/c.rb:21:in `new'
       from samples/c.rb:21

   <========< samples/d.rb >========>

   ~ > cat samples/d.rb

     require 'slave'
     #
     # at_exit hanlders are handled correctly in both child and parent
     #
       at_exit{ p 'parent' }
       slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }
     #
     # this will print 'child', then 'parent'
     #

   ~ > ruby samples/d.rb

     "child"
     "parent"

   <========< samples/e.rb >========>

   ~ > cat samples/e.rb

     require 'slave'
     #
     # slaves never outlive their parent. if the parent exits, even under kill -9,
     # the child will die.
     #
       slave = Slave.new{ at_exit{ p 'child' }; 'the server is this string' }

       Process.kill brutal=9, the_parent_pid=Process.pid
     #
     # even though parent dies a nasty death the child will still print 'child'
     #

   ~ > ruby samples/e.rb

     "child"

   <========< samples/f.rb >========>

   ~ > cat samples/f.rb

     require 'slave'
     #
     # slaves created previously are visible to newly created slaves - in this
     # example the child process of slave_a communicates directly with the child
     # process of slave_a
     #
       slave_a = Slave.new{ Array.new }
       slave_b = Slave.new{ slave_a.object }

       a, b = slave_b.object, slave_a.object

       b << 42
       puts a #=> 42

   ~ > ruby samples/f.rb

     42

   <========< samples/g.rb >========>

   ~ > cat samples/g.rb

     require 'slave'
     #
     # Slave.object can used when you want to construct an object in another
     # process. in otherwords you want to fork a process and retrieve a single
     # returned object from that process as opposed to setting up a server.
     #
       this = Process.pid
       that = Slave.object{ Process.pid }

       p 'this' => this, 'that' => that

     #
     # any object can be returned and it can be returned asychronously via a thread
     #
       thread = Slave.object(:async => true){ sleep 2 and [ Process.pid, Time.now ] }
       this = [ Process.pid, Time.now ]
       that = thread.value

       p 'this' => this, 'that' => that

   ~ > ruby samples/g.rb

     {"that"=>14406, "this"=>14405}
     {"that"=>[14407, Tue Nov 28 09:47:31 MST 2006], "this"=>[14405, Tue Nov 28 09:47:29 MST 2006]}

enjoy.

-a
--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama

Does Slave work on the Win32 platform, or is this a UNIX only gem?

TwP

···

On 11/28/06, Ara.T.Howard <ara.t.howard@noaa.gov> wrote:

SYNOPSIS

   the Slave class forks a process and starts a drb server in the child using
   any object as the server. the process is detached so it is not required
   (nor possible) to wait on the child pid. a Heartbeat is set up between the
   parent and child processes so that the child will exit of the parent exits
   for any reason - preventing orphaned slaves from running indefinitely. the
   purpose of Slaves is to be able to easily set up a collection of objects
   communicating via drb protocols instead of having to use IPC.

Slave is UNIX only since it relies on fork(). I've looked at the code. Ara does a nice job of keeping the fork stuff encapsulated. If you have a Win32 mechanism that could be bolted on to Slave and replace fork(), you'd get the rest of the code for free.

cr

···

On Nov 28, 2006, at 1:49 PM, Tim Pease wrote:

On 11/28/06, Ara.T.Howard <ara.t.howard@noaa.gov> wrote:

SYNOPSIS

   the Slave class forks a process and starts a drb server in the child using
   any object as the server. the process is detached so it is not required
   (nor possible) to wait on the child pid. a Heartbeat is set up between the
   parent and child processes so that the child will exit of the parent exits
   for any reason - preventing orphaned slaves from running indefinitely. the
   purpose of Slaves is to be able to easily set up a collection of objects
   communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

what is this "windows" you speak of?

:wink:

-a

···

On Wed, 29 Nov 2006, Tim Pease wrote:

On 11/28/06, Ara.T.Howard <ara.t.howard@noaa.gov> wrote:

SYNOPSIS

   the Slave class forks a process and starts a drb server in the child using
   any object as the server. the process is detached so it is not required
   (nor possible) to wait on the child pid. a Heartbeat is set up between the
   parent and child processes so that the child will exit of the parent exits
   for any reason - preventing orphaned slaves from running indefinitely. the
   purpose of Slaves is to be able to easily set up a collection of objects
   communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama

in all seriousness i'd be happy if it could be made to work - have a look and
let me know what you think if you get a chance.

-a

···

On Wed, 29 Nov 2006, Tim Pease wrote:

On 11/28/06, Ara.T.Howard <ara.t.howard@noaa.gov> wrote:

SYNOPSIS

   the Slave class forks a process and starts a drb server in the child using
   any object as the server. the process is detached so it is not required
   (nor possible) to wait on the child pid. a Heartbeat is set up between the
   parent and child processes so that the child will exit of the parent exits
   for any reason - preventing orphaned slaves from running indefinitely. the
   purpose of Slaves is to be able to easily set up a collection of objects
   communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama

I think Daniel Berger is your man for getting this to work on Windows.
Just one of those things that would be "nice to have" in the future.

TwP

···

On 11/28/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:

On Wed, 29 Nov 2006, Tim Pease wrote:

> Does Slave work on the Win32 platform, or is this a UNIX only gem?

in all seriousness i'd be happy if it could be made to work - have a look and
let me know what you think if you get a chance.

I think it might be what these GNU folks are talking about
in their coding standards document,

  If you do support Windows, please do not abbreviate it as “win”.
  In hacker terminology, calling something a “win” is a form of
  praise. You're free to praise Microsoft Windows on your own
  if you want, but please don't do this in GNU packages. Instead
  of abbreviating “Windows” to “un”, you can write it in full or
  abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
  use ‘w32’ in file names of Windows-specific files, but the macro
  for Windows conditionals is called WINDOWSNT.

Later,

···

ara.t.howard@noaa.gov wrote:

On Wed, 29 Nov 2006, Tim Pease wrote:

Does Slave work on the Win32 platform, or is this a UNIX only gem?

what is this "windows" you speak of?

--
Bil Kleb
http://fun3d.larc.nasa.gov

Tim Pease wrote:

>
> > Does Slave work on the Win32 platform, or is this a UNIX only gem?
>
> in all seriousness i'd be happy if it could be made to work - have a look and
> let me know what you think if you get a chance.
>

I think Daniel Berger is your man for getting this to work on Windows.
Just one of those things that would be "nice to have" in the future.

It's probably doable, but I'll have to look at the source code.

I don't know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara's use case for
this...

Regards,

Dan

···

On 11/28/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
> On Wed, 29 Nov 2006, Tim Pease wrote:

that's __awesome__. more humour in docs i say!

-a

···

On Thu, 30 Nov 2006, Bil Kleb wrote:

ara.t.howard@noaa.gov wrote:

On Wed, 29 Nov 2006, Tim Pease wrote:

Does Slave work on the Win32 platform, or is this a UNIX only gem?

what is this "windows" you speak of?

I think it might be what these GNU folks are talking about
in their coding standards document,

If you do support Windows, please do not abbreviate it as “win”.
In hacker terminology, calling something a “win” is a form of
praise. You're free to praise Microsoft Windows on your own
if you want, but please don't do this in GNU packages. Instead
of abbreviating “Windows” to “un”, you can write it in full or
abbreviate it to “woe” or “w”. In GNU Emacs, for instance, we
use ‘w32’ in file names of Windows-specific files, but the macro
for Windows conditionals is called WINDOWSNT.

GNU Coding Standards

--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

···

On 11/28/06, Daniel Berger <djberg96@gmail.com> wrote:

Tim Pease wrote:
> On 11/28/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
> > On Wed, 29 Nov 2006, Tim Pease wrote:
> >
> > > Does Slave work on the Win32 platform, or is this a UNIX only gem?
> >
> > in all seriousness i'd be happy if it could be made to work - have a look and
> > let me know what you think if you get a chance.
> >
>
> I think Daniel Berger is your man for getting this to work on Windows.
> Just one of those things that would be "nice to have" in the future.

It's probably doable, but I'll have to look at the source code.

I don't know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara's use case for
this...

Tim Pease wrote:

<snip>

> I don't know much about slave, but one question I have right off the
> bat is why you would want one drb server per object, instead of just
> one drb server that stored objects in a hash table that you could push
> and pop at will. Maybe I need to go back and see Ara's use case for
> this...
>

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn't IPC, what is it?

Thanks,

Dan

the purpose is two-fold:

   1) easy ipc
   2) scalling expensive operations across multiple processors easily

you certainly could setup a hashtable of objects. but a few things to
consider

   1) drop-in scalibility

     object = ExpensiveCalcuation.new

     # 1000 lines of code

   change to

     object = Slave.new{ ExpensiveCalcuation.new }

     # exact same 1000 lines of code

   and, whammo. you scale to multi-processor machines.

   2) DRbUndumped nuances

     let's say you build a table of objects like this:

       harp:~ > cat a.rb
       require "slave"
       require "yaml"

       slave = Slave.new{ Hash.new }
       table = slave.object

       array =

       table["array"] = array

       table["array"] << Process.pid

       y "slave.pid" => slave.pid
       y "array" => array
       y "table['array']" => table["array"]

     what will happen?

       harp:~ > ruby a.rb
       slave.pid: 5258
       array:
       table['array']:

     yikes!!! what went wrong? well, consider:

       array = # array is in parent

       table["array"] = array # array is marshaled across to child as a copy

       table["array"] << Process.pid # array is marshaled back across to parent and this __local_copy__ is appended too

       p "array" => array # local copy empty

       p "table['hash']" => table["hash"] # remote copy empty

     here is a fix

       array =
       array.extend DRbUndumped # the important part

     now, running again

       harp:~ > ruby a.rb
       slave.pid: 5302

       array:
       - 5301

       table['array']:
       - 5301

     but look carefully this example is even __more__ fubar: the array lives in
     the parent! we setup a table and sent only the proxy (DRbUndumped) across
     to the child! we've completely negated the entire benefit of running
     objects in another process - all we have is a lookup table for objects in
     our own address space.

     ok. take two. this time we'll try this:

       harp:~ > cat a.rb
       require "slave"
       require "yaml"

       slave = Slave.new{ # build table in child
         table = Hash.new
         array =
         table["array"] = array
         table
       }

       table = slave.object

       table["array"] << Process.pid

       y "slave.pid" => slave.pid
       y "table['array']" => table["array"]

     we give a whirl:

       harp:~ > ruby a.rb
       slave.pid: 5358
       table['array']:

     sigh. now our table exists only in the child, and so does all it's
     elements. why doesn't it work. DRbUndumped is screwing us again. here's
     why:

       table["array"] << Process.pid # here the child marshals a copy back to
                                             # parent. we append to the copy

       y "table['array']" => table["array"] # and here we get a fresh __empty__ copy

     dang. we know the fix though, in the child we do:

       array.extend DRbUndumped

     trying again

       harp:~ > ruby a.rb
       slave.pid: 5405
       table['array']: !ruby/object:DRb::DRbObject
         ref: -609755446
         uri: drbunix:///tmp/hash_-609755436_5404_5405_0

     wtf? well, since our object now lives in the child and we only get a drb
     proxy back some things work, like '<<', but other methods, like 'to_yaml', and
     'inspect' are handled locally by drb. to get around this 'to_yaml' bug we have
     to force a copy

       harp:~ > cat a.rb
       require "slave"
       require "yaml"

       slave = Slave.new{ # build table in child
         table = Hash.new
         array =
         array.extend DRbUndumped
         table["array"] = array
         table
       }

       table = slave.object

       table["array"] << Process.pid

       y "slave.pid" => slave.pid
       y "table['array']" => table["array"].map # force local copy, neither dup nor clone will cut it!

       harp:~ > ruby a.rb
       slave.pid: 5465
       table['array']:
       - 5464

     ah, finally.

so, you see, managing a hierarchy of object across drb is tricky. not that it
can't be done - i just happen to think that managing __one__ object and it's
DRbUndumped nuances is about all most hackers can manage - thus that's the
default usage. nothing will stop you from making your server object a factory
or table though - it just gets confusing fast.

kind regards.

-a

···

On Wed, 29 Nov 2006, Daniel Berger wrote:

Tim Pease wrote:

<snip>

I don't know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara's use case for
this...

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn't IPC, what is it?

--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama