[DRb] Using DRb to implement object database

ruby-talk
1

Hello all,

I am try to implement a simple object database where one process is a
DRb server that holds the objects in the database and uses pstore to
save/restore the database. In my client program I want to create an
object that I then want to put into the database. So for example here is
a simple class:

class Foo
attr_accessor :x,:y

def initailize(x,y)
    @x, @y = x,y
end

end

On the client I create an object from that class:

foo = Foo.new(1,2)

I connect to the database (it is a simple array)

DRb.start_service()
object_database = DRbObject.new(nil, “drb://localhost:9000”)

I put the object into the database

object_database.push(foo)

Now If I modify the object locally

foo.x = 4

It only modifies it locally (the object on the DRb server is unmodified)
If I instead get the object back from database

bar = object_database[0]
bar.x = 5

bar is still just local.

Now If I include DRbUndumped in class Foo, then my changes are available
at the server and another client can see them. However, when I go to
save the object with pstore, it doesn’t save something of class Foo, it
saves a DRbObject because the object still really only exists in the
client and is proxied at the server.

If I do the following to create foo at the server (assume pusheval does
the eval at the server):

object_database.pusheval(“Foo.new(1,2)”)

Now the object at the server is of class Foo and if I get it on the
client it is a DRbObject (thus proxied). I can set the values on the
client and they happen on the server. Great!

Unfortunately, I can no longer save the object because module
DRbUndumped is the following:

module DRbUndumped
def _dump(dummy)
raise TypeError, 'can’t dump’
end
end

And so pstore can’t dump it to the disk.

Any ideas of how I should do this?

Thanks in advance,

Steve Tuckner

2

Hello all,

I am try to implement a simple object database where one process is a
[…]
Now If I modify the object locally
[…]
It only modifies it locally (the object on the DRb server is unmodified)
If I instead get the object back from database
[…]
bar is still just local.

The short answer is that you might get on better looking at the
Rinda examples. See RubyTalk:15023 for more on this.

The long answer involves more understanding of the use of
DRbUndumped than I currently have.

    Hugh
···

On Fri, 8 Aug 2003, Steve Tuckner wrote:

3

I don’t know anything about pstore, but I do use DRb a lot, so I might be
able to help partly.

If you generate an instance of class Foo on the server, then running
foo.x = 4
on the client (where foo is a DRb object pointing at the remote object)
it will be sent as a method call to the object on the server, which will
update itself.

However if pstore is a remote object store, then I imagine the only methods
it offers will be “fetch” and “put” operations. So if you use DRb directly
as a front to the object store itself, that will be all you can do. In order
to modify an object, the client will have to:

- get foo (makes a local copy on the client)
- update it (on the client)
- put foo (replace the object in the object store)

If you want to have a way to update foo directly on the server, then your
DRb server needs to implement a method to dispatch a call to it:

/---------- SERVER ------------\        DRB        /--- CLIENT ---\

  pstore <----- proxy object <-------------------------- client

The client calls a method on the proxy object, which you have written. It in
turn gets an object from pstore, performs an operation, and puts it back (or
putting it back might not be necessary since it’s on the same machine):

e.g. in pseudo-code

   class MyProxy 
     def initialize
       @store = Pstore.new
       3.times { @store.push(Foo.new) }   # put some objects in
     end
     def setx(index, value)
       @store.fetch(index).x = value
     end
   end

You might think of implementing a generic send-to method:

     def send_to(index, meth, *args)
       @store.fetch(index).send(meth, *args)
     end

     # client can do:
     #     pstore.send_to(2, :y=, 99)

but this is extremely dangerous, because it bypasses the protection of
‘private’ methods. In particular, you can send any Kernel methods like
system and backtick:

irb(main):003:0> a = “”
irb(main):004:0> a.send(:system,“ls”)

So if the client does

     a.send_to(0, :system, "rm -rf /*")

this would probably not be a welcome method call :slight_smile:

Regards,

Brian.

···

On Fri, Aug 08, 2003 at 12:26:19AM +0900, Steve Tuckner wrote:

I am try to implement a simple object database where one process is a
DRb server that holds the objects in the database and uses pstore to
save/restore the database. In my client program I want to create an
object that I then want to put into the database. So for example here
is a simple class:

4

Hello, Steve.

Do you want it?
This database pass a root object by reference.

— foo.rb —
class Foo
def initialize(x, y)
@x = x
@y = y
end
attr_accessor :x, :y
end

— db.rb —
require 'pstore’
require 'drb/drb’
require 'drb/timeridconv’
require ‘foo’

class Database < PStore
include DRbUndumped
def
if Thread.current[‘DRb’]
return DRbObject.new(super)
else
super
end
end
end

store = Database.new(‘test.db’)

DRb.install_id_conv(DRb::TimerIdConv.new)
DRb.start_service(‘druby://localhost:12345’, store)
DRb.thread.join

— client.rb —
require 'drb/drb’
require ‘foo’

def main
DRb.start_service
db = DRbObject.new_with_uri(‘druby://localhost:12345’)
db.transaction do
db[‘foo’] = Foo.new(1, 2) unless db.root?(‘foo’)
end
db.transaction do
foo = db[‘foo’] # pass by reference
foo.x = foo.x + 1
end
db.transaction do
foo = db[‘foo’]
p [foo.x, foo.y, foo.class]
end
end

main

— client2.rb —
require 'drb/drb’
require ‘foo’

def main
DRb.start_service
db = DRbObject.new_with_uri(‘druby://localhost:12345’)
db.transaction do
db[‘foo’] = Foo.new(1, 2) unless db.root?(‘foo’)
db[‘bar’] = Foo.new(db[‘foo’], 2) unless db.root?(‘bar’)
end
db.transaction do
bar = db[‘bar’] # pass by reference
p bar
p bar.x # pass by value
bar.x.x = bar.x.x + 1 # can’t change bar.x.x
bar.y = bar.y + 1 # can change bar.y
end
db.transaction do
bar = db[‘bar’]
p [bar.x, bar.y, bar.class]
end
end

main

5

I don’t know anything about pstore, but I do use DRb a lot,
so I might be
able to help partly.

I had some trouble understanding your solution, so I will attempt to explain
how PStore works and what the exact problem is. PStore is just a set of
marshalled objects on the server’s local disk. To store the objects on disk,
just create the pstore

require “pstore”

store = PStore.new(“test.db”)
store.transaction do
store[“objects”] = object_list # where object_list is the array of objects
in the database
end

When I do that however foo can’t be dumped because its _dump function has
been replaced by a function that raises a TypeError exception (and produces
the following output).

d:/a/ruby180/lib/ruby/site_ruby/1.8/drb/drb.rb:27:in _dump': can't dump (TypeError) from d:/a/ruby180/lib/ruby/1.8/pstore.rb:120:in dump’
from d:/a/ruby180/lib/ruby/1.8/pstore.rb:120:in transaction' from d:/projects/ruby/lib/obdb/ObjectDatabase.rb:154:in save’
from fxobjectdatabase.rb:116

Now it seems that the DRbUndumped include which allows the DRb to generate a
proxy object (a DRbObject) on the client doesn’t do anything else other than
act as a marker to allow DRb to know to use a proxy object. In
DRbMessage.dump (below)

def dump(obj)
  obj = DRbObject.new(obj) if obj.kind_of? DRbUndumped
  begin
str = Marshal::dump(obj)
  rescue
str = Marshal::dump(DRbObject.new(obj))
  end
  [str.size].pack('N') + str
end

is the only reference to DRbUndumped. So I figured if I just didn’t redefine
_dump in the DRbUndumped module, it could still see the object needed to be
proxied, but dump would still work. However that didn’t work. Now I could
Marshal the object to disk but proxying on the client stopped working.

In essence, what I want to be able to do is create an object on the server
and be able to :

  1. Save it to disk (via Marshal)
  2. Allow DRb to proxy it on the client, so that the client can modify it.

It doesn’t seem like too much to ask, but I don’t understand DRb well enough
to make it happen…yet… :wink:

Steve Tuckner

6

So by looking at your changes, I modified my example s.rb as below:

require “drb”
require “a”

class OD < Array
include DRbUndumped
def
if Thread.current[‘DRb’]
return DRbObject.new(super)
else
super
end
end
end

od = OD.new
od.push(A.new(1,2))

puts “at beginning, od[0].x = #{od[0].x.inspect}”

DRb.start_service(‘druby://localhost:9000’, od)

···

This made it unnecessary to include DRbUndumped in class Foo and therefore,
I can now Marshal Foo to a pstore (or other places).

So thanks for that! :slight_smile:

This brings up more questions however.

  1. What is DRb.install_id_conv(DRb::TimerIdConv.new) ? I didn’t use it in my
    code and everything worked OK, so what is it for?

  2. In client2.rb you made a reference in db[“bar”] to db[“foo”] and it
    turned out to be a local setting only. Is there a way to make that not
    local? Would there be a way to create a Foo object on the client which
    references objects on the server that when this Foo object is put to the
    server the references are translated into local references to the objects on
    the server?

Thanks again for your help.

Steve Tuckner

-----Original Message-----
From: Masatoshi SEKI [mailto:seki@mr.nasu.toshiba.co.jp]On Behalf Of
m_seki@mva.biglobe.ne.jp
Sent: Thursday, August 07, 2003 5:50 PM
To: ruby-talk ML
Subject: Re: [DRb] Using DRb to implement object database

Hello, Steve.

Do you want it?
This database pass a root object by reference.

— foo.rb —
class Foo
def initialize(x, y)
@x = x
@y = y
end
attr_accessor :x, :y
end

— db.rb —
require ‘pstore’
require ‘drb/drb’
require ‘drb/timeridconv’
require ‘foo’

class Database < PStore
include DRbUndumped
def
if Thread.current[‘DRb’]
return DRbObject.new(super)
else
super
end
end
end

store = Database.new(‘test.db’)

DRb.install_id_conv(DRb::TimerIdConv.new)
DRb.start_service(‘druby://localhost:12345’, store)
DRb.thread.join

— client.rb —
require ‘drb/drb’
require ‘foo’

def main
DRb.start_service
db = DRbObject.new_with_uri(‘druby://localhost:12345’)
db.transaction do
db[‘foo’] = Foo.new(1, 2) unless db.root?(‘foo’)
end
db.transaction do
foo = db[‘foo’] # pass by reference
foo.x = foo.x + 1
end
db.transaction do
foo = db[‘foo’]
p [foo.x, foo.y, foo.class]
end
end

main

— client2.rb —
require ‘drb/drb’
require ‘foo’

def main
DRb.start_service
db = DRbObject.new_with_uri(‘druby://localhost:12345’)
db.transaction do
db[‘foo’] = Foo.new(1, 2) unless db.root?(‘foo’)
db[‘bar’] = Foo.new(db[‘foo’], 2) unless db.root?(‘bar’)
end
db.transaction do
bar = db[‘bar’] # pass by reference
p bar
p bar.x # pass by value
bar.x.x = bar.x.x + 1 # can’t change bar.x.x
bar.y = bar.y + 1 # can change bar.y
end
db.transaction do
bar = db[‘bar’]
p [bar.x, bar.y, bar.class]
end
end

main

7

I had some trouble understanding your solution, so I will attempt to explain
how PStore works and what the exact problem is. PStore is just a set of
marshalled objects on the server’s local disk. To store the objects on disk,
just create the pstore

require “pstore”

store = PStore.new(“test.db”)
store.transaction do
store[“objects”] = object_list # where object_list is the array of objects
in the database

Sorry if I wasn’t clear :slight_smile:

So ‘store’ looks like a hash? store[key] = some_object

If you update store or one of the objects it refers to, is there anything
you need to do to write it back to disk? Hmm, probably that’s what the
transaction syntax achieves.

OK, I’ve had a play. The attached code hopefully demonstrates what I was
trying to say.

DRb provides a remote ‘facade’ to a single object on a particular port, 9000
in this case. So you create a single object which contains a set of methods
which do all the different work parcels you want - including invoking
methods on other objects if necessary.

That’s unless you want a separate DRb front-end to each individual Foo
object, which would mean binding them to separate TCP ports. That’s not an
approach I would recommend unless you’ve only got two or three objects you
want to access remotely.

If you decided to make the PStore object itself available by DRb, then the
only methods you would get would be those methods which PStore itself
provides - such as and = to fetch and set the contained objects. And
running over DRb, in its normal operation, you would get copies of those
objects. Updating state on a local copy of an object will do nothing on the
server side of course; you would have to explicitly overwrite the old object
with the new one using =.

When I do that however foo can’t be dumped because its _dump function has
been replaced by a function that raises a TypeError exception (and produces
the following output).

I don’t know anything about DRbUndumped (have you got a reference to a
description? [*]) Standard DRb just marshals a method call and its arguments
across a TCP stream, and marshals the method result back over the stream. So
it’s like a local method call except the arguments and the result are
'dup’ed (deep-copied, actually)

I don’t know anything about ‘Linda’ or Java either, so Rinda doesn’t mean
much to me.

If DRbUndumped automatically puts in some kind of proxy-object for each
remote object on the server, then that would be very cool indeed. But then I
see your problem with PStore, which relies on being able to dump objects to
get them to disk. Have you tried just commenting out the _dump function? It
looks like DRb only tests whether the module has been included:

  obj = DRbObject.new(obj) if obj.kind_of? DRbUndumped

Hence it probably doesn’t actually need _dump to be overridden.

Regards,

Brian.

[*] In the source tarball there’s a reference in doc/drb-ref.rd.ja , but the
text in that file looks like it has been marshaled in a strange binary
format that I’m not able to read :slight_smile:

foo.rb (92 Bytes)

server.rb (656 Bytes)

client.rb (237 Bytes)

···

On Fri, Aug 08, 2003 at 04:36:42AM +0900, Steve Tuckner wrote:

8

Try doing something like this: (untested, no warranty :slight_smile:

class Class
def self.proxy?
false
end
end

module Proxy
def self.proxy?
true
end
end

class YourClass
include Proxy
end

def dump(obj)
obj = DRbObject.new(obj) if obj.kind_of? DRbUndumped
begin
raise if obj.class.proxy? # add this line
str = Marshal::dump(obj)
rescue
str = Marshal::dump(DRbObject.new(obj))
end
[str.size].pack(‘N’) + str
end

···

–
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

9

DRb provides a remote ‘facade’ to a single object on a
particular port, 9000
in this case. So you create a single object which contains a
set of methods
which do all the different work parcels you want - including invoking
methods on other objects if necessary.

Yes I agree.

That’s unless you want a separate DRb front-end to each individual Foo
object, which would mean binding them to separate TCP ports.
That’s not an
approach I would recommend unless you’ve only got two or
three objects you
want to access remotely.

I also agree.

If you decided to make the PStore object itself available by
DRb,

I don’t

If DRbUndumped automatically puts in some kind of
proxy-object for each
remote object on the server, then that would be very cool
indeed.

This it seems to do. Any object that is returned by a remote method call is
proxied if it includes DRbUndumped. If you look at my attached example. If
you comment out the include of DRbUndumped, you will see that the object
returned is not a DRbObject and thus all changes to the A class object are
local.

see your problem with PStore, which relies on being able to
dump objects to
get them to disk. Have you tried just commenting out the
_dump function? It
looks like DRb only tests whether the module has been included:

  obj = DRbObject.new(obj) if obj.kind_of? DRbUndumped

Hence it probably doesn’t actually need _dump to be overridden.

I tried to comment out the _dump redefinition in the module DRbUndumped and
the proxying no longer worked. I am not sure why.

Steve Tuckner

a.rb (130 Bytes)

c.rb (301 Bytes)

s.rb (314 Bytes)