Idea: Ruby Object Persistence Service

I was just thinking last night of working on a TCP/IP server which would
create, restore, and save ruby objects to a Ruby DBI compatible database,
and optionally to a series of ruby hashes that would temporarily cache
objects in cases where that behavior would be desirable.

What I was thinking of was a module that could be mixed in, would provide
the hooks into create, restore, and save, and would call a remote TCP/IP
server which would provide the functionality for handling these calls.

The server would implement a socket listener which would receive connections
and then pass them off to separate stateful, reliable threads which would
handle the rest of the transaction.

The mapping between RDBMS and object layer would be expressed in an XML
file, and would allow for two levels of object lookups. The first would be
direct queries on the database, the default being an equality to an OID
generated by a global sequence (postgresql) or equivalent, or equality to a
field in the table corresponding to that object. The second would be
lookups on the return values of attr_readers on the objects, probably in
some sort of objects.each.collect type of statement.

For create, restore, and save, a boolean flag for whether or not the cache
could be consulted would be required. The behavior of each action would be
different depending on the setting of this flag.

The XML mapping file would be loaded along with the server. At the same
time it would do a ‘require’ on each business object in question. The real
win here is for web apps, which must ‘require’ these business objects at
run time, and even in the event of require caching due to persistence owing
to mod_ruby, new apache processes still must do these ‘requires’ again.

Upon object creation, a factory would set the @mapping hash on the object,
which it could carry with it. A save or restore would consult this
mapping. This is a big win again because objects creating these mapped
objects would not need to do their own ‘require’.

Restore would accept either a scalar which would be interpreted as the
special field OID (which would be included on all tables) or as a hash of
field name, operator, and value, expressed like “foo=” => 5, “bar<” => 3,
“baz=” => “alliwantforchristmasismytwofrontteeth”

Without too much trouble, the persistence objects could probably also allow
for persistable collections, so long as the objects in those collections
are homogeneous and are themselves mapped by the xml mapping file. So an
attribute of object Foo could be a list of objects of type Baz, so long as
Baz itself is mapped.

The connection between the OPS client and the OPS server would be in the
form of a message stating the required action, followed by a
content-length, and then the serialized object, if these are appropriate.

Having said all of this, does anyone have something similar, or are there
existing technologies I should use while implementing this, or have I come
up with a completely silly idea?

Additional ideas include adding on a web-accessible monitor for the cache
and object creation, as well as perhaps a separate naming service.

–Gabriel Emerson

gabriel-

admitedly i’ve only skim read this, but i’ll go over it again in a bit.

i know i’ve mentioned them before, so perhaps i’m missing something, but
besides the caching i believe i am actually doing a good bit of this
already, albiet far from complete (i’m dealing with other issues
presently), with ONI and DBize. ONI is similar to SOAP but uses straight
TCP/IP and YAML, while DBize is a mixin that automatically maps objects
to a database. they both need some work, and you note a few things that
may be nice addtions to them, but i am using both of these in my current
project and they are working well thus far.

-transami

···

On Mon, 2002-09-02 at 16:21, Gabriel Emerson wrote:

I was just thinking last night of working on a TCP/IP server which would
create, restore, and save ruby objects to a Ruby DBI compatible database,
and optionally to a series of ruby hashes that would temporarily cache
objects in cases where that behavior would be desirable.

What I was thinking of was a module that could be mixed in, would provide
the hooks into create, restore, and save, and would call a remote TCP/IP
server which would provide the functionality for handling these calls.

The server would implement a socket listener which would receive connections
and then pass them off to separate stateful, reliable threads which would
handle the rest of the transaction.

The mapping between RDBMS and object layer would be expressed in an XML
file, and would allow for two levels of object lookups. The first would be
direct queries on the database, the default being an equality to an OID
generated by a global sequence (postgresql) or equivalent, or equality to a
field in the table corresponding to that object. The second would be
lookups on the return values of attr_readers on the objects, probably in
some sort of objects.each.collect type of statement.

For create, restore, and save, a boolean flag for whether or not the cache
could be consulted would be required. The behavior of each action would be
different depending on the setting of this flag.

The XML mapping file would be loaded along with the server. At the same
time it would do a ‘require’ on each business object in question. The real
win here is for web apps, which must ‘require’ these business objects at
run time, and even in the event of require caching due to persistence owing
to mod_ruby, new apache processes still must do these ‘requires’ again.

Upon object creation, a factory would set the @mapping hash on the object,
which it could carry with it. A save or restore would consult this
mapping. This is a big win again because objects creating these mapped
objects would not need to do their own ‘require’.

Restore would accept either a scalar which would be interpreted as the
special field OID (which would be included on all tables) or as a hash of
field name, operator, and value, expressed like “foo=” => 5, “bar<” => 3,
“baz=” => “alliwantforchristmasismytwofrontteeth”

Without too much trouble, the persistence objects could probably also allow
for persistable collections, so long as the objects in those collections
are homogeneous and are themselves mapped by the xml mapping file. So an
attribute of object Foo could be a list of objects of type Baz, so long as
Baz itself is mapped.

The connection between the OPS client and the OPS server would be in the
form of a message stating the required action, followed by a
content-length, and then the serialized object, if these are appropriate.

Having said all of this, does anyone have something similar, or are there
existing technologies I should use while implementing this, or have I come
up with a completely silly idea?

Additional ideas include adding on a web-accessible monitor for the cache
and object creation, as well as perhaps a separate naming service.

–Gabriel Emerson


tom sawyer, aka transami
transami@transami.net

This sounds really cool. I’m working on a project that, while its
objectives are a bit different from yours, may be complementary in some
interesting ways.

My effort is called RuFWS (Ruby Framework for Web Services–pronounced
‘rufus’); there’s no package or coherent documentation for it yet, but
I’ve got it set up as a SourceForge project, and I hope to have some
proof-of-concept code checked in this week. And you can be sure that
once I feel it’s ready to expose to the world, I’ll publicize the crap
out of it :wink:

But not to keep you completely in the dark, what I know so far about
RuFWS is that:

  • the essence of the project is to provide a set of Ruby libraries
    that make it easy to construct and deploy XML-based web services

  • it uses plugins to the max, and any Ruby library that conforms to
    the very simple plugin API can be used as a plugin

  • it uses macros, which are combinations of built-in methods, plugin
    methods, and/or other macros, described in XML

  • it will probably provide both SOAP and REST interfaces, but it will
    have a REST interface first
    (what’s REST? see REST)

  • it uses REXML as its primary XML processor

  • it operates as an Apache extension, using mod_ruby. I have no
    intention of writing yet another HTTP server

  • I’m trying to avoid getting too deep into object persistence issues.
    It’s not my forte, and I’d rather work with some of the good software
    that already exists (or will exist by the time this project matures).

···

On Tue, Sep 03, 2002 at 07:21:31AM +0900, Gabriel Emerson wrote:

Having said all of this, does anyone have something similar, or are there
existing technologies I should use while implementing this, or have I come
up with a completely silly idea?

Additional ideas include adding on a web-accessible monitor for the cache
and object creation, as well as perhaps a separate naming service.


Matt Gushee
Englewood, Colorado, USA
mgushee@havenrock.com

I have never heard of DBize before, does this maintain mappings of collections
for you as well? The holy grail is to not write SQL (or whatever you need to
store persistently) for the majority of the cases.

···

On Tue, 03 Sep 2002 12:37, Tom Sawyer wrote:

gabriel-

admitedly i’ve only skim read this, but i’ll go over it again in a bit.

i know i’ve mentioned them before, so perhaps i’m missing something, but
besides the caching i believe i am actually doing a good bit of this
already, albiet far from complete (i’m dealing with other issues
presently), with ONI and DBize. ONI is similar to SOAP but uses straight
TCP/IP and YAML, while DBize is a mixin that automatically maps objects
to a database. they both need some work, and you note a few things that
may be nice addtions to them, but i am using both of these in my current
project and they are working well thus far.

-transami

On Mon, 2002-09-02 at 16:21, Gabriel Emerson wrote:

I was just thinking last night of working on a TCP/IP server which would
create, restore, and save ruby objects to a Ruby DBI compatible database,
and optionally to a series of ruby hashes that would temporarily cache
objects in cases where that behavior would be desirable.

What I was thinking of was a module that could be mixed in, would provide
the hooks into create, restore, and save, and would call a remote TCP/IP
server which would provide the functionality for handling these calls.

The server would implement a socket listener which would receive
connections and then pass them off to separate stateful, reliable threads
which would handle the rest of the transaction.

The mapping between RDBMS and object layer would be expressed in an XML
file, and would allow for two levels of object lookups. The first would
be direct queries on the database, the default being an equality to an
OID generated by a global sequence (postgresql) or equivalent, or
equality to a field in the table corresponding to that object. The
second would be lookups on the return values of attr_readers on the
objects, probably in some sort of objects.each.collect type of statement.

For create, restore, and save, a boolean flag for whether or not the
cache could be consulted would be required. The behavior of each action
would be different depending on the setting of this flag.

The XML mapping file would be loaded along with the server. At the same
time it would do a ‘require’ on each business object in question. The
real win here is for web apps, which must ‘require’ these business
objects at run time, and even in the event of require caching due to
persistence owing to mod_ruby, new apache processes still must do these
‘requires’ again.

Upon object creation, a factory would set the @mapping hash on the
object, which it could carry with it. A save or restore would consult
this mapping. This is a big win again because objects creating these
mapped objects would not need to do their own ‘require’.

Restore would accept either a scalar which would be interpreted as the
special field OID (which would be included on all tables) or as a hash of
field name, operator, and value, expressed like “foo=” => 5, “bar<” => 3,
“baz=” => “alliwantforchristmasismytwofrontteeth”

Without too much trouble, the persistence objects could probably also
allow for persistable collections, so long as the objects in those
collections are homogeneous and are themselves mapped by the xml mapping
file. So an attribute of object Foo could be a list of objects of type
Baz, so long as Baz itself is mapped.

The connection between the OPS client and the OPS server would be in the
form of a message stating the required action, followed by a
content-length, and then the serialized object, if these are appropriate.

Having said all of this, does anyone have something similar, or are there
existing technologies I should use while implementing this, or have I
come up with a completely silly idea?

Additional ideas include adding on a web-accessible monitor for the cache
and object creation, as well as perhaps a separate naming service.

–Gabriel Emerson


Signed,
Holden Glova

well, if i understand you correctly, then yes. DBize includes two mixins
actually, DBize::Record and DBize::Subrecords. If you subclass Array and
mixin DBize::Subrecords, the you have a means of dealing with a
collection of objects mapped to the database. if, like i said, that is
what you mean by “mappings of collections”

indeed, no sql required, in all cases i think.

essentially DBize knows the meta-inforamation of the table your object
pertains too. so it uses the dynamicism of ruby to make method calls
using #send with the table’s attribute (field) names. for instance,
here’s a mockup:

CREATE TABLE mapped_table
( record SERIAL PRIMARY KEY,
name text DEFAULT ‘’,
value text DEFAULT ‘’
);

class MappedObject
include DBize::Record
attr_accessor :record
attr_accessor :name
attr_accessor :value
def table
‘mapped_table’
end
end

mo = MappedObject.new
mo.load_from_database(1)
mo.name=“A New Name”
mo.update_database

this loads the record for mapped_table where record=1 into the object.
changes the name attribute and saves it back to the table.

notice the database atrributes (fields) and the object accessor
attributes have the exact same names. this is really the key to DBizes
ease of use. and is not limiting since you can always define accessor
methods by hand, even complex ones.

the record attribute is required by DBize to get the record’s id
attribute. if that field isn’t called record in the table then simply
define record manually to access whatever it is:

attr_accessor :myid

def record
myid
end

def record=(x)
myid = x
end

i’ll probably improve on this in the future. the method #table is
required as well, to tell DBize which table the object is stored in,
this i beleive i am in the process of moving to a class method instead.
anyway you get the idea.

···

On Mon, 2002-09-02 at 18:48, Holden Glova wrote:

I have never heard of DBize before, does this maintain mappings of collections
for you as well? The holy grail is to not write SQL (or whatever you need to
store persistently) for the majority of the cases.


tom sawyer, aka transami
transami@transami.net