[ANN] VAPOR 0.06, Transparent Persistence to PostgreSQL

ruby-talk
1

Hi!

I’m in the process of writing (yet another) persistence framework
for Ruby that supports versioning and transactions: Vapor.
The base functionality as an Object Repository is roughly done and
it has reached a state where it can actually played with, so I’d like to
release it to the public to get some feedback.

http://www.cip.informatik.uni-muenchen.de/~bolzer/vapor/

Vapor is being developed at the University of Munich’s Department for
Informatics as storage backend for an ongoing project to better manage
network elements (including User and Accounts) of the department network.
As such, the missing features will definetly be implemented in the next
months.

From the README:

Description

···

Vapor is a persistent Object-Repository for Ruby, providing transparent
persistence of interrelated Ruby application objects to a PostgreSQL database.
It’s goal is to provide developers with an easy-to-use framework for
persistence, that is consistent with OO design and does not interfere with
the code of classes that are to be persistently stored. Vapor does not require
any knowledge about relational databases to be used, so that developers can
concentrate on the task of writing their application logic.

Some of Vapor’s general design was inspired by the JDO (Java Data Objects)
standard.

Features

  • Object-Oriented API
  • transparent to persistable classes
  • support for inheritance
  • flexible queries for object retrieval
  • generation of human-readable RDBMS schema
  • metadata specification through separate XML markup
  • concurrent access by multiple users
  • uniqueness constrains

Planned:

  • Transactions (ACID) with rollback support
  • Versioning with complete audit trail
  • Long Transactions (for Workflow-support, as branches in versioning)
  • more backends (other RDBMS, local files,…)

It has some dependencies, notably Ruby-DBI from CVS (for Array support
in DBD::Pg, patch included in Vapor’s tar-ball).

Documentation is still only VERY basic, but Code examples on how to use Vapor
are linked to from the web page.

Any feedback or suggestions will be appreciated.

–
Oliver M. Bolzer
oliver@gol.com

GPG (PGP) Fingerprint = 621B 52F6 2AC1 36DB 8761 018F 8786 87AD EF50 D1FF

2

Vapor is a persistent Object-Repository for Ruby, providing transparent
persistence of interrelated Ruby application objects to a PostgreSQL database.
…
It has some dependencies, notably Ruby-DBI from CVS (for Array support
in DBD::Pg, patch included in Vapor’s tar-ball).
…
Any feedback or suggestions will be appreciated.

If it uses the DBI API, is it possible that it may work with other SQL
databases?

Otherwise, if it is very specific to Postgres, perhaps it does not need to
use the DBI layer at all - it could use the native Postgres API?

(From my point of view, it would be nice if it supported Oracle and/or
Mysql)

Regards,

Brian.

···

On Fri, Jun 06, 2003 at 07:40:05PM +0900, Oliver M. Bolzer wrote:

3

Hi!

This shounds like a great idea! I will go right ahead and see if you
made something I can steal and use in my own object persistence library :slight_smile:

As it seems everyone and his sister is making this we might compare
notes. There is no need for us all to make the same mistakes… sadly I
won’t have time to look at your library 'till tuesday, but until then:
please look at my take at this:

sql-serialize (presently in ver. 0.0.3):
sdfhttps://sf.net/rpojects/rsqlserial - project page
http://rsqlserial.sf.net - documentation

my goals are:

  • no xml metadata
  • almost transparent to persistent classes
  • database independent (I started out with mysql but postgresql is
    comming next week)

I made a pseudo-EER-diagram trying to show how I store the data in the
database: http://rsqlserial.sf.net/overview.png

Some of our mutual competition in the field of transparent persistence:

db-backend (it’s not on the RAA, and a one-second reseach through google
didn’t find it, so I cannot give you an url atm)

madeleine isn’t really using a database, but rather marshalls the
classes, so if you want to have your data stored in a real database
madeleine isn’t the way to go…

db-backend (like your lib) uses xml-metadata, there may be some stuff to
compare yourself with there.

my own (humble) lib is describing the data by itself in stead of having
the xml metadata. there are both pros and cons to this approach. For
instance I need to have an object to model the ddl after for creating
the tables. The instance variables of this object will be used to figure
out what columns should be in the tables and which datatypes they should
have.

as soon as I get a chance to look at your library I will most likely
have lots of feedback :slight_smile: it would of cause be nice if you would return
the favor :wink:

/Anders

Oliver M. Bolzer wrote:

···

Hi!

I’m in the process of writing (yet another) persistence framework
for Ruby that supports versioning and transactions: Vapor.
The base functionality as an Object Repository is roughly done and
it has reached a state where it can actually played with, so I’d like to
release it to the public to get some feedback.

http://www.cip.informatik.uni-muenchen.de/~bolzer/vapor/

Vapor is being developed at the University of Munich’s Department for
Informatics as storage backend for an ongoing project to better manage
network elements (including User and Accounts) of the department network.
As such, the missing features will definetly be implemented in the next
months.

From the README:

Description

Vapor is a persistent Object-Repository for Ruby, providing transparent
persistence of interrelated Ruby application objects to a PostgreSQL database.
It’s goal is to provide developers with an easy-to-use framework for
persistence, that is consistent with OO design and does not interfere with
the code of classes that are to be persistently stored. Vapor does not require
any knowledge about relational databases to be used, so that developers can
concentrate on the task of writing their application logic.

Some of Vapor’s general design was inspired by the JDO (Java Data Objects)
standard.

Features

  • Object-Oriented API
  • transparent to persistable classes
  • support for inheritance
  • flexible queries for object retrieval
  • generation of human-readable RDBMS schema
  • metadata specification through separate XML markup
  • concurrent access by multiple users
  • uniqueness constrains

Planned:

  • Transactions (ACID) with rollback support
  • Versioning with complete audit trail
  • Long Transactions (for Workflow-support, as branches in versioning)
  • more backends (other RDBMS, local files,…)

It has some dependencies, notably Ruby-DBI from CVS (for Array support
in DBD::Pg, patch included in Vapor’s tar-ball).

Documentation is still only VERY basic, but Code examples on how to use Vapor
are linked to from the web page.

Any feedback or suggestions will be appreciated.

–
dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

4

Brian Candler wrote:

Vapor is a persistent Object-Repository for Ruby, providing transparent
persistence of interrelated Ruby application objects to a PostgreSQL database.

…

It has some dependencies, notably Ruby-DBI from CVS (for Array support
in DBD::Pg, patch included in Vapor’s tar-ball).

…

Any feedback or suggestions will be appreciated.

If it uses the DBI API, is it possible that it may work with other SQL
databases?

not very likely… any sql the library generates will most likely have
to be taylored to the database in question.

Otherwise, if it is very specific to Postgres, perhaps it does not need to
use the DBI layer at all - it could use the native Postgres API?

dbi is very easy to use, I myself am using it even tho I write
db-specific sql. with all the quirks of all the databases dbi is no
guarantee for database independence, just an easy way to access a database.

(From my point of view, it would be nice if it supported Oracle and/or
Mysql)

my transparent persistence library has mysql support

···

On Fri, Jun 06, 2003 at 07:40:05PM +0900, Oliver M. Bolzer wrote:

Regards,

Brian.

–
dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

5

Brian Candler wrote:

If it uses the DBI API, is it possible that it may work with other SQL
databases?

Otherwise, if it is very specific to Postgres, perhaps it does not need to
use the DBI layer at all - it could use the native Postgres API?

The current backend (that handles Tuples) uses PostgreSQL’s support for
Arrays and table inheritance. If another DBD supports these features, only
minor tweaks (or checks) for SQL conformance should be needed. If I’m not
totally wrong, it should be doable with Oracle.

For RDBMSs like MySQL that don’t do table inheritance and array, another
database scheme needs to be tought of and parts of the backend willneed to
be rewritten to adjust to this. (current backend: ~270 lines incl.
comments)

Adding other backends is on my list of TODOs, but unfortunately only after
Transactions and Versioning, these are the I need first :slight_smile:

6

I’m writing such a library too :wink: It’s not yet released, but hopefully
will be soon. It is modelled basing on Enterprise Objects Framework
(Java), so it aims to be bigger and more complete than yours, I think :slight_smile:

Features:

  • totally database independence: no sql is used in application - sql
    queries are created using “qualifiers” and converted to sql using
    adaptors - all you need to do is to write database-specific adaptor
    (which is trivial and sometimes even not necessary, if the generic
    adaptor works well with the database). My goal is to make it as db
    independent as EOF is - switching to another RDBMS will require only
    changing one line in config file and generating new schema in database.
  • support for transactions
  • XML metadata (database independence again)
  • support for relationships transparent for developer (I’m not sure if
    any of existing libraries has this feature) - eg. if you have
    Employee->Department relationship you only need to write
    employee.add_to_relationship ( “department”,department ) and when saving
    to database proper foreign and primary key values will be generated.
    Then you can eg. delete department object and all its employees will be
    deleted recursively (or their department will be set to null - you can
    set this behavour in config file).

Features not implemented and required before release:

  • writing documentation :slight_smile:
  • some more testing
  • minor changes to the adaptor

I write this library (called SDS) because I need it for my second
project (called in turn SWS :), which is a web application library
similar to Apple WebObjects (Java). Anyone who used WO can confirm it is
extremally powerful and easy to use framework, so maybe (hopefully:) SWS
will be good enough to write Ruby businnes applications in it - people
are definitely lacking good web development library. SWS has a lot of
advantages over traditional CGI/FCGI/mod_ruby applications because it
uses MVC approach. It also has some key advantages over HTML template
libraries (such as Amrita) - eg. you can bind a String with a HTML
textfield and its value will be displayed in the textfield when
rendering the page, but additionally after submitting the form the value
of the String object will be set to the value entered in text field,
with no additional work by developer! Another example - you can make a
HTML table with a header row and second row wrapped in something called
“repetition”, bind some object_array and object_iterator with the
repetition and you will get table with displayed attributes of object in
the table. If you place a button in the repetition row, you can bind an
action (basically a Ruby method) with it - when it will be called,
before invoking a method object_iterator will be set to object in the
row in which you clicked the button!

There is already one library based on WebObject - CGIKit, but it is
much simpler than mine and has some WO features lacking: it runs as a
CGI script - SWS runs as a standalone application and is connected with
WWW server through adaptors (CGI/FCGI/mod_ruby/whatever - they are not
written yet :), has no persistent Application object, Session support is
lacking a bit. It is very good as a simple library with most of
fundamental WO features implemented, but I wanted something much more
complete.

If anobody is interested in SDS and/or SWS please email me at
marek at janukowicz dot net - I can send you the sources in current
state (almost no documentation :). I think a lot of people would
appreciate making some libraries thay could write SERIOUS applications
with (just joking :).

Phew… I think this post is much longer than all my previous ones
together :slight_smile:

···

On Fri, 06 Jun 2003 13:44:43 +0200, Anders Borch wrote:

This shounds like a great idea! I will go right ahead and see if you
made something I can steal and use in my own object persistence library :slight_smile:

As it seems everyone and his sister is making this we might compare
notes. There is no need for us all to make the same mistakes… sadly I
won’t have time to look at your library 'till tuesday, but until then:
please look at my take at this:

sql-serialize (presently in ver. 0.0.3):
sdfhttps://sf.net/rpojects/rsqlserial - project page
http://rsqlserial.sf.net - documentation

my goals are:

  • no xml metadata
  • almost transparent to persistent classes
  • database independent (I started out with mysql but postgresql is
    comming next week)

–
Marek Janukowicz

7

this sounds really interesting… what can I say… release fast release
often, as someone said told long time ago :wink:

Possibly someone could even take a look at mephle, I suppose it offers
object persistency someway

···

il 7 Jun 2003 08:52:01 GMT, Marek Janukowicz childNOSPAM@t17.ds.pwr.wroc.pl ha scritto::

I’m writing such a library too :wink: It’s not yet released, but hopefully
will be soon. It is modelled basing on Enterprise Objects Framework
(Java), so it aims to be bigger and more complete than yours, I think :slight_smile:

Features:

  • totally database independence: no sql is used in application - sql
    queries are created using “qualifiers” and converted to sql using
    adaptors - all you need to do is to write database-specific adaptor
    (which is trivial and sometimes even not necessary, if the generic
    adaptor works well with the database). My goal is to make it as db
    independent as EOF is - switching to another RDBMS will require only
    changing one line in config file and generating new schema in database.
8

Marek Janukowicz wrote:

Features:

  • totally database independence: no sql is used in application - sql
    queries are created using “qualifiers” and converted to sql using
    adaptors - all you need to do is to write database-specific adaptor
    (which is trivial and sometimes even not necessary, if the generic
    adaptor works well with the database). My goal is to make it as db
    independent as EOF is - switching to another RDBMS will require only
    changing one line in config file and generating new schema in database.

hmm… someone should take this code and see if it could be used in a
statement parser for ruby-dbi, see their todo list at
http://ruby-dbi.sourceforge.net/ToDo.html

  • support for transactions

how is this going to show in your api? In my implementation
inserting/updating/deleting a class and all associated classes is
wrapped in a transaction. When calling insert() on a persistent class
that class and all associated classes will be inserted:

snippet from my code:

def insert (dbi)
begin
dbi.do(“BEGIN”)
sql_insert (dbi)
sql_insert_array (dbi)
sql_insert_hash (dbi)
sql_insert_assoc (dbi)
dbi.do(“COMMIT”)
rescue DBI::DatabaseError => dbe
dbi.do(“ROLLBACK”)
throw dbe
end
return @sql_assoc_id
end

  • XML metadata (database independence again)

what are you storing in this xml document? I seem to achieve database
independence just fine without any xml metadata. Maybe I’m missing
something?

  • support for relationships transparent for developer (I’m not sure if
    any of existing libraries has this feature) - eg. if you have
    Employee->Department relationship you only need to write
    employee.add_to_relationship ( “department”,department ) and when saving
    to database proper foreign and primary key values will be generated.
    Then you can eg. delete department object and all its employees will be
    deleted recursively (or their department will be set to null - you can
    set this behavour in config file).

uhm… my library has this. If you have a an object that has an
associated object, then this association will persist without having to
write a single line of code. The programmer doesn’t even have to make
any add_to_relationship(), my persistence library will notice all
associations when insert/update is called.

Features not implemented and required before release:

  • writing documentation :slight_smile:

I know this one… that’s a really though one… imho rdoc helps out a
lot here :slight_smile:

If anobody is interested in SDS and/or SWS please email me at
marek at janukowicz dot net - I can send you the sources in current
state (almost no documentation :).

why not put it on sourceforge as it is? that way we can all look at it
and even send contributions to you. I put mine on sourceforge almost
immediately.

···

–
dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

9

snippet from my code:

def insert (dbi)
begin
dbi.do(“BEGIN”)
sql_insert (dbi)
sql_insert_array (dbi)
sql_insert_hash (dbi)
sql_insert_assoc (dbi)
dbi.do(“COMMIT”)

Or you could write:

   dbi.transaction do
     sql_insert(dbi)
     ...
   end

(which might in future work better; beware that right now ruby-dbi has no
concept of “begin transaction” so it’s rather broken in this regard)

rescue DBI::DatabaseError => dbe
  dbi.do("ROLLBACK")
  throw dbe
end

Are you sure that’s right??? I think you want to do a rollback for all
exceptions. That’s one reason why the ‘transaction do … end’ model might
be better.

Regards,

Brian.

···

On Sun, Jun 08, 2003 at 03:13:56PM +0900, Anders Borch wrote:

10

Hi!

had no access to news over the weekend+holyday

Anders Borch wrote:

  • support for transactions

how is this going to show in your api? In my implementation
inserting/updating/deleting a class and all associated classes is
wrapped in a transaction. When calling insert() on a persistent class

I’m currently working on transactions with optimistic locking. (done except
rollback :slight_smile:

A code snipped would look like

Transaction as a block

@pmgr.transaction{|t|
munich.altitude # => 530
munich.altitude = 550
munich.altitude # => 550
munich.state # => Vapor::Persistable::DIRTY
t.rollback
}
munich.altitude # => 530
munich.state # => Vapor::Persistable::PERSISTENT

If any exception is raised inside the block, the transaction is
automatically rolled-back and the state before the transaction restored.

Alternatively transactions can be done outside a block, in which case
you must make sure yourself that the transaction is rolled back.

using an Transaction-object

t = @pmgr.transaction
t.begin
munich.make_persistent
t.commit
t.commit # => Vapor::StaleTransactionError

  • XML metadata (database independence again)

what are you storing in this xml document? I seem to achieve database
independence just fine without any xml metadata. Maybe I’m missing
something?

First, I didn’t want to “pollute” the class’s code with this metadata.
But mainly, I don’t want it to change too easily and don’t want the
developer to believe that changing the metadata definition (in-class)
automatically ajusts the Repository to it. To not need to keep the XML
files around, I store the metadata in the Datastore using an utility
that reads the XML files and populates the Repository.

  • support for relationships transparent for developer (I’m not sure if
    uhm… my library has this. If you have a an object that has an
    associated object, then this association will persist without having to
    write a single line of code. The programmer doesn’t even have to make
    any add_to_relationship(), my persistence library will notice all
    associations when insert/update is called.

Mine too. If an attribute is declared as of type “Reference” is the metadata
definition and the actual reference points to a class that include’s
Vapor::Persistable and is known to the Repository, the job gets done.

I know this one… that’s a really though one… imho rdoc helps out a
lot here :slight_smile:

The best API documentation helps nothing, if one doesn’t know which methods
to look for. I find it essential, that simple code snippets are included
as documentation. Most likely such snippets are written during API design
anyway to see how the API would feel. Why don’t just collect them ?
I’ve done so in
http://www.cip.informatik.uni-muenchen.de/~bolzer/vapor/example.html
an will eventually evolve this into a tutorial.

11

Brian Candler wrote:

snippet from my code:

def insert (dbi)
begin
dbi.do(“BEGIN”)
sql_insert (dbi)
sql_insert_array (dbi)
sql_insert_hash (dbi)
sql_insert_assoc (dbi)
dbi.do(“COMMIT”)

Or you could write:

   dbi.transaction do
     sql_insert(dbi)
     ...
   end

(which might in future work better; beware that right now ruby-dbi has no
concept of “begin transaction” so it’s rather broken in this regard)

rescue DBI::DatabaseError => dbe
dbi.do(“ROLLBACK”)
throw dbe
end

Are you sure that’s right??? I think you want to do a rollback for all
exceptions. That’s one reason why the ‘transaction do … end’ model might
be better.

uhm… what are you thinking about specifically?

···

On Sun, Jun 08, 2003 at 03:13:56PM +0900, Anders Borch wrote:

Regards,

Brian.

–
dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

12

Oliver Bolzer wrote:

Hi!

had no access to news over the weekend+holyday

Anders Borch wrote:

  • support for transactions

how is this going to show in your api? In my implementation
inserting/updating/deleting a class and all associated classes is
wrapped in a transaction. When calling insert() on a persistent class

I’m currently working on transactions with optimistic locking. (done except
rollback :slight_smile:

A code snipped would look like

Transaction as a block

@pmgr.transaction{|t|
munich.altitude # => 530
munich.altitude = 550
munich.altitude # => 550
munich.state # => Vapor::Persistable::DIRTY
t.rollback
}
munich.altitude # => 530
munich.state # => Vapor::Persistable::PERSISTENT

hmm… how are you maintaining the dirty state? using #hash at
store-time compared to #hash now? Or something similar?

If any exception is raised inside the block, the transaction is
automatically rolled-back and the state before the transaction restored.

Alternatively transactions can be done outside a block, in which case
you must make sure yourself that the transaction is rolled back.

using an Transaction-object

t = @pmgr.transaction
t.begin
munich.make_persistent
t.commit
t.commit # => Vapor::StaleTransactionError

and, if im not wrapping my “munich.altitude = 550” in a transaction will
it then be committed immediately to the database, or when you call
#insert next time? can the developer choose to autocommit an object to
database after each change? how do you detect changes?

I’m trying to find out how to implement this in a way that interferes as
little as possible with the application code. I’d hate to restrict the
developer from overriding #hash for instance.

oh, btw. if you detect that Vapor::StaleTransactionError, why not just
ignore the #commit call? If you check that an instance has been changed
before commiting it to the database, why then the error? I could of
cause rescue your exception and do nothing about it, but what purpose
does it serve in the first place? (I’m trying to determine if I should
implement something similar)

  • XML metadata (database independence again)

what are you storing in this xml document? I seem to achieve database
independence just fine without any xml metadata. Maybe I’m missing
something?

First, I didn’t want to “pollute” the class’s code with this metadata.
But mainly, I don’t want it to change too easily and don’t want the
developer to believe that changing the metadata definition (in-class)
automatically ajusts the Repository to it. To not need to keep the XML
files around, I store the metadata in the Datastore using an utility
that reads the XML files and populates the Repository.

hmmm… I’m trying to store data in my database in a way that lets the
developer change the metadata and have that reflected in the database
without having to involve the developer. Hopefully the developer will be
able to change the number of attributes and associated classes in an
object at any time and have this reflected in the database too.

  • support for relationships transparent for developer (I’m not sure if

uhm… my library has this. If you have a an object that has an
associated object, then this association will persist without having to
write a single line of code. The programmer doesn’t even have to make
any add_to_relationship(), my persistence library will notice all
associations when insert/update is called.

Mine too. If an attribute is declared as of type “Reference” is the metadata
definition and the actual reference points to a class that include’s
Vapor::Persistable and is known to the Repository, the job gets done.

no need to declare an attribute as a reference in my library. If you
have an instance variable that points to another object (well, all do,
but you know what I mean) that reference is reflected in the database
automagically.

I know this one… that’s a really though one… imho rdoc helps out a
lot here :slight_smile:

The best API documentation helps nothing, if one doesn’t know which methods
to look for. I find it essential, that simple code snippets are included
as documentation. Most likely such snippets are written during API design
anyway to see how the API would feel. Why don’t just collect them ?
I’ve done so in
http://www.cip.informatik.uni-muenchen.de/~bolzer/vapor/example.html
an will eventually evolve this into a tutorial.

You indeed did document that nicely :slight_smile:

Im trying to do this as well… I have a test.rb that I use to test my
code as I develop it. This test also serves as documentation of how to
use my library. Still there is more to documentation than examples.
Evolving from examples to tutorials takes some work (for me at least,
you seem to be doing fine).

/Anders

13

Absolutely any exception. NameError or ValueError for programming errors you
hadn’t thought about. System errors if the database suddenly becomes
unreachable or you run out of RAM - anything. If you don’t rollback changes
in those circumstances, then the half-transaction which you have already
written may get written later along with a subsequent transaction.

B.

···

On Sun, Jun 08, 2003 at 05:15:00PM +0900, Anders Borch wrote:

rescue DBI::DatabaseError => dbe
dbi.do(“ROLLBACK”)
throw dbe
end

Are you sure that’s right??? I think you want to do a rollback for all
exceptions. That’s one reason why the ‘transaction do … end’ model might
be better.

uhm… what are you thinking about specifically?

14

Anders Borch wrote:

hmm… how are you maintaining the dirty state? using #hash at
store-time compared to #hash now? Or something similar?

Couldn’t find a way to automagically detect that instance variables have
changed. The default #hash doesn’t reflect state change inside the object.
Till I find a better way, Persistable#mark_dirty has to be called on the
persistent object, either externally or from a setter of the object itself.
This is something I really want to eliminate without performance penalty.

and, if im not wrapping my “munich.altitude = 550” in a transaction will
it then be committed immediately to the database, or when you call
#insert next time? can the developer choose to autocommit an object to
database after each change? how do you detect changes?

There is an Auto-Commit mode where all changes (things like #make_persistent
and #delete_persistent and when #mark_dirty is called, for now) are
instantly flushed to the Datastore. If Autocommit-Mode is turned off and no
transaction is active, an exception is raised.

I’m trying to find out how to implement this in a way that interferes as
little as possible with the application code. I’d hate to restrict the
developer from overriding #hash for instance.

Matz is thinking to add a method to add “before” or “after”-hooks to methods
without needing to alias the original method under a different name and
then defining one’s own method that calls the original. It might make sense
to hook all methods that might change an instance variable.
I’ve thinking about redefining attr_writter, attr_accessor to call
#mark_dirty, coo.

oh, btw. if you detect that Vapor::StaleTransactionError, why not just
ignore the #commit call? If you check that an instance has been changed
before commiting it to the database, why then the error? I could of
cause rescue your exception and do nothing about it, but what purpose
does it serve in the first place? (I’m trying to determine if I should
implement something similar)

Good point. Havn’t tought this part much out yet. Maybe I should not
require that a transaction is explicty started.

hmmm… I’m trying to store data in my database in a way that lets the
developer change the metadata and have that reflected in the database
without having to involve the developer. Hopefully the developer will be
able to change the number of attributes and associated classes in an
object at any time and have this reflected in the database too.

One of my goals is to make the SQL schema as similar to the original classes
as possible. Each class has it’s own table and each attribute it’s own
columns. Because SQL has types and I want to use native SQL-types for things
like String, Integer and Float, I can’t be that flexible. I don’t want to
ALTER TABLE during normal operation. It was a requirement for this project
that manual manipulation in the RDBMS by the admin or read-only access with
traditionaly reporting tools is not too complicated.

Guess I really need to look at the SQL schema used by sql-serialize before
commenting further…

15

Brian Candler wrote:

···

On Sun, Jun 08, 2003 at 05:15:00PM +0900, Anders Borch wrote:

rescue DBI::DatabaseError => dbe
dbi.do(“ROLLBACK”)
throw dbe
end

Are you sure that’s right??? I think you want to do a rollback for all
exceptions. That’s one reason why the ‘transaction do … end’ model might
be better.

uhm… what are you thinking about specifically?

Absolutely any exception. NameError or ValueError for programming errors you
hadn’t thought about. System errors if the database suddenly becomes
unreachable or you run out of RAM - anything. If you don’t rollback changes
in those circumstances, then the half-transaction which you have already
written may get written later along with a subsequent transaction.

B.

if a nameerror, typeerror or the like happens then hopefully my program
will instantly crash and the user will scream at me until I fix it. I’m
trying to catch as few exceptions as possible so I may see where they
occur the hard way… it may just be me, but I find it easier to debug
my code if it fails the instant an exception occurs.

A.

–
dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

16

Oliver Bolzer wrote:

Anders Borch wrote:

hmm… how are you maintaining the dirty state? using #hash at
store-time compared to #hash now? Or something similar?

Couldn’t find a way to automagically detect that instance variables
have changed. The default #hash doesn’t reflect state change inside
the object. Till I find a better way, Persistable#mark_dirty has to
be called on the persistent object, either externally or from a
setter of the object itself.
This is something I really want to eliminate without performance
penalty.

and, if im not wrapping my “munich.altitude = 550” in a transaction
will it then be committed immediately to the database, or when you
call #insert next time? can the developer choose to autocommit an
object to database after each change? how do you detect changes?

There is an Auto-Commit mode where all changes (things like
#make_persistent and #delete_persistent and when #mark_dirty is
called, for now) are instantly flushed to the Datastore. If
Autocommit-Mode is turned off and no
transaction is active, an exception is raised.

I’m trying to find out how to implement this in a way that
interferes as little as possible with the application code. I’d
hate to restrict the developer from overriding #hash for instance.

Matz is thinking to add a method to add “before” or “after”-hooks to
methods without needing to alias the original method under a different
name and then defining one’s own method that calls the original. It
might make sense to hook all methods that might change an instance
variable.

that would help people like us out alot

I’ve thinking about redefining attr_writter, attr_accessor to call
#mark_dirty, coo.

oh, btw. if you detect that Vapor::StaleTransactionError, why not
just ignore the #commit call? If you check that an instance has
been changed before commiting it to the database, why then the
error? I could of cause rescue your exception and do nothing about
it, but what purpose does it serve in the first place? (I’m trying
to determine if I should implement something similar)

Good point. Havn’t tought this part much out yet. Maybe I should not
require that a transaction is explicty started.

hmmm… I’m trying to store data in my database in a way that lets
the developer change the metadata and have that reflected in the
database without having to involve the developer. Hopefully the
developer will be able to change the number of attributes and
associated classes in an object at any time and have this reflected
in the database too.

One of my goals is to make the SQL schema as similar to the original
classes as possible. Each class has it’s own table and each attribute
it’s own columns. Because SQL has types and I want to use native
SQL-types for things
like String, Integer and Float, I can’t be that flexible. I don’t
want to ALTER TABLE during normal operation. It was a requirement for
this project that manual manipulation in the RDBMS by the admin or
read-only access with
traditionaly reporting tools is not too complicated.

there are a number of data types that I could not easily port to sql.
Booleans for instance, mysql haven’t really got a boolean datatype, but
dbd_mysql returns the strings “1” or “0” in stead of true and false, so
I couldn’t store booleans as columns in the same table as the rest of
the attributes. I want, like you, to have integers, floats and strings
as columns in a table representing a class. Files, booleans, Exceptions
(and anything derived from Exception), and Bignums are some of the
datatypes I had to treat differently. I put them in separate tables with
references to the class table.

Guess I really need to look at the SQL schema used by sql-serialize
before commenting further…

I have a pseudo-EER diagram (most of my modelling knowledge is kinda
rusty after working in a company for a long time that didn’t have the
time for such stuff!!) here:

The general idea is to have a sql_ids table which holds the id and class
name (identical to the class name) of all objects stored in the
database. If an object has (for instance) an array of other objects as
an attribute, then that will be stored in the Classname_array table,
which has symbol, index, and value id as columns. value id is a foreign
key to sql_ids from which I can find the table name and column for each
object in the array. Am I making sense? The implementation is very
simple (I think so, at least) and very fleksible, but I have some
difficulty actually describing it to others.

Aside from the sql schema I have some trouble serializing a lot of the
built in classes. I’d rather not have to use Marshal to store the built
in classes (it kind of defeats the purpose of having a sql database) but
File for instance needs a path, position and open mode, which means that
it cannot be stored with any conventional sql data type. Other built in
classes (Binding for instance) are a bit harder (for me at least) to
describe and therefore to serialize. How are you accomplishing this?

/Anders

17

Oliver Bolzer wrote:

Anders Borch wrote:

hmm… how are you maintaining the dirty state? using #hash at
store-time compared to #hash now? Or something similar?

Couldn’t find a way to automagically detect that instance variables have
changed. The default #hash doesn’t reflect state change inside the object.
Till I find a better way, Persistable#mark_dirty has to be called on the
persistent object, either externally or from a setter of the object itself.
This is something I really want to eliminate without performance penalty.

and, if im not wrapping my “munich.altitude = 550” in a transaction will
it then be committed immediately to the database, or when you call
#insert next time? can the developer choose to autocommit an object to
database after each change? how do you detect changes?

There is an Auto-Commit mode where all changes (things like #make_persistent
and #delete_persistent and when #mark_dirty is called, for now) are
instantly flushed to the Datastore. If Autocommit-Mode is turned off and no
transaction is active, an exception is raised.

I’m trying to find out how to implement this in a way that interferes as
little as possible with the application code. I’d hate to restrict the
developer from overriding #hash for instance.

Matz is thinking to add a method to add “before” or “after”-hooks to methods
without needing to alias the original method under a different name and
then defining one’s own method that calls the original. It might make sense
to hook all methods that might change an instance variable.

that would help people like us out alot

I’ve thinking about redefining attr_writter, attr_accessor to call
#mark_dirty, coo.

oh, btw. if you detect that Vapor::StaleTransactionError, why not just
ignore the #commit call? If you check that an instance has been changed
before commiting it to the database, why then the error? I could of
cause rescue your exception and do nothing about it, but what purpose
does it serve in the first place? (I’m trying to determine if I should
implement something similar)

Good point. Havn’t tought this part much out yet. Maybe I should not
require that a transaction is explicty started.

hmmm… I’m trying to store data in my database in a way that lets the
developer change the metadata and have that reflected in the database
without having to involve the developer. Hopefully the developer will be
able to change the number of attributes and associated classes in an
object at any time and have this reflected in the database too.

One of my goals is to make the SQL schema as similar to the original classes
as possible. Each class has it’s own table and each attribute it’s own
columns. Because SQL has types and I want to use native SQL-types for things
like String, Integer and Float, I can’t be that flexible. I don’t want to
ALTER TABLE during normal operation. It was a requirement for this project
that manual manipulation in the RDBMS by the admin or read-only access with
traditionaly reporting tools is not too complicated.

there are a number of data types that I could not easily port to sql.
Booleans for instance, mysql haven’t really got a boolean datatype, but
dbd_mysql returns the strings “1” or “0” in stead of true and false, so
I couldn’t store booleans as columns in the same table as the rest of
the attributes. I want, like you, to have integers, floats and strings
as columns in a table representing a class. Files, booleans, Exceptions
(and anything derived from Exception), and Bignums are some of the
datatypes I had to treat differently. I put them in separate tables with
references to the class table.

Guess I really need to look at the SQL schema used by sql-serialize before
commenting further…

I have a pseudo-EER diagram (most of my modelling knowledge is kinda
rusty after working in a company for a long time that didn’t have the
time for such stuff!!) here:

The general idea is to have a sql_ids table which holds the id and class
name (identical to the class name) of all objects stored in the
database. If an object has (for instance) an array of other objects as
an attribute, then that will be stored in the Classname_array table,
which has symbol, index, and value id as columns. value id is a foreign
key to sql_ids from which I can find the table name and column for each
object in the array. Am I making sense? The implementation is very
simple (I think so, at least) and very fleksible, but I have some
difficulty actually describing it to others.

Aside from the sql schema I have some trouble serializing a lot of the
built in classes. I’d rather not have to use Marshal to store the built
in classes (it kind of defeats the purpose of having a sql database) but
File for instance needs a path, position and open mode, which means that
it cannot be stored with any conventional sql data type. Other built in
classes (Binding for instance) are a bit harder (for me at least) to
describe and therefore to serialize. How are you accomplishing this?

/Anders

18

Quoteing spam@deck.dk, on Mon, Jun 09, 2003 at 03:56:46AM +0900:

hadn’t thought about. System errors if the database suddenly becomes
unreachable or you run out of RAM - anything. If you don’t rollback changes
in those circumstances, then the half-transaction which you have already
written may get written later along with a subsequent transaction.

if a nameerror, typeerror or the like happens then hopefully my program
will instantly crash and the user will scream at me until I fix it. I’m
trying to catch as few exceptions as possible so I may see where they
occur the hard way… it may just be me, but I find it easier to debug
my code if it fails the instant an exception occurs.

You can rethrow the exception, after rolling back the transaction, can’t you?

Sam

19

if a nameerror, typeerror or the like happens then hopefully my program
will instantly crash and the user will scream at me until I fix it.

Not if sql-serialize is meant to be a general-purpose library. The main
program very possibly has an exception catcher which formats or logs the
error and then lets the user continue.

If sql-serialize is just your own self-contained application, of course, you
can choose to have it work in the way you describe.

I’m
trying to catch as few exceptions as possible so I may see where they
occur the hard way… it may just be me, but I find it easier to debug
my code if it fails the instant an exception occurs.

You just catch the exception and re-raise it, exactly as you showed (except
‘rescue Exception => e’ not a DBI::Exception).

Regards,

Brian.

···

On Mon, Jun 09, 2003 at 03:56:46AM +0900, Anders Borch wrote:

20

postgresql has a boolean datatype and supports user-defined datatypes,
if you’re not locked in to mysql for your application.

Regards,

Mark

···

On Thursday, June 19, 2003, at 11:45 AM, Anders Borch wrote:

[snip]

there are a number of data types that I could not easily port to sql.
Booleans for instance, mysql haven’t really got a boolean datatype, but
dbd_mysql returns the strings “1” or “0” in stead of true and false, so
I couldn’t store booleans as columns in the same table as the rest of
the attributes. I want, like you, to have integers, floats and strings
as columns in a table representing a class. Files, booleans, Exceptions
(and anything derived from Exception), and Bignums are some of the
datatypes I had to treat differently. I put them in separate tables
with
references to the class table.
[snip]