This is the first release of Madeleine. Later releases will have
features like transparent command generation and clustering, but for
now the focus is a simple and well-designed implementation of the
basics.
Requirements:
Ruby 1.7/1.8 (uses File#fsync which isn’t in 1.6)
Known problems:
Doesn’t work in Windows yet.
Short on documentation. You’ll have to do with Prevayler’s
docs for now.
This is the first release of Madeleine. Later releases will have
features like transparent command generation and clustering, but for
now the focus is a simple and well-designed implementation of the
basics.
“Mnemonic is object prevalence layer for Ruby. It provides a persistence mechanism that is orders of magnitude FASTER and SIMPLER
than a DBMS. It is based on the work of the Prevayler project at http://www.prevayler.org”
Small suggestion: take class ‘Marshal’ as a parameter. Then you can choose a
different Marshal-type class like ‘YAML’. Although right now YAML is slow,
this would give a language-independent database which you could even edit
with ‘vi’ (great for data recovery). It might end up containing a lot of
spaces, but it should gzip very well.
Note to WHY: please can we have a YAML::dump(object [,IO]) method, for
symmetry with Marshal?
For testing I just replaced ‘Marshal’ with ‘YAML’, added the following:
require ‘yaml’
module YAML
def self.dump(obj,stream)
stream.write(obj.to_yaml)
end
end
This gives a database which looks like this:
$ cat dictionary-base/000000000000000000001.snapshot
— !ruby/object:Dictionary
data:
foo: the opposite of bar
bar: the opposite of foo
and a logfile like this:
$ cat dictionary-base/000000000000000000001.command_log
— !ruby/object:Addition
key: baz
value: the opposite of foobar
Having such readable files could go a long way towards addressing the
resistance which may arise towards treating your entire database of (say)
a million customer records as a single opaque binary blob…
Cheers,
Brian.
P.S. Another note to WHY: the last line of YAML output doesn’t seem to
contain a trailing newline, which is just a slight nuisance.
···
On Mon, Mar 17, 2003 at 07:00:35AM +0900, Anders Bengtsson wrote:
This is the first release of Madeleine. Later releases will have
features like transparent command generation and clustering, but for
now the focus is a simple and well-designed implementation of the
basics.
“Mnemonic is object prevalence layer for Ruby. It provides a persistence mechanism that is orders of magnitude FASTER and SIMPLER
than a DBMS. It is based on the work of the Prevayler project at http://www.prevayler.org”
Madeleine is an alive project and the author is responding to emails,
that’s the biggest difference at this point.
The big feature difference at this point is that Mnemonic has a
(broken!) implementation of automatic commands.
/Anders
···
–
A n d e r s B e n g t s s o n | ndrsbngtssn@yahoo.se
Stockholm, Sweden |
Small suggestion: take class ‘Marshal’ as a parameter. Then you can choose
a different Marshal-type class like ‘YAML’. Although right now YAML is
slow, this would give a language-independent database which you could even
edit with ‘vi’ (great for data recovery). It might end up containing a lot
of spaces, but it should gzip very well.
Brilliant.
Note to WHY: please can we have a YAML::dump(object [,IO]) method, for
symmetry with Marshal?
Yeah. Will be in YAML.rb 0.60.
P.S. Another note to WHY: the last line of YAML output doesn’t seem to
contain a trailing newline, which is just a slight nuisance.
Okay, fair enough. I’ll release YAML.rb 0.50 with this change, 0.60 as well.
_why
···
On Monday 17 March 2003 06:54 am, Brian Candler wrote:
Small suggestion: take class ‘Marshal’ as a parameter. Then you can
choose a
different Marshal-type class like ‘YAML’. Although right now YAML is
slow,
this would give a language-independent database which you could even
edit
with ‘vi’ (great for data recovery). It might end up containing a lot
of
spaces, but it should gzip very well.
Good idea. I like the YAML concept a lot.
Note to WHY: please can we have a YAML::dump(object [,IO]) method,
for
symmetry with Marshal?
I could just add wrapping classes that wrap either Marshal or YAML, so
a change in YAML won’t be necessary. Unless it’s a good idea for YAML
anyway?
For testing I just replaced ‘Marshal’ with ‘YAML’, added the
following:
[readable data example snipped]
Having such readable files could go a long way towards addressing the
resistance which may arise towards treating your entire database of
(say)
a million customer records as a single opaque binary blob…
There has been some talk about XML serialization on the Prevayler
lists, which gives the advantage of using XSL to upgrade the data
format. But I hope YAML’s closer integration with Ruby itself makes
upgrading even simpler.
Cheers,
Brian.
/Anders
···
=====
Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden
I’ve also started benchmarking Syck against the current YAML parser and things
are getting speedy. Here’s loading a comparable XML document 100 times in
REXML, Marshal, Syck, and YAML.rb:
Excellent. Now just to make it dump at that speed as well
···
On Mon, Mar 17, 2003 at 11:57:07PM +0900, why the lucky stiff wrote:
I’ve also started benchmarking Syck against the current YAML parser and things
are getting speedy. Here’s loading a comparable XML document 100 times in
REXML, Marshal, Syck, and YAML.rb:
Note to WHY: please can we have a YAML::dump(object [,IO]) method,
for
symmetry with Marshal?
I could just add wrapping classes that wrap either Marshal or YAML, so
a change in YAML won’t be necessary. Unless it’s a good idea for YAML
anyway?
I don’t think your code should have anything YAML-specific, because there
are many other marshalling modules you might want to use (SOAP being another
example). I think it should be up to the end-user to wrap them so that there
is a Foo::load and Foo::dump, if they don’t have them already.
There has been some talk about XML serialization on the Prevayler
lists, which gives the advantage of using XSL to upgrade the data
format. But I hope YAML’s closer integration with Ruby itself makes
upgrading even simpler.
And XML serialisation means SOAP, and if you want to use XSL to apply
on-the-fly changes to SOAP object structure, well good luck to you
The point about the Prevayler approach is that your whole database is in RAM
so if you want to make wholesale changes to its format, you just read it
in, modify it using code, and write it back out again.
Regards,
Brian.
···
On Mon, Mar 17, 2003 at 11:37:56PM +0900, Anders Bengtsson wrote:
Madeleine is an alive project and the author is responding to emails,
that’s the biggest difference at this point.
The big feature difference at this point is that Mnemonic has a
(broken!) implementation of automatic commands.
I wonder in what way are Mnemonic’s automatic commands broken?
I’d like to know because I am actively using it, and I don’t seem to
have any problems.
I do find that I need to use my own method wrapper module for objects
that are contained in my persisted class, so that changes in these
objects are also persisted. But it is quite easy to intercept method
calls to these objects, I just need to include my little module in
each class.
Are you planning to implement some kind of automatic commands?
I don’t think your code should have anything YAML-specific, because
there
are many other marshalling modules you might want to use (SOAP being
another
example). I think it should be up to the end-user to wrap them so
that there
is a Foo::load and Foo::dump, if they don’t have them already.
I’ll probably ship a few out-of-the-box wrappers for different
marshalling implementations with Madeleine and at the same time give
the user the option to write their own. This won’t really affect the
core parts of Madeleine.
I already have an option to provide different command loggers, by
giving a logger-factory as an argument.
And XML serialisation means SOAP
Does it?
, and if you want to use XSL to apply
on-the-fly changes to SOAP object structure, well good luck to you
I was more thinking in terms of how to migrate a snapshot of one
version of a program to a newer version, where the class structure has
changed. Nothing on-the-fly, really.
I wonder in what way are Mnemonic’s automatic commands broken?
I’d like to know because I am actively using it, and I don’t seem to
have any problems.
I do find that I need to use my own method wrapper module for objects
that are contained in my persisted class, so that changes in these
objects are also persisted. But it is quite easy to intercept method
calls to these objects, I just need to include my little module in
each class.
Hi Haldane,
You kind of answered your own question here. With one exception, the
root class of the system, the automatic commands are broken.
Are you planning to implement some kind of automatic commands?
Yes. There is a nice implementation for Java included as a sample in
the Aspect Oriented Programming project “Nanning”, that I intended to
use as the model for Madeleine’s design. Of course, in Ruby you can do
things directly in the language that in Java requires tools like AOP.
Lovely language, this.
Could you share your method wrapping code? It seems to be something
like what I intended to write.
/Anders
···
=====
Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden
From what poking around I’ve managed to do, I don’t think there is any determination on what Prevayler XML serialization would look
like. And, as has been discussed on this list numerous times before, there are many ways besides SOAP to use XML for serialization.
, and if you want to use XSL to apply
on-the-fly changes to SOAP object structure, well good luck to you
SOAP or not, XPath queries would be nice. Or using an XForms UI to manage objects.
Anyway, XSLT on SOAP is not rocket science.
I was more thinking in terms of how to migrate a snapshot of one
version of a program to a newer version, where the class structure has
changed. Nothing on-the-fly, really.
Interesting idea.
James
···
/Anders
=====
Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden
I’ve not come across anything else which does the job. It would have to be
able to cope with cyclic graphs of object references, for example. Are the
Prevayler people really thinking of inventing a new standard for serialising
objects to XML?
Regards,
Brian.
···
On Tue, Mar 18, 2003 at 01:18:09AM +0900, Anders Bengtsson wrote:
but actually it’s not compatible with Marshal, because:
you can’t pass an IO object to load, it must be a string
you can’t pass an IO object as the second parameter to dump
(it takes a mappingRegistry instead)
I think that’s a weakness of soap4r, because if you have a 100MB database,
it forces you to create (say) 400MB of XML, and then write it to a file.
Suddenly you need 5 times as much RAM as you did originally, and also
concatenation of large strings is inefficient in Ruby 1.6.
For the purposes of a database, I think it’s reasonable for you to require
dump(obj,IO) and load(IO).
Regards,
Brian.
···
On Tue, Mar 18, 2003 at 01:18:09AM +0900, Anders Bengtsson wrote:
I don’t think your code should have anything YAML-specific, because
there
are many other marshalling modules you might want to use (SOAP being
another
example). I think it should be up to the end-user to wrap them so
that there
is a Foo::load and Foo::dump, if they don’t have them already.
I’ll probably ship a few out-of-the-box wrappers for different
marshalling implementations with Madeleine and at the same time give
the user the option to write their own. This won’t really affect the
core parts of Madeleine.
Could you share your method wrapping code? It seems to be something
like what I intended to write.
Um, sure. Except that just looking closely at this part of my system
(which I am not using for anything important) I notice it is utterly
broken. The problem is that the persistence layer needs to remember
which objects the commands were for exactly because the object ids
will be different next time they are Marshal.load -ed. And there is
no way of doing that implemented at present in Mnemonic.
This is one way to wrap method calls anyway. Note that I am careful
to not make any singleton methods - these cannot be marshaled.
Need recent Ruby
Command module - sends any method calls through the persisted object
note that include Command should be the last line (not first) in
your class
module Command
class <<self
def included(klass)
klass.instance_methods.each {|m|
eval %{
klass.class_eval {
alias_method :cmdnew_#{m}, :#{m}
def #{m}(*args, &block)
Persisted.thing.execute(self.id, :cmdnew_#{m}, *args,
&block)
end
}
}
}
end
end
end
Assuming you have a persisted thing like this:
this is the persisted object
all commands come through here
class Thing
#…
def execute(objid, method, *args, &block)
ObjectSpace._id2ref(objid).send(method, *args, &block)
end
end
this class is for convenience only - could use a global variable too
class Persisted
include Singleton
attr_reader :x
def initialize @x = Mnemonic.new(Thing.new, “thingstore”, “execute”)
end
def Persisted.thing
Persisted.instance.x
end
end
Any classes that include Command at the bottom will have all their
method calls persisted.
Except, as I say, it won’t work because we don’t know which objects
were which the next time we load up the commands. Creating objects is
not deterministic - they come with a different id each time.
For Prevayler, there’s just an interface for plugging in your own
logging and snapshot stuff, nothing more really.
There has been a lot of talk about different formats, but I don’t think
I’ve seen SOAP mentioned.
Anyway, it’s not that big an issue. Unless you want to use external
tools on the snapshot, for whatever reason, the default
serialization/marshalling will do.
/Anders
···
On 2003-03-17 Brian Candler wrote:
And XML serialisation means SOAP
Does it?
I’ve not come across anything else which does the job. It would have to be
able to cope with cyclic graphs of object references, for example. Are the
Prevayler people really thinking of inventing a new standard for serialising
objects to XML?
–
A n d e r s B e n g t s s o n | ndrsbngtssn@yahoo.se
Stockholm, Sweden |
From what poking around I’ve managed to do, I don’t think
there is any determination on what Prevayler XML
serialization would look
like. And, as has been discussed on this list numerous times
before, there are many ways besides SOAP to use XML for serialization.
Once upon a time in 1999, he looked for XML serialization
format instead of using CSV or proprietary text format for
his product. He found several candidates; XML-RPC, WDDX,
XMOP, or so but only SOAP seems to be promising for him.
XMOP seems too COMed. Other formats is too simple for
serialization. Is SOAP Envelop complicated? It’s only an
envelope. Is SOAP Encoding complicated? It’s only
WXS Part 2 + array + struct. He thought that anyone cannot
simplify SOAP’s array and struct anymore (SOAP/1.2 simplified
the array though. sigh.)
Back to the topic, though I know nothing about Preveyler
now, SOAP can be used for the serialization format or not,
depends on Preveyler’s Data model. SOAP Data Model is
“graph of typed node”. Table data model does not fit SOAP.
I think this could be done with an ordinary shell, using xargs maybe,
but I know how to do it in Ruby. If you have a more elegant way to
write the Ruby lines, let me know. Looking at it, I can see that if I
remembered the directory methods I could make the first 3 lines one
line.
