Metakit for Ruby - Would you want it?

I have a gentleman in England who I have been talking with who is
interested in creating bindings for Metakit to the Ruby language. He
has been hesitant because the Metakit list group has been silent for
the most part on support. So I thought I would throw it out here.

Metakit is a nice embedded small footprint database.

Go here to read up on it: http://www.equi4.com/metakit/

Respond here in the group.

He is on Linux and will be using GCC and I will be using MingW. If you
have VC++ and would like to do a native port let me know as well.

Bob

i am extremely interested in this. how many time i have wished one could
share data from a pstore or yaml with c and c++… i like sqlite a bit, but
the ruby binding is incomplete. this would be a very valuable addition to
ruby IMHO. personally, i love writing cgi programs to use embedded databases
because you can move a program to another host using tar cvfz/xvfz - much
easier than a dozen sql scripts…

-a

···

On Thu, 29 May 2003, Bob wrote:

I have a gentleman in England who I have been talking with who is
interested in creating bindings for Metakit to the Ruby language. He
has been hesitant because the Metakit list group has been silent for
the most part on support. So I thought I would throw it out here.

Metakit is a nice embedded small footprint database.

Go here to read up on it: Metakit embedded database library

Respond here in the group.

He is on Linux and will be using GCC and I will be using MingW. If you
have VC++ and would like to do a native port let me know as well.

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================

ahoward wrote:

I have a gentleman in England who I have been talking with who is
interested in creating bindings for Metakit to the Ruby language. He
has been hesitant because the Metakit list group has been silent for
the most part on support. So I thought I would throw it out here.

Metakit is a nice embedded small footprint database.

Go here to read up on it: Metakit embedded database library

Respond here in the group.

He is on Linux and will be using GCC and I will be using MingW. If you
have VC++ and would like to do a native port let me know as well.

i am extremely interested in this. how many time i have wished one could
share data from a pstore or yaml with c and c++… i like sqlite a bit, but
the ruby binding is incomplete. this would be a very valuable addition to
ruby IMHO. personally, i love writing cgi programs to use embedded databases
because you can move a program to another host using tar cvfz/xvfz - much
easier than a dozen sql scripts…

-a

what I would like is an embedded sql database. It seems to me that
metakit does not have an sql parser. having an sql parser would let me
make a dbi interface and use an embedded database w/o changing a single
line of code.

does anyone know of an embedded sql database? preferrebly with a simple
api so I could easily make a ruby interface to it (or even better: one
that already has a ruby interface).

/Anders

···

On Thu, 29 May 2003, Bob wrote:


dc -e 4ddod3ddn1-0nn1dnd+nn3dn1+n3*1+ddn1+dn2-n5dn1+dnrn1-p | tr
0123456 yorh@k. | dd conv=lcase

You mean like sqlite? http://www.sqlite.com/

There are ruby bindings which work more or less (get the latest ruby-dbi-all
from CVS)

However, IMO, SQL makes for a dreadful API. I have been writing some code
which I try to run with sqlite (for Unit testing), mysql (for a staging
area) and Oracle (for live production). It’s been a complete nightmare. It’s
all just so non-standardised when it gets down to the nitty gritty.

If you just want objects which persist, and your database isn’t so enormous,
you might like to take a completely different approach (e.g. madeleine)

Regards,

Brian.

···

On Fri, May 30, 2003 at 04:02:07AM +0900, Anders Borch wrote:

what I would like is an embedded sql database. It seems to me that
metakit does not have an sql parser. having an sql parser would let me
make a dbi interface and use an embedded database w/o changing a single
line of code.

what I would like is an embedded sql database. It seems to me that
metakit does not have an sql parser. having an sql parser would let me
make a dbi interface and use an embedded database w/o changing a single
line of code.

I’m also interested in this. I suppose the db code is all
part of the app, and the database data is more or less a
single file? That’s my idea of small footprint.

You mean like sqlite? http://www.sqlite.com/

There are ruby bindings which work more or less (get the latest
ruby-dbi-all
from CVS)

Hmm. Can it do record-level locking? E.g., suppose multiple
threads or processes are accessing the database?

However, IMO, SQL makes for a dreadful API. I have been writing some code
which I try to run with sqlite (for Unit testing), mysql (for a staging
area) and Oracle (for live production). It’s been a complete nightmare.
It’s
all just so non-standardised when it gets down to the nitty gritty.

Ha… I once did ANSI compliance testing for a SQL implementation.
Boy, that was entertaining. At that time, there were as many errors
in the test suite as in the database code.

I’ve toyed with the idea of “RQL” (Ruby Query Language) which would be
similar in spirit to SQL but allow things like “MATCHES regex” and
so on.

But that’s something that will never actually materialize. There are
a dozen higher-priority things on my list, many of which will also
never get implemented. :slight_smile:

Cheers,
Hal

···

----- Original Message -----
From: “Brian Candler” B.Candler@pobox.com
To: “ruby-talk ML” ruby-talk@ruby-lang.org
Sent: Thursday, May 29, 2003 2:11 PM
Subject: Re: Metakit for Ruby - Would you want it?

On Fri, May 30, 2003 at 04:02:07AM +0900, Anders Borch wrote:

Brian Candler wrote:

what I would like is an embedded sql database. It seems to me that
metakit does not have an sql parser. having an sql parser would let me
make a dbi interface and use an embedded database w/o changing a single
line of code.

You mean like sqlite? http://www.sqlite.com/

yeah! exactly like sqlite! you just saved my day :slight_smile:

There are ruby bindings which work more or less (get the latest ruby-dbi-all
from CVS)

and it even has a DBI interface! this is so cool! :smiley:

However, IMO, SQL makes for a dreadful API. I have been writing some code
which I try to run with sqlite (for Unit testing), mysql (for a staging
area) and Oracle (for live production). It’s been a complete nightmare. It’s
all just so non-standardised when it gets down to the nitty gritty.

sadly sql isn’t standardized at all (in practice), but that does not
change the fact that everyone and their mothers speak some subset of
sql. Therefore if anyone but me is going to maintain and develop for
systems that I made, then I need to use stuff they know - like sql in
stead of some odd db interface… also half of the time i’m building
software using ruby and storing data in some sql database and someone
else is writing some other software in java that works with the same
data. If I am to work together with other people across languages it’s
often better to have a common ground that is an sql database.

If you just want objects which persist, and your database isn’t so enormous,
you might like to take a completely different approach (e.g. madeleine)

yeah, I have been looking at a some of those solutions… db-backend was
the one I was most impressed with. It was also the one most likely to
fulfill my needs. I would like to have a nice database backend stucture
(as made possible in db-backend) combined with ease of use (as in some
of the non-sql variants out there).

db-backend makes it possible to have a nice database structure but the
price is that I need to define the database to ruby object mappings.
madeleine (and the like - i looked at a couple of those) lets me easily
have persistent objects but I cannot easily interact with the java
people whom I would like to share my objects with.

So, in the spirit of so many other open source developers, I did not
expand upon existing code, but in stead I started my own project. I aim
to have easily persisten objects and at the same time have an easily
manageable database structure.

The project is awailable at http://rsqlserial.sf.net (and at RAA -
search for sql and it should be among the top 10 hits).

I ended up rather off-topic and with a not very subtle plug for my own
(still alpha) project. But hey - I got my sqlite now!

In an effort to get back on topic: more embedded databases can only
benefit us all. And for those who - like Brian - do not like sql,
metakit looks very nice indeed. From what I read of the API it seems
very nice and sleek. Go ahead and make a ruby binding for it :slight_smile:

Regards,

Brian.

/Anders

···

On Fri, May 30, 2003 at 04:02:07AM +0900, Anders Borch wrote:

You mean like sqlite? http://www.sqlite.com/

There are ruby bindings which work more or less (get the latest
ruby-dbi-all
from CVS)

Hmm. Can it do record-level locking? E.g., suppose multiple
threads or processes are accessing the database?

You can run it safely with multiple threads/processes, but it locks the
whole file (i.e. the whole database) when it needs to.

This leads to a nasty behaviour with DBI and AutoCommit. If you have
AutoCommit set to false (which is the default), then the sqlite DBD handles
this by immediately issuing a “begin transaction” as soon as you open the
database. When you do a commit it issues “commit” followed immediately by
another “begin transaction”. This has the very nasty side-effect of keeping
the whole database locked, so it can only be accessed from a single thread.

You can turn AutoCommit on, but many databases don’t handle transactions
properly in that case, because there is no ‘begin’ method in the DBD
interface - only ‘commit’ and ‘rollback’. In fact, ruby-dbi is (in my
opinion) seriously broken when it comes to transactions. It’s a shame,
because this is one of the absolutely key pieces of infrastructure for many
applications.

Ruby-dbi is just a wrapper around the ‘native’ APIs for each of the
databases of course, so there’s nothing to stop you using say the mysql or
postgresql APIs directly, if you don’t mind hard-wiring your code to that
API. But sqlite has only a DBD as far as I know.

I’ve toyed with the idea of “RQL” (Ruby Query Language) which would be
similar in spirit to SQL but allow things like “MATCHES regex” and
so on.

Yeah, but how would you implement it as a layer on top of an SQL database?
This is just one of the problems I came across:

  • In Oracle, “like” is case-sensitive (only)
  • In Sqlite, “like” is case-insensitive (only)
  • In Mysql, you have the choice of “like” or “ilike” operators

It’s pretty much impossible to layer consistent semantics on top of all
three databases. I have had to approximate. In the case of Oracle, I have to
issue queries of the form

 "foo like ? or foo like ? or foo like ? or foo like ?",
     var, var.downcase, var.upcase, var.capitalize

It’s awful. Even worse is date and time handling.

Cheers,

Brian.

···

On Fri, May 30, 2003 at 04:34:18AM +0900, Hal E. Fulton wrote:

no - i think it uses flock or similar. there mailing list is quite active and
helpful though (and the developer is super active on it) so you might want to
post there too.

-a

···

On Fri, 30 May 2003, Hal E. Fulton wrote:

You mean like sqlite? http://www.sqlite.com/

There are ruby bindings which work more or less (get the latest
ruby-dbi-all
from CVS)

Hmm. Can it do record-level locking? E.g., suppose multiple
threads or processes are accessing the database?

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================

Brian Candler wrote:

what I would like is an embedded sql database. It seems to me that
metakit does not have an sql parser. having an sql parser would let me
make a dbi interface and use an embedded database w/o changing a single
line of code.

You mean like sqlite? http://www.sqlite.com/

yeah! exactly like sqlite! you just saved my day :slight_smile:

There are ruby bindings which work more or less (get the latest ruby-dbi-all
from CVS)

and it even has a DBI interface! this is so cool! :smiley:

i had problems with the api - if the api returned an error code the connection
went stale and all sorts of trouble ensued… eg.

#!/usr/bin/env ruby
require ‘sqlite’

db = SQLite.new(‘bar.db’)

sql = “create table bar (answer int);”
db.exec (sql)

sql = “insert into bar values (42);”
#sql = “insert into foobar values (42);” # ERROR
db.exec (sql)

query = “select * from bar;”
tuples = db.exec (query)

when run produces

[answer]
42

but if you use the line marked ‘ERROR’ instead it gives

bar.rb:7:in `exec’: method call on terminated object (NotImplementedError)
from bar.rb:7

sadly sql isn’t standardized at all (in practice), but that does not
change the fact that everyone and their mothers speak some subset of
sql. Therefore if anyone but me is going to maintain and develop for
systems that I made, then I need to use stuff they know - like sql in
stead of some odd db interface… also half of the time i’m building
software using ruby and storing data in some sql database and someone
else is writing some other software in java that works with the same
data. If I am to work together with other people across languages it’s
often better to have a common ground that is an sql database.

If you just want objects which persist, and your database isn’t so enormous,
you might like to take a completely different approach (e.g. madeleine)

yeah, I have been looking at a some of those solutions… db-backend was
the one I was most impressed with. It was also the one most likely to
fulfill my needs. I would like to have a nice database backend stucture
(as made possible in db-backend) combined with ease of use (as in some
of the non-sql variants out there).

db-backend makes it possible to have a nice database structure but the
price is that I need to define the database to ruby object mappings.
madeleine (and the like - i looked at a couple of those) lets me easily
have persistent objects but I cannot easily interact with the java
people whom I would like to share my objects with.

So, in the spirit of so many other open source developers, I did not
expand upon existing code, but in stead I started my own project. I aim
to have easily persisten objects and at the same time have an easily
manageable database structure.

The project is awailable at http://rsqlserial.sf.net (and at RAA -
search for sql and it should be among the top 10 hits).

I ended up rather off-topic and with a not very subtle plug for my own
(still alpha) project. But hey - I got my sqlite now!

In an effort to get back on topic: more embedded databases can only
benefit us all. And for those who - like Brian - do not like sql,
metakit looks very nice indeed. From what I read of the API it seems
very nice and sleek. Go ahead and make a ruby binding for it :slight_smile:

very well put - and totally correct IMHO. i will check out your project.

-a

···

On Thu, 29 May 2003, Anders Borch wrote:

On Fri, May 30, 2003 at 04:02:07AM +0900, Anders Borch wrote:

====================================
Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================

> Yeah, but how would you implement it as a layer on top of an SQL database? > This is just one of the problems I came across: > > - In Oracle, "like" is case-sensitive (only) > - In Sqlite, "like" is case-insensitive (only) > - In Mysql, you have the choice of "like" or "ilike" operators > > It's pretty much impossible to layer consistent semantics on top of all > three databases. I have had to approximate. In the case of Oracle, I have to > issue queries of the form > > "foo like ? or foo like ? or foo like ? or foo like ?", > var, var.downcase, var.upcase, var.capitalize > > It's awful. Even worse is date and time handling.

i fell your pain - and my application is nothing but data and time handling
which is why i’m so anti dbi. in my mind dbi (or similar) give you as many
headaches with none of the perks (since you can’t take advantage of db
specific feature w/o losing portability) - sort of like writing OO code in c++
:wink: genericity is, all too often, evil.

-a

···

On Fri, 30 May 2003, Brian Candler wrote:

====================================

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================

I also found a couple of nasty bugs, which to be fair were fixed pretty soon
after I reported them (which is why I suggested using the CVS version). If
you have a script which demonstrates a bug you could try posting it to the
ruby-dbi-devel mailing list - linked from ruby-dbi.sourceforge.net

I think the ruby-dbi people are a bit busy to work on it much though. This
is an area that I think could do with some extra hands if anyone’s
interested.

Cheers,

Brian.

···

On Fri, May 30, 2003 at 06:43:59AM +0900, ahoward wrote:

yeah! exactly like sqlite! you just saved my day :slight_smile:

There are ruby bindings which work more or less (get the latest ruby-dbi-all
from CVS)

and it even has a DBI interface! this is so cool! :smiley:

i had problems with the api - if the api returned an error code the connection
went stale and all sorts of trouble ensued… eg.

ahoward wrote:
[snip]

very well put - and totally correct IMHO. i will check out your project.

I will make an effort to have it in a usable state asap :slight_smile:

···


dc -e
4ddod3dddn1-89danrn10-dan3+ann6dan2an13dn1+dn2-dn3+5ddan2/9+an13nap

Brian Candler wrote:

  • In Oracle, “like” is case-sensitive (only)
  • In Sqlite, “like” is case-insensitive (only)
  • In Mysql, you have the choice of “like” or “ilike” operators

It’s pretty much impossible to layer consistent semantics on top of all
three databases. I have had to approximate. In the case of Oracle, I have to
issue queries of the form

 "foo like ? or foo like ? or foo like ? or foo like ?",
     var, var.downcase, var.upcase, var.capitalize

I know this is offtopic, but you could probably use “UPPER(foo) LIKE
UPPER(?)” on all 3 databases… :slight_smile:

Regards,

Peter

To be fair, it’s often the databases themselves which are broken. I was
shocked when I discovered that Oracle’s DATE type (which actually stores a
date+time to the second) stores a YYYYMMDDhhmmss value without any concept
of a timezone. So if someone writes an app which does

 insert into foo (timestamp) values (sysdate);

then what you get is a local time value stored. For anything other than
mickey-mouse applications this is pretty useless.

The DBI layer in principle could at least abstract a BCD timestamp
YYYYMMDDhhmmss[.sss] into a consistent form, and leave it up to the
application to deal with the problems of timezones. But I haven’t seen this
done properly. In particular, when you insert into a date column, you
generally end up having to format dates to match what the database expects.
That, I think, is the fault of the SQL spec, or lack of it; ultimately the
SQL query has to be an ASCII string to send to the database.

So, if you’re writing an application from scratch, IMO timestamps are much
better just stored as INTEGER values containing Unix UTC values.
Unfortunately, you often have to interface with an existing broken database
design :frowning:

Regards,

Brian.

···

On Fri, May 30, 2003 at 06:44:28AM +0900, ahoward wrote:

It’s awful. Even worse is date and time handling.

i fell your pain - and my application is nothing but data and time handling
which is why i’m so anti dbi. in my mind dbi (or similar) give you as many
headaches with none of the perks (since you can’t take advantage of db
specific feature w/o losing portability) - sort of like writing OO code in c++
:wink: genericity is, all too often, evil.

Thanks, not a bad idea, although I imagine that Oracle would not use the
indexes at all in that case, which for a million-record database is a bit of
a problem.

What I’d really like is foo RLIKE ‘[hH][eE][lL][lL][oO]’

(which I think mysql has, but I couldn’t find anything in Oracle like that)

Cheers,

Brian.

···

On Mon, Jun 02, 2003 at 06:41:38AM +0900, Nospam wrote:

Brian Candler wrote:

  • In Oracle, “like” is case-sensitive (only)
  • In Sqlite, “like” is case-insensitive (only)
  • In Mysql, you have the choice of “like” or “ilike” operators

It’s pretty much impossible to layer consistent semantics on top of all
three databases. I have had to approximate. In the case of Oracle, I have
to
issue queries of the form

"foo like ? or foo like ? or foo like ? or foo like ?",
    var, var.downcase, var.upcase, var.capitalize

I know this is offtopic, but you could probably use “UPPER(foo) LIKE
UPPER(?)” on all 3 databases… :slight_smile:

To be fair, it’s often the databases themselves which are broken. I was
shocked when I discovered that Oracle’s DATE type (which actually stores a
date+time to the second) stores a YYYYMMDDhhmmss value without any concept
of a timezone. So if someone writes an app which does

 insert into foo (timestamp) values (sysdate);

then what you get is a local time value stored. For anything other than
mickey-mouse applications this is pretty useless.

wow. yet another reason to hate oralce

The DBI layer in principle could at least abstract a BCD timestamp
YYYYMMDDhhmmss[.sss] into a consistent form, and leave it up to the
application to deal with the problems of timezones. But I haven’t seen this
done properly. In particular, when you insert into a date column, you
generally end up having to format dates to match what the database expects.
That, I think, is the fault of the SQL spec, or lack of it; ultimately the
SQL query has to be an ASCII string to send to the database.

i use postgresql - it is insanely good with timestamps:

#!/usr/bin/env ruby
require ‘postgres’

c = PGconn.new

times = [
‘epoch’, ‘-infinity’, ‘infinity’,
‘2003-02-01’, ‘2003-02-01 00:01:01’,
Time.now, Time.now.to_s,
]

sql = <<-sql
drop table foobar;
create table foobar (t timestamp(0) without time zone)
sql
c.exec sql

sql = <<-sql
insert into foobar values (‘%s’)
sql
times.map{|t| c.exec sql % [t]}

sql = <<-sql
select * from foobar
sql
p c.query sql

~ > ruby pg.rb
[[“1970-01-01 00:00:00”], [“-infinity”], [“infinity”], [“2003-02-01 00:00:00”],
[“2003-02-01 00:01:01”], [“2003-05-29 23:50:46”], [“2003-05-29 23:50:46”]]

So, if you’re writing an application from scratch, IMO timestamps are much
better just stored as INTEGER values containing Unix UTC values.
Unfortunately, you often have to interface with an existing broken database
design :frowning:

i know what you mean - but i prefer the iso-8601 fmt:

‘%Y-%m-%d %H:%M:%S’

because

  • sorts corectly even as a string
  • human readable
  • just about any time parser understands it - if not one is quite easy to
    write in any lang

-a

···

On Fri, 30 May 2003, Brian Candler wrote:

====================================

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================

[snip]

What I’d really like is foo RLIKE ‘[hH][eE][lL][lL][oO]’

(which I think mysql has, but I couldn’t find anything in Oracle like
that)

What I’d really like is a new db with a proprietary
SQL version, for Ruby. They’re all somewhat incompatible
anyway, right?

Then I’d do it like:

…where foo matches /hello/i

Hal

···

----- Original Message -----
From: “Brian Candler” B.Candler@pobox.com
To: “ruby-talk ML” ruby-talk@ruby-lang.org
Sent: Monday, June 02, 2003 1:30 PM
Subject: Re: [OT] Re: Metakit for Ruby - Would you want it?

Brian Candler wrote:

Thanks, not a bad idea, although I imagine that Oracle would not use the
indexes at all in that case, which for a million-record database is a bit of
a problem.

That’s why Oracle let you create function based indexes, e.g. you can
create an index on UPPER(foo). :slight_smile: But this is going probably way off
topic now :).

Regards,

Peter

If you’re going The Ruby Way, why not

table.select { |row| row.foo =~ /hello/i }

:slight_smile:

Pit

···

On 3 Jun 2003 at 3:36, Hal E. Fulton wrote:

What I’d really like is a new db with a proprietary
SQL version, for Ruby. They’re all somewhat incompatible
anyway, right?

Then I’d do it like:

…where foo matches /hello/i

or even:

relation.select(:field0, :field1){|tuple| tuple[:field0, :field1] === [/foo/, /bar/]}

i actually have something which does the above using named fields in arrays.
it’s very cool but slower than dirt :wink:

-a

···

On Tue, 3 Jun 2003, Pit Capitain wrote:

If you’re going The Ruby Way, why not

table.select { |row| row.foo =~ /hello/i }

:slight_smile:

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
~ > ruby -e ‘p % ^) .intern’
====================================