Ruby Data Structure Query Abstractions/Patterns

Hi all,

I've written a small ruby application that uses YAML and the
associated hash/array Ruby structure.

I am now looking to do some query-like actions on the basic ruby data
structure. I'm wondering if there are any packages of
query-abstractions that make sense for Ruby in a generalized sort-of
way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.

Currently I've written my structure-parsing code with a few of my
cases, and I'm starting to refactor with blocks and lambdas in order
to keep duplication low and offer an easily configurable higher-level
of abstraction. Now the code is starting to pressure me to create a
simple query language. Maybe I should go to a ruby db or sqlite, but I
expect I ultimately only need the most basic functionality.

Any suggestions or recommendations?

Thanks,
Nick

···

--
Nicholas Van Weerdenburg

Not quite sure how well this would fit with Ruby, but you may want to take a look at SODA [1] which could be used to query generic object graphs.

In addition, you may want to consider EOQualifier for, er, "inspiration":

http://developer.apple.com/documentation/WebObjects/Reference/API/com/webobjects/eocontrol/EOQualifier.html

Cheers,

PA.

[1] http://sodaquery.sourceforge.net/

···

On Jan 05, 2005, at 19:26, Nicholas Van Weerdenburg wrote:

Any suggestions or recommendations?

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag news:632154f705010510264aea7a10@mail.gmail.com...

Hi all,

I've written a small ruby application that uses YAML and the
associated hash/array Ruby structure.

I am now looking to do some query-like actions on the basic ruby data
structure. I'm wondering if there are any packages of
query-abstractions that make sense for Ruby in a generalized sort-of
way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.

Currently I've written my structure-parsing code with a few of my
cases, and I'm starting to refactor with blocks and lambdas in order
to keep duplication low and offer an easily configurable higher-level
of abstraction. Now the code is starting to pressure me to create a
simple query language. Maybe I should go to a ruby db or sqlite, but I
expect I ultimately only need the most basic functionality.

Any suggestions or recommendations?

Another thought: maybe you can even use Ruby itself. I mean, one can produce quite readable select expressions in Ruby and we have Enumerable#all?, #any? etc. You might want to add some functionality for recursive traversal though.

Kind regards

    robert

Quoteing vanweerd@gmail.com, on Thu, Jan 06, 2005 at 03:26:22AM +0900:

Hi all,

I've written a small ruby application that uses YAML and the
associated hash/array Ruby structure.

I am now looking to do some query-like actions on the basic ruby data
structure. I'm wondering if there are any packages of
query-abstractions that make sense for Ruby in a generalized sort-of
way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.

This is why you might want to consider XML... then you can just USE
XPath.

Sam

···

Currently I've written my structure-parsing code with a few of my
cases, and I'm starting to refactor with blocks and lambdas in order
to keep duplication low and offer an easily configurable higher-level
of abstraction. Now the code is starting to pressure me to create a
simple query language. Maybe I should go to a ruby db or sqlite, but I
expect I ultimately only need the most basic functionality.

Any suggestions or recommendations?

Thanks,
Nick
--
Nicholas Van Weerdenburg

Thanks. Both links are very interesting.

Regards,
Nick

···

On Thu, 6 Jan 2005 03:41:23 +0900, PA <petite.abeille@gmail.com> wrote:

On Jan 05, 2005, at 19:26, Nicholas Van Weerdenburg wrote:

> Any suggestions or recommendations?

Not quite sure how well this would fit with Ruby, but you may want to
take a look at SODA [1] which could be used to query generic object
graphs.

In addition, you may want to consider EOQualifier for, er,
"inspiration":

http://developer.apple.com/documentation/WebObjects/Reference/API/com/
webobjects/eocontrol/EOQualifier.html

Cheers,

PA.

[1] http://sodaquery.sourceforge.net/

--
Nicholas Van Weerdenburg

I like the idea of using Ruby, and have been extending my code in that
manner. However, XML becomes interesting soon because of it's DOM
traversal capabilities.

I'm also wondering if I'm missing some common Ruby/Perl/Python idiom
for handling in-memory data-structure traversal in a more abstract
way, but so far it doesn't look like it.

Does it make sense to have a general data-structure traversal/query
api in a language like Ruby?

Thanks,
Nick

···

On Thu, 6 Jan 2005 05:56:31 +0900, Robert Klemme <bob.news@gmx.net> wrote:

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag
news:632154f705010510264aea7a10@mail.gmail.com...
> Hi all,
>
> I've written a small ruby application that uses YAML and the
> associated hash/array Ruby structure.
>
> I am now looking to do some query-like actions on the basic ruby data
> structure. I'm wondering if there are any packages of
> query-abstractions that make sense for Ruby in a generalized sort-of
> way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.
>
> Currently I've written my structure-parsing code with a few of my
> cases, and I'm starting to refactor with blocks and lambdas in order
> to keep duplication low and offer an easily configurable higher-level
> of abstraction. Now the code is starting to pressure me to create a
> simple query language. Maybe I should go to a ruby db or sqlite, but I
> expect I ultimately only need the most basic functionality.
>
> Any suggestions or recommendations?

Another thought: maybe you can even use Ruby itself. I mean, one can
produce quite readable select expressions in Ruby and we have
Enumerable#all?, #any? etc. You might want to add some functionality for
recursive traversal though.

Kind regards

    robert

--
Nicholas Van Weerdenburg

Quoteing vanweerd@gmail.com, on Thu, Jan 06, 2005 at 03:26:22AM +0900:
> Hi all,
>
> I've written a small ruby application that uses YAML and the
> associated hash/array Ruby structure.

> I am now looking to do some query-like actions on the basic ruby data
> structure. I'm wondering if there are any packages of
> query-abstractions that make sense for Ruby in a generalized sort-of
> way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.

This is why you might want to consider XML... then you can just USE
XPath.

Sam

This is looking like the most reasonable choice at the moment- though
not the most fun, so I may continue with a Ruby solution for now.

Thanks,
Nick

···

On Thu, 6 Jan 2005 10:10:26 +0900, Sam Roberts <sroberts@uniserve.com> wrote:

>
> Currently I've written my structure-parsing code with a few of my
> cases, and I'm starting to refactor with blocks and lambdas in order
> to keep duplication low and offer an easily configurable higher-level
> of abstraction. Now the code is starting to pressure me to create a
> simple query language. Maybe I should go to a ruby db or sqlite, but I
> expect I ultimately only need the most basic functionality.
>
> Any suggestions or recommendations?
>
> Thanks,
> Nick
> --
> Nicholas Van Weerdenburg
>

--
Nicholas Van Weerdenburg

PA ha scritto:

Any suggestions or recommendations?

Not quite sure how well this would fit with Ruby, but you may want to take a look at SODA [1] which could be used to query generic object graphs.

In addition, you may want to consider EOQualifier for, er, "inspiration":

http://developer.apple.com/documentation/WebObjects/Reference/API/com/ webobjects/eocontrol/EOQualifier.html

Cheers,

PA.

[1] http://sodaquery.sourceforge.net/

also, JXPath allows XPAth to be applied to generic graphs, maybe of interest

···

On Jan 05, 2005, at 19:26, Nicholas Van Weerdenburg wrote:

Quoteing vanweerd@gmail.com, on Thu, Jan 06, 2005 at 01:20:40PM +0900:

> Quoteing vanweerd@gmail.com, on Thu, Jan 06, 2005 at 03:26:22AM +0900:
> > Hi all,
> >
> > I've written a small ruby application that uses YAML and the
> > associated hash/array Ruby structure.
>
> > I am now looking to do some query-like actions on the basic ruby data
> > structure. I'm wondering if there are any packages of
> > query-abstractions that make sense for Ruby in a generalized sort-of
> > way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.
>
> This is why you might want to consider XML... then you can just USE
> XPath.
>
> Sam
>

This is looking like the most reasonable choice at the moment- though
not the most fun, so I may continue with a Ruby solution for now.

I don't know, what's not fun about XML?

Rant: for years the world fills up with adhoc 1/2 solutions to data
structuring, then people decide that reinventing the wheel is dumb, and
make XML, and start adding tons of useful tools to it, like query
languages. The folks come along and say 'its too big and complicated"
and start again... There will soon be a YPath (YAML query language),
mark my words!

Ok, I feel better.

That aside, it sounds kind of cool making a rubyish query language for
ruby data structures... maybe XPath from Rexml could actually be abused
for your purpose, or used to implement a similar language?

Have fun!
Sam

···

On Thu, 6 Jan 2005 10:10:26 +0900, Sam Roberts <sroberts@uniserve.com> wrote:

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag news:632154f705010520196f99e845@mail.gmail.com...

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag
news:632154f705010510264aea7a10@mail.gmail.com...
> Hi all,
>
> I've written a small ruby application that uses YAML and the
> associated hash/array Ruby structure.
>
> I am now looking to do some query-like actions on the basic ruby data
> structure. I'm wondering if there are any packages of
> query-abstractions that make sense for Ruby in a generalized sort-of
> way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.
>
> Currently I've written my structure-parsing code with a few of my
> cases, and I'm starting to refactor with blocks and lambdas in order
> to keep duplication low and offer an easily configurable higher-level
> of abstraction. Now the code is starting to pressure me to create a
> simple query language. Maybe I should go to a ruby db or sqlite, but I
> expect I ultimately only need the most basic functionality.
>
> Any suggestions or recommendations?

Another thought: maybe you can even use Ruby itself. I mean, one can
produce quite readable select expressions in Ruby and we have
Enumerable#all?, #any? etc. You might want to add some functionality for
recursive traversal though.

Kind regards

    robert

I like the idea of using Ruby, and have been extending my code in that
manner. However, XML becomes interesting soon because of it's DOM
traversal capabilities.

I'm also wondering if I'm missing some common Ruby/Perl/Python idiom
for handling in-memory data-structure traversal in a more abstract
way, but so far it doesn't look like it.

Does it make sense to have a general data-structure traversal/query
api in a language like Ruby?

I'd say yes, but maybe only for certain types (DOM tree nodes) - not necessarily for general Hash, Array and the like. OTOH you can cook up something quite easily though:

class Object
  def traverse() yield self end
end

module Enumerable
  def traverse(&b)
    each {|o| o.traverse(&b)}
  end
end

class Hash
  def traverse(&b)
    each {|k,v| v.traverse(&b)}
  end
end

Note that this does not take care of looks in the object graph. To do that, you need to do a bit more:

class Object
  def traverse(context = {}) context[self.id] ||= (yield self; true) end
end

module Enumerable
  def traverse(context = {}, &b)
    unless context[self.id]
      context[self.id] = true
      each {|o| o.traverse(context, &b)}
    end
  end
end

class Hash
  def traverse(context = {}, &b)
    unless context[self.id]
      context[self.id] = true
      each {|k,v| v.traverse(context, &b)}
    end
  end
end

Kind regards

    robert

···

On Thu, 6 Jan 2005 05:56:31 +0900, Robert Klemme <bob.news@gmx.net> wrote:

Since you did not include any smileys ...

(1) Having a uniform and extensible concrete syntax for tree structures is
great
(2) Having a whole set of uniform manipulations on (1) is great
(3) Insisting on applying (1) to the concrete syntax of (2) is absurd and
the result, ugly

Just about every (computer) language has a natural underlying tree
structure, with lots of cross-tree links. So, should we now start to write
our Ruby using XML ?

Ok, I feel better now :slight_smile:

···

"Sam Roberts" <sroberts@uniserve.com> wrote

I don't know, what's not fun about XML?

That looks pretty good.

Thanks,
Nick

···

`On Thu, 6 Jan 2005 21:36:30 +0900, Robert Klemme <bob.news@gmx.net> wrote:

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag
news:632154f705010520196f99e845@mail.gmail.com...
> On Thu, 6 Jan 2005 05:56:31 +0900, Robert Klemme <bob.news@gmx.net> wrote:
>>
>> "Nicholas Van Weerdenburg" <vanweerd@gmail.com> schrieb im Newsbeitrag
>> news:632154f705010510264aea7a10@mail.gmail.com...
>> > Hi all,
>> >
>> > I've written a small ruby application that uses YAML and the
>> > associated hash/array Ruby structure.
>> >
>> > I am now looking to do some query-like actions on the basic ruby data
>> > structure. I'm wondering if there are any packages of
>> > query-abstractions that make sense for Ruby in a generalized sort-of
>> > way. E.g. XPath and XQuery for XML, SQL for RDMS, etc.
>> >
>> > Currently I've written my structure-parsing code with a few of my
>> > cases, and I'm starting to refactor with blocks and lambdas in order
>> > to keep duplication low and offer an easily configurable higher-level
>> > of abstraction. Now the code is starting to pressure me to create a
>> > simple query language. Maybe I should go to a ruby db or sqlite, but I
>> > expect I ultimately only need the most basic functionality.
>> >
>> > Any suggestions or recommendations?
>>
>> Another thought: maybe you can even use Ruby itself. I mean, one can
>> produce quite readable select expressions in Ruby and we have
>> Enumerable#all?, #any? etc. You might want to add some functionality for
>> recursive traversal though.
>>
>> Kind regards
>>
>> robert
>>
>>
>
> I like the idea of using Ruby, and have been extending my code in that
> manner. However, XML becomes interesting soon because of it's DOM
> traversal capabilities.
>
> I'm also wondering if I'm missing some common Ruby/Perl/Python idiom
> for handling in-memory data-structure traversal in a more abstract
> way, but so far it doesn't look like it.
>
> Does it make sense to have a general data-structure traversal/query
> api in a language like Ruby?

I'd say yes, but maybe only for certain types (DOM tree nodes) - not
necessarily for general Hash, Array and the like. OTOH you can cook up
something quite easily though:

class Object
  def traverse() yield self end
end

module Enumerable
  def traverse(&b)
    each {|o| o.traverse(&b)}
  end
end

class Hash
  def traverse(&b)
    each {|k,v| v.traverse(&b)}
  end
end

Note that this does not take care of looks in the object graph. To do that,
you need to do a bit more:

class Object
  def traverse(context = {}) context[self.id] ||= (yield self; true) end
end

module Enumerable
  def traverse(context = {}, &b)
    unless context[self.id]
      context[self.id] = true
      each {|o| o.traverse(context, &b)}
    end
  end
end

class Hash
  def traverse(context = {}, &b)
    unless context[self.id]
      context[self.id] = true
      each {|k,v| v.traverse(context, &b)}
    end
  end
end

Kind regards

    robert

--
Nicholas Van Weerdenburg

Quoteing itsme213@hotmail.com, on Thu, Jan 06, 2005 at 02:46:30PM +0900:

> I don't know, what's not fun about XML?

Since you did not include any smileys ...

(1) Having a uniform and extensible concrete syntax for tree structures is
great
(2) Having a whole set of uniform manipulations on (1) is great
(3) Insisting on applying (1) to the concrete syntax of (2) is absurd and
the result, ugly

I don't know what (3) is talking about. The syntax of XPath isn't XML.
It's a domain-specific language inside XML, much as regexs are a
domain-specific language inside ruby.

Just about every (computer) language has a natural underlying tree
structure, with lots of cross-tree links. So, should we now start to write
our Ruby using XML ?

No, nor YAML, has anybody suggested doing so?

Sam

···

"Sam Roberts" <sroberts@uniserve.com> wrote

"Sam Roberts" <sroberts@uniserve.com> wrote in message
news:20050106145338.GA2998@ensemble.local...

> (1) Having a uniform and extensible concrete syntax for tree structures

is

> great
> (2) Having a whole set of uniform manipulations on (1) is great
> (3) Insisting on applying (1) to the concrete syntax of (2) is absurd

and

> the result, ugly

I don't know what (3) is talking about. The syntax of XPath isn't XML.
It's a domain-specific language inside XML, much as regexs are a
domain-specific language inside ruby.

Well, my analogy to writing Ruby was off the mark, but I suppose I was doing
my rant to your "what's not to like about XML" rant. Because (3) is
definitely something I dislike about the XML (and it sounds like you would
agree too).

I had trouble understanding point 3, but I'm guessing your point was
that XSLT shouldn't be an XML dialect, but rather a proper language?

Similar to:

Nick

···

On Fri, 7 Jan 2005 00:46:35 +0900, itsme213 <itsme213@hotmail.com> wrote:

"Sam Roberts" <sroberts@uniserve.com> wrote in message
news:20050106145338.GA2998@ensemble.local...
> > (1) Having a uniform and extensible concrete syntax for tree structures
is
> > great
> > (2) Having a whole set of uniform manipulations on (1) is great
> > (3) Insisting on applying (1) to the concrete syntax of (2) is absurd
and
> > the result, ugly
>
> I don't know what (3) is talking about. The syntax of XPath isn't XML.
> It's a domain-specific language inside XML, much as regexs are a
> domain-specific language inside ruby.

Well, my analogy to writing Ruby was off the mark, but I suppose I was doing
my rant to your "what's not to like about XML" rant. Because (3) is
definitely something I dislike about the XML (and it sounds like you would
agree too).

--
Nicholas Van Weerdenburg

> > > (1) Having a uniform and extensible concrete syntax for tree

structures

> is
> > > great
> > > (2) Having a whole set of uniform manipulations on (1) is great
> > > (3) Insisting on applying (1) to the concrete syntax of (2) is

absurd

> and
> > > the result, ugly
> >
> > I don't know what (3) is talking about. The syntax of XPath isn't XML.
> > It's a domain-specific language inside XML, much as regexs are a
> > domain-specific language inside ruby.
>
> Well, my analogy to writing Ruby was off the mark, but I suppose I was

doing

> my rant to your "what's not to like about XML" rant. Because (3) is
> definitely something I dislike about the XML (and it sounds like you

would

> agree too).
>
>

I had trouble understanding point 3, but I'm guessing your point was
that XSLT shouldn't be an XML dialect, but rather a proper language?

Roughly yes. Specifically I thought the designers really failed to separate:

- the conceptual model of each language (the abstract syntax graph, or
roughly the AST with all cross-tree references resolved into edges and with
its static well-formedness rules plus semantics of what a given abstract
graph actually means e.g. when executed); XML is quite inadequate for this.

- the concrete interchange syntax of the language (for which an XML dialect
typically works fine); and

- the concrete human syntax(es) of the language, whether text or graphical;
XML is typically awful for this;

- the mappings between these (e.g. more than one possible XML encoding of an
abstract syntax)

In the case of XSLT they went further and designed a very low S/N-ratio
pointy-bracket concrete syntax for an untyped language that used some
unconventional combination of functional and rule-based features to
transform statically typed data. It's a good thing XQuery backed away from
some of that earlier unbridled XML enthusiasm. And I'm sure there is a pent
up rant in there somewhere...:slight_smile:

···

"Nicholas Van Weerdenburg" <vanweerd@gmail.com> wrote