NODE tree introspection

I’ve been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully…)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?

···


“The best index to a person’s character is a) how he treats people who can’t
do him any good and b) how he treats people who can’t fight back.”
– Abigail Van Buren

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

I believe somebody attempted to … uhmmm… do something using rockit.

···

On Thu, Oct 24, 2002 at 12:59:54AM +0900, Simon Cozens wrote:

I’ve been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully…)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?


_ _

__ __ | | ___ _ __ ___ __ _ _ __
'_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |
__ \ | | | | | (| | | | |
.__/ _,
|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

We are using Linux daily to UP our productivity - so UP yours!
– Adapted from Pat Paulsen by Joe Sloan

In article 86fzuxuppd.fsf@squash.oucs.ox.ac.uk,

I’ve been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully…)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?

Hmmmm… interesting idea. So basically what you’re proposing would be
able to walk the AST of the currently running program (or any arbitrary
snippet that you send off to eval?)? I actually have some use for this
kind of thing so if you do it please keep us informed.

I don’t think anyone has approached the problem in the way you propose.
Can you give some more details of how you’re thinking of doing it? I know
that Bob Calco is working on a Ruby parser based on Matz’ Ruby yacc file
and he intends it to be a Ruby extension (on second thought, maybe what
he is doing is similar to what you’re thinking of doing).

Phil

···

Simon Cozens simon@ermine.ox.ac.uk wrote:

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Hmmmm… interesting idea. So basically what you’re proposing would be
able to walk the AST of the currently running program (or any arbitrary
snippet that you send off to eval?)?

Yes. That’s what Perl’s B.pm does.

% perl -MB -le ‘$x= B::main_start; print $x->name while $x=$x->next’
nextstate
pushmark
gv
entersub
gvsv
sassign

(New statement, add a mark to the stack, add a symbol table entry
[B::main_start] to the stack, call it, put a variable on the stack,
assign the result of the call to the variable…)

I don’t think anyone has approached the problem in the way you propose.
Can you give some more details of how you’re thinking of doing it?

I can even give you some sample code, because I’ve already got a fair bit
of it working. :slight_smile:

It says a lot for Ruby’s ease of use, clarity of code and
well-developed extensions system that I’ve only been using the
bloody language for three days and I’ve already got most of a bytecode
walker constructed.

Here’s the core of it:

static VALUE
bytecode_get_root(VALUE class)
{
return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);
}

void Init_Bytecode () {
cByteNode = rb_define_class(“ByteNode”, rb_cObject);
rb_define_singleton_method(cByteNode, “root”, bytecode_get_root, 0);
rb_define_method(cByteNode, “type”, bytecode_nd_type, 0);
rb_define_method(cByteNode, “line”, bytecode_nd_line, 0);
rb_define_method(cByteNode, “head”, bytecode_nd_head, 0);
rb_define_method(cByteNode, “next”, bytecode_nd_next, 0);

}

The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
nd_* macro, and wrap the result.

Now I can say:

% ruby -e ‘require “Bytecode”; p ByteNode.root; p ByteNode.root.next.type’
#ByteNode:0x401a6550
4

Is this making more sense now?

Of course, it’ll make even more sense when I write something which converts
type back into a string rather than an enum, etc. :slight_smile:

···


Darn it, who spiked my coffee with water?!
– Larry Wall in 200002141819.KAA18235@kiev.wall.org

In article 20021023172146.GB26640@student.ei.uni-stuttgart.de,

···

Mauricio Fernández batsman.geo@yahoo.com wrote:

On Thu, Oct 24, 2002 at 12:59:54AM +0900, Simon Cozens wrote:

I’ve been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully…)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?

http://www.rubygarden.org/ruby?RubyInRuby
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

I believe somebody attempted to … uhmmm… do something using rockit.

Yes there are other projects out there with the goal of doing what Simon
is trying to accomplish (see also Ruth). However, it seems that perhaps
the approach that he is taking might lead to something…

Phil

In article 86bs5luk1m.fsf@squash.oucs.ox.ac.uk,

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Hmmmm… interesting idea. So basically what you’re proposing would be
able to walk the AST of the currently running program (or any arbitrary
snippet that you send off to eval?)?

Yes. That’s what Perl’s B.pm does.

% perl -MB -le ‘$x= B::main_start; print $x->name while $x=$x->next’
nextstate
pushmark
gv
entersub
gvsv
sassign

(New statement, add a mark to the stack, add a symbol table entry
[B::main_start] to the stack, call it, put a variable on the stack,
assign the result of the call to the variable…)

I don’t think anyone has approached the problem in the way you propose.
Can you give some more details of how you’re thinking of doing it?

I can even give you some sample code, because I’ve already got a fair bit
of it working. :slight_smile:

It says a lot for Ruby’s ease of use, clarity of code and
well-developed extensions system that I’ve only been using the
bloody language for three days and I’ve already got most of a bytecode
walker constructed.

:wink:

Here’s the core of it:

static VALUE
bytecode_get_root(VALUE class)
{
return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);
}

void Init_Bytecode () {
cByteNode = rb_define_class(“ByteNode”, rb_cObject);
rb_define_singleton_method(cByteNode, “root”, bytecode_get_root, 0);
rb_define_method(cByteNode, “type”, bytecode_nd_type, 0);
rb_define_method(cByteNode, “line”, bytecode_nd_line, 0);
rb_define_method(cByteNode, “head”, bytecode_nd_head, 0);
rb_define_method(cByteNode, “next”, bytecode_nd_next, 0);

}

The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
nd_* macro, and wrap the result.

Now I can say:

% ruby -e ‘require “Bytecode”; p ByteNode.root; p ByteNode.root.next.type’
#ByteNode:0x401a6550
4

Is this making more sense now?

Yes and no. Since Ruby doesn’t use a bytecode interpretting virtual
machine (instead it walks an AST) what exactly do you mean by bytecode
here (or are you defining a bytecode for Ruby).

Of course, it’ll make even more sense when I write something which converts
type back into a string rather than an enum, etc. :slight_smile:

I think what you’re working on is very important and would be useful for a
variety of projects (Like Cardinal for example). Please if possible can
you start up a sourceforge or savannah project for this? After the Ruby
conference I think I might have some time to help out with this.

Phil

···

Simon Cozens simon@ermine.ox.ac.uk wrote:

Funny, I should have read more about previous attempts to do ‘Ruby in
Ruby’, for I really thought that Simon’s approach was THE approach :slight_smile:

···

On Thu, Oct 24, 2002 at 04:41:24AM +0900, Phil Tomson wrote:

In article 20021023172146.GB26640@student.ei.uni-stuttgart.de,
Mauricio Fernández batsman.geo@yahoo.com wrote:

On Thu, Oct 24, 2002 at 12:59:54AM +0900, Simon Cozens wrote:

I’ve been considering writing something for Ruby similar to the Perl
B.pm module; this is a module which provides Perl-level access to the
bytecode op tree, so that other modules can walk the op tree in
various interesting ways. This is the way the Perl to C compiler was
implemented. (however unsuccessfully…)

It seems reasonably easy to do in Ruby, just by wrapping the NODE*
structures as Ruby objects. Has anyone already done this or tried
working on it?

http://www.rubygarden.org/ruby?RubyInRuby
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/8693
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/6885

I believe somebody attempted to … uhmmm… do something using rockit.

Yes there are other projects out there with the goal of doing what Simon
is trying to accomplish (see also Ruth). However, it seems that perhaps
the approach that he is taking might lead to something…


_ _

__ __ | | ___ _ __ ___ __ _ _ __
'_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |
__ \ | | | | | (| | | | |
.__/ _,
|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Problem solving under Linux has never been the circus that it is under
AIX.
– Pete Ehlke in comp.unix.aix

In article 86bs5luk1m.fsf@squash.oucs.ox.ac.uk,

···

Simon Cozens simon@ermine.ox.ac.uk wrote:

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Hmmmm… interesting idea. So basically what you’re proposing would be
able to walk the AST of the currently running program (or any arbitrary
snippet that you send off to eval?)?

Yes. That’s what Perl’s B.pm does.

% perl -MB -le ‘$x= B::main_start; print $x->name while $x=$x->next’
nextstate
pushmark
gv
entersub
gvsv
sassign

(New statement, add a mark to the stack, add a symbol table entry
[B::main_start] to the stack, call it, put a variable on the stack,
assign the result of the call to the variable…)

I don’t think anyone has approached the problem in the way you propose.
Can you give some more details of how you’re thinking of doing it?

I can even give you some sample code, because I’ve already got a fair bit
of it working. :slight_smile:

It says a lot for Ruby’s ease of use, clarity of code and
well-developed extensions system that I’ve only been using the
bloody language for three days and I’ve already got most of a bytecode
walker constructed.

Here’s the core of it:

static VALUE
bytecode_get_root(VALUE class)
{
return Data_Wrap_Struct(class, 0, 0, ruby_eval_tree);
}

void Init_Bytecode () {
cByteNode = rb_define_class(“ByteNode”, rb_cObject);
rb_define_singleton_method(cByteNode, “root”, bytecode_get_root, 0);
rb_define_method(cByteNode, “type”, bytecode_nd_type, 0);
rb_define_method(cByteNode, “line”, bytecode_nd_line, 0);
rb_define_method(cByteNode, “head”, bytecode_nd_head, 0);
rb_define_method(cByteNode, “next”, bytecode_nd_next, 0);

}

The bytecode_nd_* functions just do a Data_Get_Struct, call the relevant
nd_* macro, and wrap the result.

Now I can say:

% ruby -e ‘require “Bytecode”; p ByteNode.root; p ByteNode.root.next.type’
#ByteNode:0x401a6550
4

I wonder if we could do this a lot faster by using swig to wrap the
Ruby C parser? Swig Ruby itself to make an extension for Ruby.

…just a thought.

Phil

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Yes and no. Since Ruby doesn’t use a bytecode interpretting virtual
machine (instead it walks an AST) what exactly do you mean by bytecode
here (or are you defining a bytecode for Ruby).

Sorry; “bytecode” is probably the wrong word because it implies some kind
of serialized assembler. I’m using it in a much looser sense. Perl walks
an AST, too. But being able to get at and fiddle with the AST leads to a
lot of very interesting things: serialization of the internal form to disk
so you do get true bytecode, translation to C, graphing the internal
structure of the program, and so on.

My ulterior motive, of course, is to enable a Ruby->Parrot backend.

I think what you’re working on is very important and would be useful for a
variety of projects (Like Cardinal for example). Please if possible can
you start up a sourceforge or savannah project for this?

I’ll need some help; as I’m very new to Ruby, I don’t know exactly what’s
required to set up and distribute a Ruby extension the politically correct
way. :slight_smile:

What I’ll do is dump the code into my CVS tree, and make it publically
available, and I can give you or whoever else commit access to
that. I’ll also send you the code off-list for now, because I have to head
away for a few days.

···


perl -le ‘print (@) - $; +$ (- % _); * + *’

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

I wonder if we could do this a lot faster by using swig to wrap the
Ruby C parser? Swig Ruby itself to make an extension for Ruby.

First, SWIG is there when your own language’s extension mechanism is a
pain in the ass; Ruby’s extension system is just fine, (much more
sensible than Perl’s or Python’s) and so I don’t think SWIG is the
answer. (to anything) And, to be blunt, I prefered Perl’s XS to SWIG.
Your milage, of course, is sure to vary.

Second, SWIG is particularly useful when you’re trying to do cross-language
library extensions. But we’re only trying to export Ruby’s AST to Ruby, so
cross-langage concerns are irrelevant.

We dealt with SWIG in Extending and Embedding Perl
(http://www.amazon.com/exec/obidos/ASIN/1930110820/) and, to be frank,
I really don’t want to deal with it again. Ruby’s extension API,
although I’ve beaten it up in the past
(IBM Developer)
kicks seven shades of shite out of Perl’s.

···


WILLIAM: Just incidentally – why are you wearing that?
SPIKE: Ahm – combination of factors really. No clean clothes…
W: There never will be, you know, unless you actually clean your clothes.
S: Right. Vicious circle.

In article 867kg8vp5k.fsf@squash.oucs.ox.ac.uk,

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Yes and no. Since Ruby doesn’t use a bytecode interpretting virtual
machine (instead it walks an AST) what exactly do you mean by bytecode
here (or are you defining a bytecode for Ruby).

Sorry; “bytecode” is probably the wrong word because it implies some kind
of serialized assembler. I’m using it in a much looser sense. Perl walks
an AST, too. But being able to get at and fiddle with the AST leads to a
lot of very interesting things: serialization of the internal form to disk
so you do get true bytecode, translation to C, graphing the internal
structure of the program, and so on.

My ulterior motive, of course, is to enable a Ruby->Parrot backend.

I think what you’re working on is very important and would be useful for a
variety of projects (Like Cardinal for example). Please if possible can
you start up a sourceforge or savannah project for this?

I’ll need some help; as I’m very new to Ruby, I don’t know exactly what’s
required to set up and distribute a Ruby extension the politically correct
way. :slight_smile:

    • Don’t worry, nobody else knows how to do it the “politically
      correct” way yet either. There is no standard way yet to
      set up a module. There is the ruby equivalent of Config.pm
      in rbconfig.rb and rpkg is an emerging standard, but there
      is still not consensus or anything remotely like h2xs,
      Makefile.PL and CPAN.

What I’ll do is dump the code into my CVS tree, and make it publically
available, and I can give you or whoever else commit access to
that. I’ll also send you the code off-list for now, because I have to head
away for a few days.

    • Did you look at Ruth? It’s not quite the same as what you’re
      trying to do, but it does walk the AST tree fairly well.
    • Booker C. Bense
···

Simon Cozens simon@ermine.ox.ac.uk wrote:

In article 868z0irxgw.fsf@squash.oucs.ox.ac.uk,

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

I wonder if we could do this a lot faster by using swig to wrap the
Ruby C parser? Swig Ruby itself to make an extension for Ruby.

First, SWIG is there when your own language’s extension mechanism is a
pain in the ass; Ruby’s extension system is just fine, (much more
sensible than Perl’s or Python’s) and so I don’t think SWIG is the
answer. (to anything) And, to be blunt, I prefered Perl’s XS to SWIG.
Your milage, of course, is sure to vary.

Where SWIG really shines is when you’ve got some substantial C or C++
library that you want to wrap pretty-much automatically (and I’m always
amazed how little I have to do to use SWIG).

Second, SWIG is particularly useful when you’re trying to do cross-language
library extensions. But we’re only trying to export Ruby’s AST to Ruby, so
cross-langage concerns are irrelevant.

Well, I guess I should put it differently: We could use SWIG to wrap
ruby.h and node.h and basically do what you’re trying to do, but more
automatically.

That brings up an interesting point: What if one were to wrap Python with SWIG so that the Python interpreter was available from within Ruby?

We dealt with SWIG in Extending and Embedding Perl
(http://www.amazon.com/exec/obidos/ASIN/1930110820/) and, to be frank,
I really don’t want to deal with it again.

What version of SWIG did you use? I guess I started out with SWIG 1.3.13
and now use 1.3.15 and have find it very easy to use. I’ve heard that
earlier versions were more of a pain. Also, I suspect you were using SWIG
to make Perl extensions - I’ve only used it to create Ruby extensions, so
perhaps there are some differences with Perl.

Ruby’s extension API,
although I’ve beaten it up in the past
(IBM Developer)
kicks seven shades of shite out of Perl’s.

Yes, Ruby’s extension API is great, but when you’ve got lots of existing
code to wrap SWIG works great in my experience.

Phil

···

Simon Cozens simon@ermine.ox.ac.uk wrote:

In article apl07u02pmj@enews1.newsguy.com,

In article 868z0irxgw.fsf@squash.oucs.ox.ac.uk,

ptkwt@shell1.aracnet.com (Phil Tomson) writes:

Second, SWIG is particularly useful when you’re trying to do cross-language
library extensions. But we’re only trying to export Ruby’s AST to Ruby, so
cross-langage concerns are irrelevant.

Well, I guess I should put it differently: We could use SWIG to wrap
ruby.h and node.h and basically do what you’re trying to do, but more
automatically.

OK, I see the error of my ways - I just tried to wrap node.h with SWIG
and yeah, that’s not gonna work… you end up wrapping stuff that
obviously doesn’t need to be wrapped (he says sheepishly:)

Have you added any more code to your bytecode.c?

That brings up an interesting point: What if one were to wrap Python with SWIG so that the Python interpreter was available from within Ruby?

But this might still be a viable idea for someone who might want to
evaluate Python code from within Ruby (or I suppose the same could be
done with Perl or Tcl).

Phil

···

Phil Tomson ptkwt@shell1.aracnet.com wrote:

Simon Cozens simon@ermine.ox.ac.uk wrote: