Revival of RubyInRuby?

ruby-talk
1

Dear Rubyists,

Lately (as I was dissatified with Ruby's current continuation implementation), I remembered that we had a Ruby-In-Ruby project some years ago. Python has PyPy, which got even fund by EU. Squeak in mostly written in itself, too.

So the question is, do we get enough interested people together that would work on such a project? I've noticed that PyPy and StacklessPython develop much code in so-called Sprints, where the developers meet and hack a weekend or a whole week on that project. Ruby developers are probably more distributed that Python's, but maybe it would work if we build small teams, which are then connected via internet to other teams, worldwide... just dreaming :wink: Or isn't that exactly the idea of the Ruby Codefest by Ruby Central, Inc? Maybe we could even get funded :wink:

So who is really interested in this project and would spend a little bit of his/her free time (of course not on christmas or silvester :-).

= First steps?

1. setup a project... discuss, discuss, discuss... agree on some goals,
    most important is that we stand united and bundle our forces :wink:
2. write a Ruby parser in Ruby (do we already have one, Robert Feldt?)
    we could also omit this step, and use the node-tree directly, and
    implement the parser later.
3. Ruby AST-to-bytecode generator (YARV bytecode?)
4. Implement the Ruby VM (in Ruby): object model, memory management (GC),
    interface to RubyC, port core-libraries.
5. Implement the RubyVM in C (either manually or by using a MicroRuby
    approach, by translating a subset of Ruby into C)

Steps 2. and 3. could be reused by future RubyC implementations. 4. and 5. not, this are the steps that would bring us some new features.

What I'm especially dreaming of is the ability to dump any object to disk... threads, continuations, and to have an accurate (non-conservative) garbage collector, so that Ruby will under no circumstances (except bugs) leak memory (that's especially important in the case of continuations). If we can dump everything, the GC-thing would of course be optional, as we could stop and restart at any point.

Well, we could also implement a Ruby-like language first... maybe that's simpler... start small. And parts of this project could flow into Rite/Ruby2 if matz agrees.

Enough for today...tell me what you think about this (old) idea...

Regards,

   Michael

2

We've got a fairly good start w/ our ruby2c translator. The long term goal is to reimplement ruby in ruby and translate it to C in much the same way that Squeak smalltalk does. More info:

IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

···

On Dec 25, 2004, at 4:25 PM, Michael Neumann wrote:

Lately (as I was dissatified with Ruby's current continuation implementation), I remembered that we had a Ruby-In-Ruby project some years ago. Python has PyPy, which got even fund by EU. Squeak in mostly written in itself, too.

So the question is, do we get enough interested people together that would work on such a project?

3

Michael Neumann wrote:

Lately (as I was dissatified with Ruby's current continuation implementation), I remembered that we had a Ruby-In-Ruby project some years ago. Python has PyPy, which got even fund by EU. Squeak in mostly written in itself, too.

So the question is, do we get enough interested people together that would work on such a project? I've noticed that PyPy and StacklessPython

[snip]

Btw, I'm not following PyPy or the Stackless project. Have they produced something "tangible" or "real"? It seems Stackless have existed for years (back in the day I was beginning to learn Python) but I don't recall the main Python implementation being stackless even today?

Regards,
dave

4

David Garamond ha scritto:

[snip]

Btw, I'm not following PyPy or the Stackless project. Have they produced something "tangible" or "real"? It seems Stackless have existed for years (back in the day I was beginning to learn Python) but I don't recall the main Python implementation being stackless even today?

the main implementation is not stackless, but stackless is working and used in some projects, IIUC. PyPy is not usable, but it seem to be shaping fine, they recently ained a found from the EU, too.

5

Ryan Davis wrote:

Lately (as I was dissatified with Ruby's current continuation implementation), I remembered that we had a Ruby-In-Ruby project some years ago. Python has PyPy, which got even fund by EU. Squeak in mostly written in itself, too.

So the question is, do we get enough interested people together that would work on such a project?

We've got a fairly good start w/ our ruby2c translator. The long term goal is to reimplement ruby in ruby and translate it to C in much the same way that Squeak smalltalk does. More info:

That's great news. Very similar to what I thought. But I don't see a reason for implementing the parser in a subset of Ruby (except for performance). Once you've bootstrapped the parser to bytecode with real-Ruby, you can load it into the VM.

How active is the project? Who else is working on it?

Parser: I'm currently hacking a bit on my packrat project and try to get a TPDL (Top Down Parsing Language) generator working. It's pretty slow (~1000 lines per second), maybe if I implement memoizing it will become faster. Nevertheless, it would be fast enough if you precompile the whole libraries to .rbc, and use the parser only for "eval" at runtime. I don't think that this would be a big performance penalty.

Garbage Collector?
Any concrete plans, which strategy to use? Using a non conservating GC would break with existing C extensions. But if they'd be rewritten in Ruby2C language, the generator could automatically generate code for a conservating GC or a non conservating... I'm for a non-conservative due to the reasons I said in the last email.

My idea for the Ruby2C translator is very similar, but more C oriented than yours (it's not only for extensions). Example:

   class MyStruct < CStruct

     def_type :initialize, {:a => 'int', :b => 'int', :return => 'void'}
     def initialize(a, b)
       @a = a
       @b = b
     end

     def_type :inspect, {:return => 'void'}
     def inspect
       printf("%d\n%d\n", @a, @b)
     end

     ...
   end

which would then be translated into C:

   typedef struct MyStruct {
     int a;
     int b;
   };

   void MyStruct_initialize(MyStruct* self, int a, int b) {
     self->a = a;
     self->b = b;
   }

   void MyStruct_inspect(MyStruct *self) {
     printf("%d\n%d\n", self->a, self->b);
   }

The types of the instance variables are automatically infered. Types or local, argument and return values have to be specified via def_type. Special methods like deref(value) would map to *(value) in the C output. Constants map to MACROS, so they are simply replaced with their name in the C output:

   def a
     printf("%d\n", SQL_BIT)
   end

   becomes:

     printf("%d\n", SQL_BIT)

And of course method calls map to function calls in C, with the difference that, if the method is defined in the class, then the self argument is passed, otherwise a real function call (as for printf) is performed.

I think with this kind of translator, it's pretty simple to write C programs cleanly in Ruby. And it's still valid Ruby, unlike Pyrex for example (Pyrex is a Python-dialect for implementing extensions, but it's not valid Python, AFAIK).

One could extend that a bit, and e.g. if your class inherits from T_DATA, then you are specifying a Ruby-extensions, which gets automatically wrapped and unwrapped, and provides the attached C struct to you without hassle.

And of course class methods map to "self"-less functions:

   type_def :"MyStruct.new", {:i => int, :j => int, a => 'MyStruct*',
                              :return => 'MyStruct*'}
   def MyStruct.new(i, j)
     obj = cast(malloc(sizeof(MyStruct)), 'MyStruct*')
     if obj == NULL
       # error
     else
       MyStruct_initialize(obj, i, j) # or obj.initialize(i,j)
       return obj
     end
   end

Indeed, what I propose is some kind of C with classes :wink:
Maybe I'm missing something important.

···

On Dec 25, 2004, at 4:25 PM, Michael Neumann wrote:

IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

6

Ryan Davis wrote:

Lately (as I was dissatified with Ruby's current continuation implementation), I remembered that we had a Ruby-In-Ruby project some years ago. Python has PyPy, which got even fund by EU. Squeak in mostly written in itself, too.

So the question is, do we get enough interested people together that would work on such a project?

We've got a fairly good start w/ our ruby2c translator. The long term goal is to reimplement ruby in ruby and translate it to C in much the same way that Squeak smalltalk does. More info:

IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

Is the source somewhere available? I can't find it on the rubyforge project page.

Regards,

   Michael

···

On Dec 25, 2004, at 4:25 PM, Michael Neumann wrote:

7

That's great news. Very similar to what I thought. But I don't see a reason for implementing the parser in a subset of Ruby (except for performance). Once you've bootstrapped the parser to bytecode with real-Ruby, you can load it into the VM.

I wasn't planning on making a bytecode step. The system is architected in such a way that it is possible (and easy--eric has done a prototype), but it wasn't in the plans.

How active is the project? Who else is working on it?

Taking a minor vacation this month, but it is Eric and me w/ some close-by projects that may merge in the future.

Parser: I'm currently hacking a bit on my packrat project and try to get a TPDL (Top Down Parsing Language) generator working. It's pretty slow (~1000 lines per second), maybe if I implement memoizing it will become faster. Nevertheless, it would be fast enough if you precompile the whole libraries to .rbc, and use the parser only for "eval" at runtime. I don't think that this would be a big performance penalty.

You might want to check out coco/r(uby) (on rubyforge as cocor). It is the coco/r RDP framework ported to ruby (and generating pure ruby). I haven't benchmarked it at all, but I suspect it does pretty well.

Garbage Collector?
Any concrete plans, which strategy to use? Using a non conservating GC would break with existing C extensions. But if they'd be rewritten in Ruby2C language, the generator could automatically generate code for a conservating GC or a non conservating... I'm for a non-conservative due to the reasons I said in the last email.

non-conservative won't necessarily break C extensions as far as I understand. It'll just have to jump through some extra hoops.

My idea for the Ruby2C translator is very similar, but more C oriented than yours (it's not only for extensions). Example:

ruby2c isn't for extensions at all. If you could tell me what gave you that impression (like a slide or something, I'll fix it).

The types of the instance variables are automatically infered. Types or local, argument and return values have to be specified via def_type. Special methods like deref(value) would map to *(value) in the C output. Constants map to MACROS, so they are simply replaced with their name in the C output:

We do type inference already, without a map.

And of course method calls map to function calls in C, with the difference that, if the method is defined in the class, then the self argument is passed, otherwise a real function call (as for printf) is performed.

Here is where we have maps, but only for mapping into the standard C library.

···

On Dec 26, 2004, at 4:30 AM, Michael Neumann wrote:

8

Michael Neumann ha scritto:

My idea for the Ruby2C translator is very similar, but more C oriented than yours (it's not only for extensions). Example:

<snip>
note that also koichi sasada (the YARV guy) is planning to provide a c translation engine. There was some stuff about it in the rubyconf slides, but I can't remember anything :confused:

9

Not yet no... but SOON! We have one more round of private review that'll happen in a day or two and then it'll go public after we incorporate the feedback. I'm also working on expanding its capabilities as much as possible before it goes to the next review, but that isn't critical.

···

On Dec 26, 2004, at 2:29 PM, Michael Neumann wrote:

Ryan Davis wrote:

We've got a fairly good start w/ our ruby2c translator. The long term goal is to reimplement ruby in ruby and translate it to C in much the same way that Squeak smalltalk does. More info:
IRC: #ruby2c
rubyforge: http://rubyforge.org/projects/ruby2c/
Propaganda: http://www.zenspider.com/~ryand/Ruby2C.pdf

Is the source somewhere available? I can't find it on the rubyforge project page.

10

gabriele renzi wrote:

Michael Neumann ha scritto:

My idea for the Ruby2C translator is very similar, but more C oriented than yours (it's not only for extensions). Example:

<snip>
note that also koichi sasada (the YARV guy) is planning to provide a c translation engine. There was some stuff about it in the rubyconf slides, but I can't remember anything :confused:

But remember that koichi's ahead-of-time compiler is quite different. This code still depends on the runtime+libraries, whereas we aim at writing the runtime and core libraries (Array, Hash, String...) in a Ruby-subset. Both makes sense!

Regards,

   Michael