Seven new VMs, all in a row

[Peter_Suk]

Hello everyone,

I thought I'd talk about my new project here, since there is a good chance that someone might be interested in it. I'm planning to put Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar under the covers, so Smalltalk VMs are a very good match for the language. This will give Ruby a much faster execution environment (perhaps 30X), VMs which are capable of incremental garbage collection, generational garbage collection that is so fast your progra still works even with an infinite loop allocating new objects (I do this as a lark sometimes), a wonderful debugger which will let programmers modify methods on the fly & continue execution, a "workspace" window where you can execute arbitrary code, a visual "inspect", a powerful "Refactoring Browser," an industrial strength OODB (Gemstone) with objects and methods you can define in Ruby, and a readily accessible meta-level which will allow Rubyists to readily modify their own language. (For example, you could then use Method wrappers to very quickly implement an Aspect-Oriented Ruby.)

My strategy for doing this involves writing a Ruby parser (or, rather, translating the existing one in JRuby to Ruby) then writing a Smalltalk Parser object to request parsing of Ruby code into an AST from a Ruby program outside of Smalltalk. We then reify the AST inside the image and use it to compile Ruby methods into bytecodes. (Multiple syntaxes can coexist in one Smalltalk image.) Once this is done, we can then compile the external Ruby parser and bring it into Smalltalk. Afterwards, we can use the Refactoring Browser Smalltalk parser plus a little runtime type inferencing to incrementally transform the image into pure Ruby.

We can do all of this without changing the structure of Ruby files & Modules or requiring Rubyists to do Smalltalk style image oriented development. And for those of you sharp enough to wonder: yes, we can handle Modules, Mixins, and fully qualified Method names without changing the Smalltalk VMs. (At least those that have Namespaces.)

If anyone is interested, please drop me a line.

--Peter

···

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Mark_Volkmann]

I think it would be really cool if you could do this on top of the free Squeak
VM.

Quoting Peter Suk <peter.kwangjun.suk@mac.com>:

···

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar
under the covers, so Smalltalk VMs are a very good match for the
language. This will give Ruby a much faster execution environment
(perhaps 30X), VMs which are capable of incremental garbage collection,
generational garbage collection that is so fast your progra still works
even with an infinite loop allocating new objects (I do this as a lark
sometimes), a wonderful debugger which will let programmers modify
methods on the fly & continue execution, a "workspace" window where you
can execute arbitrary code, a visual "inspect", a powerful "Refactoring
Browser," an industrial strength OODB (Gemstone) with objects and
methods you can define in Ruby, and a readily accessible meta-level
which will allow Rubyists to readily modify their own language. (For
example, you could then use Method wrappers to very quickly implement
an Aspect-Oriented Ruby.)

My strategy for doing this involves writing a Ruby parser (or, rather,
translating the existing one in JRuby to Ruby) then writing a Smalltalk
Parser object to request parsing of Ruby code into an AST from a Ruby
program outside of Smalltalk. We then reify the AST inside the image
and use it to compile Ruby methods into bytecodes. (Multiple syntaxes
can coexist in one Smalltalk image.) Once this is done, we can then
compile the external Ruby parser and bring it into Smalltalk.
Afterwards, we can use the Refactoring Browser Smalltalk parser plus a
little runtime type inferencing to incrementally transform the image
into pure Ruby.

We can do all of this without changing the structure of Ruby files &
Modules or requiring Rubyists to do Smalltalk style image oriented
development. And for those of you sharp enough to wonder: yes, we can
handle Modules, Mixins, and fully qualified Method names without
changing the Smalltalk VMs. (At least those that have Namespaces.)

If anyone is interested, please drop me a line.

--Peter

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

--
R. Mark Volkmann
Partner, Object Computing, Inc.

[Lothar_Scholz2 (Lothar Scholz)]

Hello Peter,

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put

Is this just brainstorming or have you already done something ?

···

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

[jc11 (jc)]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

-jc

[Florian_Gross2 (Florian Gross)]

Peter Suk wrote:

I thought I'd talk about my new project here, since there is a good chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

This does indeed sound promising. Having a self-hosted Ruby would have
certain benefits as well.

a wonderful debugger which will let programmers modify methods on the
fly & continue execution

That is already possible with ruby-breakpoint which spawns an IRB
session at a specific place of your code. See
http://ruby-breakpoint.rubyforge.org/ if you are interested in this sort
of things.

It sure can't hurt to have more options, though.

[Avi_Bryant]

Peter Suk wrote:

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

Hi Peter,

I'm definitely interested in this; please keep me posted. Actually, a
couple of people are aware that I was looking into the possibility of
doing this as a commercial product - since, from my benchmarking, we
can get about a 20x to 30x speed increase by hosting Ruby on a
commercial Smalltalk VM like VisualWorks, it seemed like it might be
worthwhile. However, the interest wasn't great enough for me to pursue
it, so I'm very glad that someone is.

A small note: you say "at least those that have namespaces", but
there's no need to be concerned about that. Namespaces are an
image-level thing, not a VM-level thing; any Smalltalk VM worth its
salt can support namespaces just fine (and in fact constructs like pool
variables are effectively namespaces already, they just have somewhat
different conventions than Ruby's).

Avi

[Robert_Feldt]

I'm interested; sounds like a great idea. However, I haven't been very
lucky turning up (free) info about the ("commercial") Smalltalk VM's
and their instruction sets etc (apart from the classic smalltalk
papers and I guess squeak where everything is open). Do you have good
pointers to such info?

Best,

Robert Feldt
robert.feldt@gmail.com

···

On Apr 7, 2005 5:44 PM, Peter Suk <peter.kwangjun.suk@mac.com> wrote:

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs. Ruby and Smalltalk are very similar
under the covers, so Smalltalk VMs are a very good match for the
language. This will give Ruby a much faster execution environment
(perhaps 30X), VMs which are capable of incremental garbage collection,
generational garbage collection that is so fast your progra still works
even with an infinite loop allocating new objects (I do this as a lark
sometimes), a wonderful debugger which will let programmers modify
methods on the fly & continue execution, a "workspace" window where you
can execute arbitrary code, a visual "inspect", a powerful "Refactoring
Browser," an industrial strength OODB (Gemstone) with objects and
methods you can define in Ruby, and a readily accessible meta-level
which will allow Rubyists to readily modify their own language. (For
example, you could then use Method wrappers to very quickly implement
an Aspect-Oriented Ruby.)

My strategy for doing this involves writing a Ruby parser (or, rather,
translating the existing one in JRuby to Ruby) then writing a Smalltalk
Parser object to request parsing of Ruby code into an AST from a Ruby
program outside of Smalltalk. We then reify the AST inside the image
and use it to compile Ruby methods into bytecodes. (Multiple syntaxes
can coexist in one Smalltalk image.) Once this is done, we can then
compile the external Ruby parser and bring it into Smalltalk.
Afterwards, we can use the Refactoring Browser Smalltalk parser plus a
little runtime type inferencing to incrementally transform the image
into pure Ruby.

We can do all of this without changing the structure of Ruby files &
Modules or requiring Rubyists to do Smalltalk style image oriented
development. And for those of you sharp enough to wonder: yes, we can
handle Modules, Mixins, and fully qualified Method names without
changing the Smalltalk VMs. (At least those that have Namespaces.)

If anyone is interested, please drop me a line.

[Its_Me]

"Peter Suk" <peter.kwangjun.suk@mac.com> wrote in message

I thought I'd talk about my new project here

Sounds fantastic, I can't wait!

And for those of you sharp enough to wonder: yes, we can
handle Modules, Mixins, and fully qualified Method names without
changing the Smalltalk VMs. (At least those that have Namespaces.)

Which ones do?

And what of Ruby 2.0 selector namesapces? Will they happen? And will that
map readily to a Smalltalk VM?

[Peter_Suk]

Yes, I was thinking along these lines. However, no real Namespaces (yet) in Squeak. We could finagle this by referencing classes with Module-name prefixes. (This has already been proposed.) When real Namespaces appear in Squeak, we could then properly place the classes in modules/namespaces and rename them programatically.

The VMs I am targeting first are commercial ones, but they have free versions/licenses. (VisualWorks and Gemstone) In addition, they are very attractive for implementing servers. (VisualWorks is damn fast, and Gemstone is a mature, rock-solid OODB.)

···

On Apr 7, 2005, at 10:47 AM, R. Mark Volkmann wrote:

I think it would be really cool if you could do this on top of the free Squeak
VM.

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Peter_Suk]

Lothar,

I have my hands on a working copy of Racc, and the JRuby source. I've begun the port of the RubyLexer into Smalltalk, but I've decided to abandon that and use the bootstrap from outside the image technique instead. I've also started the port of DefaultRubyParser.y from the JRuby source. I have a simple Ruby script that uses Regexps to transform the DefaultRubyParser.ry into Ruby. (Regexps are not powerful enough to tackle the job generically -- that would take a Context Free Grammar -- but I am including enough specific context in the regexps that it will work in the specific cases needed.)

I have also worked out and demonstrated the manipulation of method dictionaries so that we can compile fully qualified Ruby method names for the VisualWorks VM. I have sketched out a design for handling Ruby Mixins and assignment-implicit accessors.

I have also recruited a helper for translating the AST node classes. (There is something to be said for little languages. It's a lot easier to build syntax driven tools for them!)

Next, I will be taking a very simplified Ruby subset (the calculator example from the "Dragon Book" which comes with Racc, and the ability to define methods in Ruby classes) and working out the Parser-Outside-The-Image scheme with that. Hopefully I'll have that by this weekend.

--Peter

···

On Apr 7, 2005, at 11:32 AM, Lothar Scholz wrote:

Hello Peter,

> Hello everyone,

> I thought I'd talk about my new project here, since there is a good
> chance that someone might be interested in it. I'm planning to put

Is this just brainstorming or have you already done something ?

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Guest15 (Guest)]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

See the MetaRuby project:

http://blog.zenspider.com/archives/metaruby/index.html (among other places)

(Although this is about Ruby in Ruby, not Ruby in smalltalk)

···

On Apr 7, 2005 12:19 PM, jc <james.cromwell@gmail.com> wrote:

-jc

--
thanks,
-pate
-------------------------
We are often unable to tell people what they need to know, because
they want to know something else, and would therefore only
misunderstand what we said
- the Raven (George MacDonald, Lilith)

[Peter_Suk]

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

Basically, you could treat the image as just a very powerful debugger/IDE. The image would spit out new versions of your module files, which you could then edit/diff to your liking. I have devised a very powerful algorithm to respect comments, based on diffs of token sequences.

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

Initially, there would be some Smalltalk behind the Ruby, but we'd fix it so you'd never see it. Eventually, we can develop a completely Ruby meta-layer.

···

On Apr 7, 2005, at 2:19 PM, jc wrote:

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Peter_Suk]

In the Smalltalk debuggers, you can spawn off as many debugger sessions as you want by highlighting arbitrary code in the debugger as well as any browser window, as well as visually "inspect" any object (with each one appearing in a window), with the context of every window sensitive to the class being displayed. The effect is basically an "irb-anywhere" which you can arbitrarily interleave with any number of visual inspects. That, and a self-hosted language, faster VM, more garbage collection options, etc...

···

On Apr 7, 2005, at 2:39 PM, Florian Groß wrote:

Peter Suk wrote:

I thought I'd talk about my new project here, since there is a good chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

This does indeed sound promising. Having a self-hosted Ruby would have
certain benefits as well.

a wonderful debugger which will let programmers modify methods on the
fly & continue execution

That is already possible with ruby-breakpoint which spawns an IRB
session at a specific place of your code. See
http://ruby-breakpoint.rubyforge.org/ if you are interested in this sort
of things.

It sure can't hurt to have more options, though.

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Ptkwt]

In article <1112914652.066124.91370@g14g2000cwa.googlegroups.com>,

Peter Suk wrote:

Hello everyone,

I thought I'd talk about my new project here, since there is a good
chance that someone might be interested in it. I'm planning to put
Ruby on top of Smalltalk VMs.

Hi Peter,

I'm definitely interested in this; please keep me posted. Actually, a
couple of people are aware that I was looking into the possibility of
doing this as a commercial product - since, from my benchmarking, we
can get about a 20x to 30x speed increase by hosting Ruby on a
commercial Smalltalk VM like VisualWorks, it seemed like it might be

This "20x to 30x speed increase" stuff is interesting. I would have
thought it would be closer to something like 5x. Can anyone elaborate on
how those increases are being acheived in the SmallTalk VMs? I'm wondering
if it might also be worthwhile to incorporate some of these ideas into YARV -
parhaps that would get us to high-speed Ruby even faster than trying to
put Ruby on top of a SmallTalk VM?

Phil

···

Avi Bryant <avi.bryant@gmail.com> wrote:

[Peter_Suk]

"Peter Suk" <peter.kwangjun.suk@mac.com> wrote in message

I thought I'd talk about my new project here

Sounds fantastic, I can't wait!

And for those of you sharp enough to wonder: yes, we can
handle Modules, Mixins, and fully qualified Method names without
changing the Smalltalk VMs. (At least those that have Namespaces.)

Which ones do?

Avi pointed out that we don't really need them -- Smalltalk Namespaces are really just data-structures to inform the Compiler, so we can have Namespaces on top of any Smalltalk VM.

And what of Ruby 2.0 selector namesapces? Will they happen? And will that
map readily to a Smalltalk VM?

Selector namespaces are hard. We can do them with compiler tricks. We'll just have to tackle that when it comes. (And who knows, maybe Ruby2/Rite will make the Alumina (Ruby on a Smalltalk VM) superfluous? I hope so. I hope Ruby2/Rite will be right and we'll have a meta-level that is freely manipulable as 1st class Objects.

--Peter

···

On Apr 8, 2005, at 2:39 PM, itsme213 wrote:

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Peter_Suk]

I am aware of MetaRuby. This project is also about Ruby in Ruby. (Smalltalk is in Smalltalk. But the VM doesn't care which language/meta-language it is running!)

However, this meta-Ruby will run two orders of magnitude faster!

···

On Apr 7, 2005, at 2:36 PM, pat eyler wrote:

On Apr 7, 2005 12:19 PM, jc <james.cromwell@gmail.com> wrote:

Hi Peter,

- Would this support programming both within or without an image,
choice up to the user?

- From what I understand Smalltalk has a small VM written in C, with
the rest written in Smalltalk itself. Would your version be the Ruby
equivalent of this (VM with everything else in Ruby), or would there
still be some Smalltalk behind the Ruby?

See the MetaRuby project:

http://blog.zenspider.com/archives/metaruby/index.html (among other places)

(Although this is about Ruby in Ruby, not Ruby in smalltalk)

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.

[Lothar_Scholz2 (Lothar Scholz)]

Hello Peter,

to the class being displayed. The effect is basically an
"irb-anywhere" which you can arbitrarily interleave with any number of

So it can crash anywhere SCNR.

Yes maybe its possible to transfer part of the language (all?) into an
smalltalk VM. But you can never get all the binary extensions to work,
an emulation layer would kill all your speed benefits if possible
at all. So maybe you should simply announce a ruby branch called SmallRuby,
since this would not have so much todo with the current ruby anymore.

But i'm courious whats your result in the next years.
Is this your Ph.D thesis ?

···

--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's

[Glenn_Parker1 (Glenn Parker)]

So, what are the potential platforms for Ruby on a Smalltalk VM?

···

--
Glenn Parker | glenn.parker-AT-comcast.net | <http://www.tetrafoil.com/>

[Isaac_Gouy1 (Isaac Gouy)]

Phil Tomson wrote:

This "20x to 30x speed increase" stuff is interesting. I would have
thought it would be closer to something like 5x.

More than 5x
http://www.lissett.com/ben/

[Peter_Suk]

Phil,

30X speed increase is for "Real World Applications" and is highly dependent on Garbage Collection. I have seen many dozens of Smalltalk business programs in my time as a consultant. One of the things they do a lot of is create objects and garbage collect them. In fact, memory management is probably the biggest single factor that comes up in the performance of business apps after IO and DB access.

The Generational GC in VisualWorks is truly fantastic. I can sit around all day with an infinite loop that allocates copies of some random string like "Doo-Dah, Doo-Dah, All the live long day" running in the Smalltalk image I am developing in, and there's no obvious effect. (Yes, I did this earlier this week!) I can crank up this loop until it is producing a billion new objects in several minutes, and my image is fine, until I try to do something serious like have the Refactoring Browser rename all implementors of the method #add: in the image, and even then, it;s fine, it just does it slowly.

Another thing you should know in terms of anecdotes. A fellow Smalltalker implemented a bunch of Encryption algorithms a few years back in VisualWorks Smalltalk. (So we're talking lots of low-level bit & byte slewing.) He benchmarked them against the reference DLLs from RSA Data Security, which were written in C. The Smalltalk algorithms ran at a comparable speed. One of them ran 3% faster! Again, this was partly due to memory management. The DLLs naively malloc'd & free'd everything immediately, while the Generational GC in Smalltalk provided what amounted to a buffer cache optimization for free! (The other part was he had our VM guys implement some bit & byte slewing primitive ops.)

And lastly, please remember that VisualWorks and most of the commercial Smalltalk implementations are actually JIT compilers. Most of the time, we're running native machine language instructions.

If YARV out does Smalltalk VMs for speed of Ruby execution, then all the better. The competitive pressure will ultimately be good for the Ruby community. We'll just settle for providing all of the other great benefits I've mentioned. (Like the fastest Ruby-In-Ruby!)

--Peter

···

On Apr 7, 2005, at 9:14 PM, Phil Tomson wrote:

This "20x to 30x speed increase" stuff is interesting. I would have
thought it would be closer to something like 5x. Can anyone elaborate on
how those increases are being acheived in the SmallTalk VMs? I'm wondering
if it might also be worthwhile to incorporate some of these ideas into YARV -
parhaps that would get us to high-speed Ruby even faster than trying to
put Ruby on top of a SmallTalk VM?

--
There's neither heaven nor hell, save what we grant ourselves.
There's neither fairness nor justice, save what we grant each other.