# When little languages grow

**URL:** <https://rubytalk.org/t/when-little-languages-grow/16052>\
**Category:** ruby-talk\
**Created:** [26 January 2005 17:08 UTC](https://rubytalk.org/t/when-little-languages-grow/16052 "2005-01-26T17:08:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 17:08 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/1 "2005-01-26T17:08:04Z")

</div>

I seem to have run into my parsing problem again. Whatever I'm  
doing I usually end up having to parse non-simplistic input, and I'm  
still not happy about the apparently available solutions to this.  
So I'm wondering what other people do.

The application is immaterial at the moment, but the problem is that  
I need to do more than can be done with a simple case statement, and  
if I were to use case statements managing the problem would get too  
big.

The conventional wisdom is to use some form of parser  
generator (Yacc, Bison, Racc, Rockit,...) but I don't have  
confidence in my ability to get these working well.[1].  
I have had great difficulty in the past, certainly.

Other possibilities I have considered and tried are to lash together  
some form of Lisp [cf Greenspun's 10th rule of programming] or Forth,  
but I don't consider myself fluent in either of those languages, and  
they are not as easy a user interface for other people as Ruby would  
be. I can get something working, but find it hard to maintain or  
improve. [2]

So the next possibility is to use something like

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value

or something, and actually make the commands in the language methods  
of some Ruby object.

It is often observed that it is difficult to add security to a  
system, compared to building it in from the start. Can I do this  
and still have a good level of security? Should I make the parser  
object (whose method's I'm using) a subclass of Nil, to limit it as  
much as possible? I need to give people enough rope to hold their  
input together, but not enough to hang themselves (or me). I don't  
want people to be able to execute arbitrary code, or fiddle with  
objects they should not need to touch.

Is there another way to handle input flexibly that I have completely  
missed? I've googled for things to do with little languages and parsing, but have found nothing enlightening.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

[1] I find that thinking in the manner of a shift/reduce parser is  
particularly unnatural to me. This might just be a weakness on my  
part or may have something to do with people's difficulties in  
handling modal interfaces: it is hard to switch contexts rapidly.  
Maybe there is something I can read which will turn the problem  
around, so it becomes easy to handle?

[2] Immensely powerful and fast systems have been written in Forth,  
and Lisp is very powerful in the right hands. I just don't have the  
experience with these to be effective, yet.

---

<div class="post-metadata">

**Author:** ![Zach\_Dennis1](https://avatars.discourse-cdn.com/v4/letter/z/ea5d25/32.png) [@Zach\_Dennis1](https://rubytalk.org/u/Zach_Dennis1)\
**Post date:** [26 January 2005 17:24 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/2 "2005-01-26T17:24:07Z")

</div>

To make sure myself and the rest of the list is correctly hearing you question. You are interested in the ruby way to write a code generator?

And you are looking for input other parsing or implemented solutions others may have experience with, with languges and tools such as; lisp, forth, yacc, bison, racc, etc.. ?

Zach

Hugh Sasse Staff Elec Eng wrote:

> **···**
>
> > I seem to have run into my parsing problem again. Whatever I'm  
> > doing I usually end up having to parse non-simplistic input, and I'm  
> > still not happy about the apparently available solutions to this.  
> > So I'm wondering what other people do.
> > 
> > The application is immaterial at the moment, but the problem is that  
> > I need to do more than can be done with a simple case statement, and  
> > if I were to use case statements managing the problem would get too  
> > big.
> > 
> > The conventional wisdom is to use some form of parser  
> > generator (Yacc, Bison, Racc, Rockit,...) but I don't have  
> > confidence in my ability to get these working well.[1].  
> > I have had great difficulty in the past, certainly.
> > 
> > Other possibilities I have considered and tried are to lash together  
> > some form of Lisp [cf Greenspun's 10th rule of programming] or Forth,  
> > but I don't consider myself fluent in either of those languages, and  
> > they are not as easy a user interface for other people as Ruby would  
> > be. I can get something working, but find it hard to maintain or  
> > improve. [2]
> > 
> > So the next possibility is to use something like
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value
> > 
> > or something, and actually make the commands in the language methods  
> > of some Ruby object.
> > 
> > It is often observed that it is difficult to add security to a  
> > system, compared to building it in from the start. Can I do this  
> > and still have a good level of security? Should I make the parser  
> > object (whose method's I'm using) a subclass of Nil, to limit it as  
> > much as possible? I need to give people enough rope to hold their  
> > input together, but not enough to hang themselves (or me). I don't  
> > want people to be able to execute arbitrary code, or fiddle with  
> > objects they should not need to touch.
> > 
> > Is there another way to handle input flexibly that I have completely  
> > missed? I've googled for things to do with little languages and parsing, but have found nothing enlightening.
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you,  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh
> > 
> > [1] I find that thinking in the manner of a shift/reduce parser is  
> > particularly unnatural to me. This might just be a weakness on my  
> > part or may have something to do with people's difficulties in  
> > handling modal interfaces: it is hard to switch contexts rapidly.  
> > Maybe there is something I can read which will turn the problem  
> > around, so it becomes easy to handle?
> > 
> > [2] Immensely powerful and fast systems have been written in Forth,  
> > and Lisp is very powerful in the right hands. I just don't have the  
> > experience with these to be effective, yet.

---

<div class="post-metadata">

**Author:** ![ts1](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@ts1](https://rubytalk.org/u/ts1)\
**Post date:** [26 January 2005 17:28 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/3 "2005-01-26T17:28:30Z")

</div>

> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value

Sorry to say this but this is the most common error that I see when  
someone try to eval some code with $SAFE \>= 4

The code will be eval'ed with $SAFE \>= 4 but the result (#value) will be  
used with $SAFE = 0 and you can have problems.

The result of #eval must be cleaned with $SAFE \>= 4, before it's  
returned.

Guy Decoux

---

<div class="post-metadata">

**Author:** ![Robert](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@Robert](https://rubytalk.org/u/Robert)\
**Post date:** [26 January 2005 17:50 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/4 "2005-01-26T17:50:51Z")

</div>

Hugh,

the one thing I didn't see in your posting is a statement about the language. What capabilities should it have? If it's just assigning constants to vars (like often needed for configurations) then Regexp is probably fine. From what you write I'm guessing that your envisioned language is more complex - but how complex? Maybe it's a special case for which someone somewhere has a solution already.

Regards

&nbsp;&nbsp;&nbsp;&nbsp;robert

"Hugh Sasse Staff Elec Eng" \<hgs@dmu.ac.uk\> schrieb im Newsbeitrag news:Pine.GSO.4.60.0501261622400.24999@brains.eng.cse.dmu.ac.uk...

> **···**
>
> > I seem to have run into my parsing problem again. Whatever I'm  
> > doing I usually end up having to parse non-simplistic input, and I'm  
> > still not happy about the apparently available solutions to this.  
> > So I'm wondering what other people do.
> > 
> > The application is immaterial at the moment, but the problem is that  
> > I need to do more than can be done with a simple case statement, and  
> > if I were to use case statements managing the problem would get too  
> > big.
> > 
> > The conventional wisdom is to use some form of parser  
> > generator (Yacc, Bison, Racc, Rockit,...) but I don't have  
> > confidence in my ability to get these working well.[1].  
> > I have had great difficulty in the past, certainly.
> > 
> > Other possibilities I have considered and tried are to lash together  
> > some form of Lisp [cf Greenspun's 10th rule of programming] or Forth,  
> > but I don't consider myself fluent in either of those languages, and  
> > they are not as easy a user interface for other people as Ruby would  
> > be. I can get something working, but find it hard to maintain or  
> > improve. [2]
> > 
> > So the next possibility is to use something like
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value
> > 
> > or something, and actually make the commands in the language methods  
> > of some Ruby object.
> > 
> > It is often observed that it is difficult to add security to a  
> > system, compared to building it in from the start. Can I do this  
> > and still have a good level of security? Should I make the parser  
> > object (whose method's I'm using) a subclass of Nil, to limit it as  
> > much as possible? I need to give people enough rope to hold their  
> > input together, but not enough to hang themselves (or me). I don't  
> > want people to be able to execute arbitrary code, or fiddle with  
> > objects they should not need to touch.
> > 
> > Is there another way to handle input flexibly that I have completely  
> > missed? I've googled for things to do with little languages and parsing, but have found nothing enlightening.
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you,  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh
> > 
> > [1] I find that thinking in the manner of a shift/reduce parser is  
> > particularly unnatural to me. This might just be a weakness on my  
> > part or may have something to do with people's difficulties in  
> > handling modal interfaces: it is hard to switch contexts rapidly.  
> > Maybe there is something I can read which will turn the problem  
> > around, so it becomes easy to handle?
> > 
> > [2] Immensely powerful and fast systems have been written in Forth,  
> > and Lisp is very powerful in the right hands. I just don't have the  
> > experience with these to be effective, yet.

---

<div class="post-metadata">

**Author:** ![why\_the\_lucky\_stiff1](https://avatars.discourse-cdn.com/v4/letter/w/898d66/32.png) [@why\_the\_lucky\_stiff1](https://rubytalk.org/u/why_the_lucky_stiff1)\
**Post date:** [26 January 2005 17:55 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/5 "2005-01-26T17:55:52Z")

</div>

> The application is immaterial at the moment, but the problem is that  
> I need to do more than can be done with a simple case statement, and  
> if I were to use case statements managing the problem would get too  
> big.
> 
> ...
> 
> So the next possibility is to use something like
> 
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value
> 
> or something, and actually make the commands in the language methods  
> of some Ruby object.

I hear ya. I wish there was a way to open a jailed namespace. It'd be  
like chroot'ing into a module. The sandbox module would be addressable  
offered under chroot.

\_why

> **···**
>
> Hugh Sasse Staff Elec Eng (hgs@dmu.ac.uk) wrote:  
> from ::Object, but would only include a limited set of modules when

---

<div class="post-metadata">

**Author:** ![Mark\_Probert](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@Mark\_Probert](https://rubytalk.org/u/Mark_Probert)\
**Post date:** [26 January 2005 18:06 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/6 "2005-01-26T18:06:48Z")

</div>

Hi ..

> I seem to have run into my parsing problem again. Whatever I'm  
> doing I usually end up having to parse non-simplistic input, and I'm  
> still not happy about the apparently available solutions to this.  
> So I'm wondering what other people do.

My personal solution to this is to use Coco/R, an LL(1) scanner/generator.  
You can find more information at:

&nbsp;&nbsp;[http://www.scifac.ru.ac.za/coco](http://www.scifac.ru.ac.za/coco)

The primary advantage of this approach, IMHO, is that all of the grammar /  
scanning rules are in a single file (rather than the lex/yacc approach).  
This makes the grammar quite easy to read and extend, once you are familiar  
with the process. Ryan Davies has a pure ruby version, and I have a ruby  
extension version. Both seem to work well for little languages.

> [1] I find that thinking in the manner of a shift/reduce parser is  
> particularly unnatural to me. ... Maybe there is something I can read which  
> will turn the problem around, so it becomes easy to handle?

Pat Terry has a book "Compilers and Compiler Generators" that covers LL(1)  
(and other) topics very well. You can find it at:

&nbsp;&nbsp;[http://www.scifac.ru.ac.za/compilers/](http://www.scifac.ru.ac.za/compilers/)

The primary disadvantage of Coco/R is the LL(1) part. This means that your  
grammar needs to be fairly well formed and not arbitrarily complex. As an  
example, Ruby can not, as far as I have tried, be converted into an LL(1)  
grammar, though C can.

A simple example of the ruby grammar (this is for the famous four function  
calculator) for my extension library. Note that this will generate a Ruby  
extension. When you compile and link, you can use it in Ruby like this:

# ---( test.rb )-------------  
require 'Calc'

f = File.readlines("calc.inp")  
t = Calc.new  
t.run(f)

if t.success  
&nbsp;&nbsp;&nbsp;puts "parsed ok!"  
&nbsp;&nbsp;&nbsp;t.capture.each { |ans| puts " ans==#{ans}" }  
else  
&nbsp;&nbsp;&nbsp;puts "Errors ::"  
&nbsp;&nbsp;&nbsp;t.errs.each { |err| puts " --\> #{err}" }  
end

# ---( calc.inp )-----------  
var a,b,c,d;

write 1+(2\*3)+4;  
write 100/10;

write a;  
b := a\*16;  
write b\*2

# ---( calc.atg )-----------  
$C /\* Generate Main Module \*/  
COMPILER Calc

#define upcase(c) ((c \>= 'a' && c \<= 'z')? c-32:c)  
int VARS[10000];

int get\_spix()  
{  
&nbsp;&nbsp;char name[20];  
&nbsp;&nbsp;LEX\_S(name, sizeof(name) - 1);  
&nbsp;&nbsp;if (strlen(name) \>= 2)  
&nbsp;&nbsp;&nbsp;&nbsp;return 26\*(upcase(name[1])-'A')+(upcase(name[0])-'A');  
&nbsp;&nbsp;else  
&nbsp;&nbsp;&nbsp;&nbsp;return (upcase(name[0])-'A');  
}

int get\_number()  
{  
&nbsp;&nbsp;char name[20];  
&nbsp;&nbsp;LEX\_S(name, sizeof(name) - 1);  
&nbsp;&nbsp;return atoi(name);  
}

void new\_var(int spix)  
{  
&nbsp;&nbsp;VARS[spix] = 0;  
}

int get\_var(int spix)  
{  
&nbsp;&nbsp;return VARS[spix];  
}

void write\_val(int val)  
{  
&nbsp;&nbsp;char tmp[20];

&nbsp;&nbsp;sprintf(tmp, "%d", val);  
&nbsp;&nbsp;t\_capture\_output(tmp);  
}

void set\_var(int spix, int val)  
{  
&nbsp;&nbsp;VARS[spix] = val;  
}

IGNORE CASE

CHARACTERS  
&nbsp;&nbsp;letter = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz".  
&nbsp;&nbsp;digit = "0123456789".  
&nbsp;&nbsp;eol = CHR(13) .  
&nbsp;&nbsp;lf = CHR(10) .

COMMENTS  
&nbsp;&nbsp;FROM '--' TO eol

IGNORE eol + lf

TOKENS  
&nbsp;&nbsp;ident = letter {letter | digit} .  
&nbsp;&nbsp;number = digit {digit} .

PRODUCTIONS  
&nbsp;&nbsp;Calc =  
&nbsp;&nbsp;&nbsp;&nbsp;[Declarations] StatSeq .

&nbsp;&nbsp;Declarations  
&nbsp;&nbsp;&nbsp;&nbsp;= (. int spix; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'VAR'  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Ident \<&spix\> (. new\_var(spix); .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ ',' Ident \<&spix\> (. new\_var(spix); .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} ';'.

&nbsp;&nbsp;StatSeq =  
&nbsp;&nbsp;&nbsp;&nbsp;Stat {';' Stat}.

&nbsp;&nbsp;Stat  
&nbsp;&nbsp;&nbsp;&nbsp;= (. int spix, val; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> "WRITE" Expr \<&val\> (. write\_val(val); .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> Ident \<&spix\> ":=" Expr \<&val\> (. set\_var(spix, val); .) .

&nbsp;&nbsp;Expr \<int \*exprVal\>  
&nbsp;&nbsp;&nbsp;&nbsp;= (. int termVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Term \<exprVal\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ '+' Term \<&termVal\> (. \*exprVal += termVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> '-' Term \<&termVal\> (. \*exprVal -= termVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} .

&nbsp;&nbsp;Term \<int \*termVal\>  
&nbsp;&nbsp;&nbsp;&nbsp;= (. int factVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Fact \<termVal\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ '\*' Fact \<&factVal\> (. \*termVal \*= factVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> '/' Fact \<&factVal\> (. \*termVal /= factVal; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} .

&nbsp;&nbsp;Fact \<int \*factVal\>  
&nbsp;&nbsp;&nbsp;&nbsp;= (. int spix; .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Ident \<&spix\> (. \*factVal = get\_var(spix); .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> number (. \*factVal = get\_number(); .)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> '(' Expr \<factVal\> ')' .

&nbsp;&nbsp;Ident \<int \*spix\>  
&nbsp;&nbsp;&nbsp;&nbsp;= ident (. \*spix = get\_spix(); .) .

END Calc.

I hope that this helps.

Regards,

> **···**
>
> On Wednesday 26 January 2005 09:08, Hugh Sasse Staff Elec Eng wrote:  
> a := 37-12-(4\*5);
> 
> --  
> -mark. (probertm at acm dot org)

---

<div class="post-metadata">

**Author:** ![PA1](https://avatars.discourse-cdn.com/v4/letter/p/71c47a/32.png) [@PA1](https://rubytalk.org/u/PA1)\
**Post date:** [26 January 2005 18:13 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/7 "2005-01-26T18:13:15Z")

</div>

Perhaps you should take a look at Lua:

[http://www.lua.org/about.html](http://www.lua.org/about.html)

It started life as a "data entry language" of sort:

[http://www.lua.org/history.html](http://www.lua.org/history.html)

Then read the book:

[http://www.lua.org/pil/](http://www.lua.org/pil/)

Cheers

> **···**
>
> On Jan 26, 2005, at 18:08, Hugh Sasse Staff Elec Eng wrote:
> 
> > I've googled for things to do with little languages and parsing, but have found nothing enlightening.
> 
> --  
> PA  
> [http://alt.textdrive.com/](http://alt.textdrive.com/)

---

<div class="post-metadata">

**Author:** ![7rans](https://avatars.discourse-cdn.com/v4/letter/7/a8b319/32.png) [@7rans](https://rubytalk.org/u/7rans)\
**Post date:** [26 January 2005 19:05 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/8 "2005-01-26T19:05:47Z")

</div>

Hi Hugh,

What are you trying to parse exactly? Is it full Ruby source or a small  
subset?

If full source, have you considered what it is exactly that might be  
unsafe? Maybe there's a way to create you own "Safety Net" by making  
certain parts of Ruby inaccessable. Not sure how extactly, seems like  
namepsaces would be needed, but it would be an interesting challange.  
Also you might try ParseTree.

OTOH, if only a subset (or something completey different), I have a  
general purpose and easy to use Parser class I've been working on for  
Carats. Perhaps you'd like to try it and see if can help? Doing so  
could also help me test/improve it for everyone.

T.

---

<div class="post-metadata">

**Author:** ![Sam\_Roberts1](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@Sam\_Roberts1](https://rubytalk.org/u/Sam_Roberts1)\
**Post date:** [27 January 2005 02:18 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/9 "2005-01-27T02:18:26Z")

</div>

Quoteing hgs@dmu.ac.uk, on Thu, Jan 27, 2005 at 02:08:04AM +0900:

> I seem to have run into my parsing problem again. Whatever I'm  
> doing I usually end up having to parse non-simplistic input, and I'm  
> still not happy about the apparently available solutions to this.  
> So I'm wondering what other people do.
> 
> The application is immaterial at the moment, but the problem is that  
> I need to do more than can be done with a simple case statement, and  
> if I were to use case statements managing the problem would get too  
> big.
> 
> The conventional wisdom is to use some form of parser  
> generator (Yacc, Bison, Racc, Rockit,...) but I don't have  
> confidence in my ability to get these working well.[1].  
> I have had great difficulty in the past, certainly.

snip

> [1] I find that thinking in the manner of a shift/reduce parser is  
> particularly unnatural to me. This might just be a weakness on my  
> part or may have something to do with people's difficulties in  
> handling modal interfaces: it is hard to switch contexts rapidly.  
> Maybe there is something I can read which will turn the problem  
> around, so it becomes easy to handle?

Hi Hugh.

Two suggestions:

Most of the docs for parser-generators assume some (lots?) knowledge of  
compiler theory. An exception is the O'Reilly Lex&Yacc book. If any of  
the generators you mention above is anything like Yacc, I think you  
might like the book. Its at a practical level, it assumes you are a  
competent programmer, you just haven't taken a compiler course. The  
first few chapters got me up to speed very quickly (I think you develop  
a calculator). After that, the example used is generating a  
mini-language to specify curses screen layouts. I don't do any curses  
stuff, but the examples were used to good effect.

I think that if you do this often, mastering one of these tools will be  
a life skill you'll never regret! And yacc (or one of the others) is a  
mini-language in its own right, always good to learn another.

On another track, I have deliberately NOT used these grammers when  
parseing the BNF for internet email. In my experience, if you can write  
the grammer down in BNF, its usually pretty easy to write a recursive  
descent parser for it. It's a little tedious though, which is why all  
the tools exist to generate the code for you...

Have fun,  
Sam

---

<div class="post-metadata">

**Author:** ![Kaspar\_Schiess](https://avatars.discourse-cdn.com/v4/letter/k/d78d45/32.png) [@Kaspar\_Schiess](https://rubytalk.org/u/Kaspar_Schiess)\
**Post date:** [27 January 2005 08:10 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/10 "2005-01-27T08:10:35Z")

</div>

(In response to news:Pine.GSO.4.60.0501261622400.24999  
@brains.eng.cse.dmu.ac.uk by Hugh Sasse Staff Elec Eng)

> I seem to have run into my parsing problem again. Whatever I'm  
> doing I usually end up having to parse non-simplistic input, and I'm  
> still not happy about the apparently available solutions to this.  
> So I'm wondering what other people do.

Just a few random thoughts:

- All of this depends on how complex your mini-language is. Should it be  
Turing complete or just really something like fstab ? See  
[http://www.faqs.org/docs/artu/ch08s01.html](http://www.faqs.org/docs/artu/ch08s01.html) (a chapter from the Art of  
Unix Programming, including a taxonomy of languages).

- If your language needs loopiness and control constructs like Ruby has  
(or should look like Ruby), then you could help me think about a cut-down  
version of Ruby for configuration files: This version of Ruby should be  
accessible from within Ruby and be totally safe (=cut off of the external  
world). Such a thing would have to be execution timeouted (endless loops  
and stack overflows) too...

- Your problem was essentially the topic of one of my semester projects.  
I tried out a lot of different parser generators, for different  
languages. None really could score in the category of simplicity.  
Probably just a complex subject. But the problem of having to create a  
minilanguage pops up all the time, as you say.

- Parse error reporting: A lot of programming languages have bad parse  
error reporting, which essentially comes from the fact that the parser  
generator does not help you in this. ANTLR is a bit better there, but IMO  
still sucks.

I think there are two really good ideas in this thread:  
a) Create and Integrate Grammars as first class objects in Ruby.  
(Clifford Heath)  
b) Use a cut-down version of Ruby that is designed with security in mind  
(various people, this has been done).

I can't help feeling that we're missing a really simple way to do all of  
this ...

kaspar

hand manufactured code - www.tua.ch/ruby

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 17:42 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/11 "2005-01-26T17:42:52Z")

</div>

> To make sure myself and the rest of the list is correctly hearing you question. You are interested in the ruby way to write a code generator?

There's probably more than one way, but yes. I don't need to  
generate an executable for later use, so interpreting my input is  
fine. I need to manage the complexity so I can cope with future  
expansion if any.

> And you are looking for input other parsing or implemented solutions others may have experience with, with languges and tools such as; lisp, forth, yacc, bison, racc, etc.. ?

I'm looking to do this in Ruby. Experience of things that simplify  
this, whether they come from other languages or not, is what I am  
after. I mention the other languages because I have tried their  
styles of handling this problem. My success has been limited. So,  
what can I do for more success? 🙂 Look at the problem differntly?  
Use another technique?

> Zach

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hope that is clearer,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, Zach Dennis wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 17:48 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/12 "2005-01-26T17:48:28Z")

</div>

> \> Thread.new(input){|source|  
> \> $SAFE=5  
> \> instance\_eval source  
> \> }.value
> 
> Sorry to say this but this is the most common error that I see when  
> someone try to eval some code with $SAFE \>= 4
> 
> The code will be eval'ed with $SAFE \>= 4 but the result (#value) will be  
> used with $SAFE = 0 and you can have problems.

Yes, that's a good point.

> The result of #eval must be cleaned with $SAFE \>= 4, before it's  
> returned.

Thank you. That would be better than cleaning it aftwerwards, I'd  
not really considered that risk.

> Guy Decoux

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, ts wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 18:18 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/13 "2005-01-26T18:18:14Z")

</div>

> Hugh,
> 
> the one thing I didn't see in your posting is a statement about the language. What capabilities should it have? If it's just assigning constants to vars

I was trying to keep this general because I run into the problem of  
parsing non-simplistic grammars so often. It's easy to do the  
&nbsp;&nbsp;\<verb\> \<direct object\> type grammars with lots of verbs[1], but...

My present example is that I want to parse Constructive Solid  
Geometry descriptions, at the moment limited to cones, spheres,  
bricks with the co-ordinates specified, and I need to specify  
material types as well.

I'd also like to be able to declare new objects so they can be  
placed. Silly example: Get two small spheres to cap the ends of  
a cylinder and call the result a Sausage. Then place several  
Sausages in the space at different points.

Later I'd have to extend the language to be able to rotate them into  
psoition. Lots of creeping featurism is likely, I suspect. I  
didn't have material types to deal with before.

> (like often needed for configurations) then Regexp is probably fine. From

Agreed

> what you write I'm guessing that your envisioned language is more complex - but how complex? Maybe it's a special case for which someone somewhere has a solution already.
> 
> Regards
> 
> &nbsp;&nbsp;robert

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

[1] some years back I got Arthur Secret's Agora (Perl, web by email)  
program working and extended it considerably. I used regexps for  
that.

> **···**
>
> On Thu, 27 Jan 2005, Robert Klemme wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 18:21 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/14 "2005-01-26T18:21:25Z")

</div>

[...]

> > So the next possibility is to use something like
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value
> > 
> > or something, and actually make the commands in the language methods  
> > of some Ruby object.
> 
> I hear ya. I wish there was a way to open a jailed namespace. It'd be  
> like chroot'ing into a module. The sandbox module would be addressable  
> from ::Object, but would only include a limited set of modules when  
> offered under chroot.

That's a nice metaphor...

> \_why

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[I'll have to explore that RedHanded site a bit more too1  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;:-)]

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, why the lucky stiff wrote:
> 
> > Hugh Sasse Staff Elec Eng (hgs@dmu.ac.uk) wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 18:28 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/15 "2005-01-26T18:28:11Z")

</div>

> > I've googled for things to do with little languages and parsing, but have found nothing enlightening.
> 
> Perhaps you should take a look at Lua:

I really like Lua, (bought the book), but I need to keep this as pure Ruby.  
Given time, energy, etc I'd love to provide Lua tables to Ruby.... I  
don't think they'd solve my problem here though.

> --  
> PA  
> [http://alt.textdrive.com/](http://alt.textdrive.com/)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, PA wrote:
> 
> > On Jan 26, 2005, at 18:08, Hugh Sasse Staff Elec Eng wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 18:33 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/16 "2005-01-26T18:33:07Z")

</div>

> Hi ..
> 
> > I seem to have run into my parsing problem again. Whatever I'm  
> > doing I usually end up having to parse non-simplistic input, and I'm  
> > still not happy about the apparently available solutions to this.  
> > So I'm wondering what other people do.
> 
> My personal solution to this is to use Coco/R, an LL(1) scanner/generator.  
> You can find more information at:
> 
> [http://www.scifac.ru.ac.za/coco](http://www.scifac.ru.ac.za/coco)

Thank you. I'd seen this on RAA but not explored it...

> The primary advantage of this approach, IMHO, is that all of the grammar /  
> scanning rules are in a single file (rather than the lex/yacc approach).  
> This makes the grammar quite easy to read and extend, once you are familiar  
> with the process. Ryan Davies has a pure ruby version, and I have a ruby  
> extension version. Both seem to work well for little languages.

I'll probably stay with the pure ruby one, but I'll certainly look  
at this. I suspect that the 1 might be the problem for what I'm  
trying to do. I can't remember how LL differs from LR now, but I'll  
find that pretty easily.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[...]

> Pat Terry has a book "Compilers and Compiler Generators" that covers LL(1)  
> (and other) topics very well. You can find it at:
> 
> [Missing cookie](http://www.scifac.ru.ac.za/compilers/)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, Mark Probert wrote:
> 
> > On Wednesday 26 January 2005 09:08, Hugh Sasse Staff Elec Eng wrote:

---

<div class="post-metadata">

**Author:** ![Booker\_C\_Bense8](https://avatars.discourse-cdn.com/v4/letter/b/c89c15/32.png) [@Booker\_C\_Bense8](https://rubytalk.org/u/Booker_C_Bense8)\
**Post date:** [26 January 2005 18:40 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/17 "2005-01-26T18:40:50Z")

</div>

In article \<20050126175623.GB35232@topi.cc\>,

> > The application is immaterial at the moment, but the problem is that  
> > I need to do more than can be done with a simple case statement, and  
> > if I were to use case statements managing the problem would get too  
> > big.
> > 
> > ...
> > 
> > So the next possibility is to use something like
> > 
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = nil  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File.open("input.txt"){|f|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;input = f.read  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thread.new(input){|source|  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$SAFE=5  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;instance\_eval source  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}.value
> > 
> > or something, and actually make the commands in the language methods  
> > of some Ruby object.
> 
> I hear ya. I wish there was a way to open a jailed namespace. It'd be  
> like chroot'ing into a module. The sandbox module would be addressable  
> from ::Object, but would only include a limited set of modules when  
> offered under chroot.

\_ This is the one place that Tcl ( at least circa 1994 Tcl )  
absolutely fits like a glove. There was a client/server program  
called sysctl written by some guys at IBM that allowed you to  
assign ACL's to every command in the language. I've never seen  
any other secure distributed scripting system that comes close.

\_ Of course it was vast overkill for 99% of what you need to  
do with that kind of system. As much as I dislike Tcl for other  
reasons, it is a very good language for extending applications  
via simple scripting. Adding your own specialized commands is  
fairly straightforward. I know quite a few scientific groups  
that are using Python to do similar things these days.

\_ Booker C. Bense

> **···**
>
> why the lucky stiff \<ruby-talk@whytheluckystiff.net\> wrote:
> 
> > Hugh Sasse Staff Elec Eng (hgs@dmu.ac.uk) wrote:

---

<div class="post-metadata">

**Author:** ![Hugh\_Sasse](https://avatars.discourse-cdn.com/v4/letter/h/d6d6ee/32.png) [@Hugh\_Sasse](https://rubytalk.org/u/Hugh_Sasse)\
**Post date:** [26 January 2005 19:31 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/18 "2005-01-26T19:31:05Z")

</div>

> Hi Hugh,
> 
> What are you trying to parse exactly? Is it full Ruby source or a small  
> subset?

Not full ruby source, just a mini-language, maybe using Ruby's  
parser instead of botching(!) my own...

> If full source, have you considered what it is exactly that might be

I don't know what would be unsafe yet: I'm assuming that there are  
some smart people out there who like doing heap overruns and other  
feindish tricks that I'd not envisage.... So, I'd like to be safe  
by default rather than be safe for the cases I have considered.

> unsafe? Maybe there's a way to create you own "Safety Net" by making  
> certain parts of Ruby inaccessable. Not sure how extactly, seems like  
> namepsaces would be needed, but it would be an interesting challange.  
> Also you might try ParseTree.

I'll have a look for that, thanks.

> OTOH, if only a subset (or something completey different), I have a  
> general purpose and easy to use Parser class I've been working on for  
> Carats. Perhaps you'd like to try it and see if can help? Doing so  
> could also help me test/improve it for everyone.

OK, bowl the URL in my direction! Thank you.

> T.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hugh

> **···**
>
> On Thu, 27 Jan 2005, Trans wrote:
> 
> >

---

<div class="post-metadata">

**Author:** ![George1](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@George1](https://rubytalk.org/u/George1)\
**Post date:** [26 January 2005 19:45 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/19 "2005-01-26T19:45:48Z")

</div>

> I hear ya. I wish there was a way to open a jailed namespace. It'd

be

> like chroot'ing into a module. The sandbox module would be

addressable

> from ::Object, but would only include a limited set of modules when  
> offered under chroot.

Sounds like capability security, where - to super-simplify - you can  
only access the objects/functionality you can name. It's used by the E  
language - see a good description at

[http://www.skyhunter.com/marcs/ewalnut.html#SEC42](http://www.skyhunter.com/marcs/ewalnut.html#SEC42)

My guess though is that Ruby is too 'sloppy' for such a thing to work  
fully securely. I'd be very happy to be proved wrong!

-- George

---

<div class="post-metadata">

**Author:** ![Clifford\_Heath4](https://avatars.discourse-cdn.com/v4/letter/c/c68b51/32.png) [@Clifford\_Heath4](https://rubytalk.org/u/Clifford_Heath4)\
**Post date:** [26 January 2005 23:40 UTC](https://rubytalk.org/t/when-little-languages-grow/16052/20 "2005-01-26T23:40:56Z")

</div>

Mark Probert wrote:

> My personal solution to this is to use Coco/R, an LL(1) scanner/generator.

I haven't used Coco, but I'd second Mark's recommendation to stay with  
an LL(1) parser if possible. If not, then LL(n) ala ANTLR, but not LALR  
ala yacc. The LL parsers are easy to write manually using recursive  
descent, which I've done a few times.

> > [1] I find that thinking in the manner of a shift/reduce parser is  
> > particularly unnatural to me. ... Maybe there is something I can read which will turn the problem around, so it becomes easy to handle?

Shift just means "delay a decision about what I've just seen". Reduce is  
the operation you do when you do decide. If you explore the ambiguity  
in your grammar rules, these start to make more sense.

> The primary disadvantage of Coco/R is the LL(1) part.

ANTLR does LL(n) for arbitrary n I believe - though you should avoid  
n \> 3 or humans start to have trouble parsing your language :-).

It's a shame that the ANTLR folk at Purdue went Java-only when they  
dropped their old C-based implementation. A multi-lingual ANTLR would  
be super-cool, especially if it would generate Ruby.

> As an example, Ruby can not, as far as I have tried, be converted into an LL(1) grammar, though C can.

Not without a tie-in to the lexical analyser to help recognise goto  
labels, which require LL(2). Such a tie-in is commonly used however.

> A simple example of the ruby grammar

Good example, thanks Mark.

I should point out that the major reason for the success of XML  
(contrary to most of the hyped claims about it) is that it allows  
people to create languages without having to create parsers. Or  
rather, they use an XML parser which yields a DOM, and can process  
the AST at will.

If you can live with the ugliness of XML and the size&speed of Rexml,  
you should consider it.

There's no good reason why a language like Ruby shouldn't have  
grammar rules as first-class objects (as Regexp's are), yielding  
Ruby objects that reflect the AST, allowing attribute-grammar  
parsers to be written and integrated directly within a program.

Such a tool, integrated into the Ruby interpreter itself, would  
allow extension modules to define \*Ruby syntax extensions\*, so  
that the language itself becomes plastic.

I haven't thought much about what these last two features would  
look like in Ruby's case.

Clifford Heath.

[Next page](https://rubytalk.org/t/when-little-languages-grow/16052.md?page=2)
