Is there lightweight lexer for Ruby? I have seen a few lexer
generators out there, but I want something simple that can be used
without generating the lexer first. I really just need the basic
functionaliy of matching a pattern and associating an action with that
pattern. For example, I was thinking something like:
class MyLexer < Lexer
rule /a/ do
# action for pattern a
end
rule /b/ do
# action for pattern b
end
rule /c/ do
# action for pattern c
end
end
l = MyLexer.new(input_stream)
l.each_token do |token|
# do something with token
end
I have something similar to the above that I have created. However, I
wanted to check and see what is out there first before I invested too
much time in it. If no one knows of anything else and\or is interested
in what I have described let me know and I will clean it up and
publish it.
--Meador
StringScanner is close if you combine it with ifs.
···
On May 12, 2006, at 8:35 PM, Meador Inge wrote:
Is there lightweight lexer for Ruby? I have seen a few lexer
generators out there, but I want something simple that can be used
without generating the lexer first. I really just need the basic
functionaliy of matching a pattern and associating an action with that
pattern. For example, I was thinking something like:
class MyLexer < Lexer
rule /a/ do
# action for pattern a
end
rule /b/ do
# action for pattern b
end
rule /c/ do
# action for pattern c
end
end
l = MyLexer.new(input_stream)
l.each_token do |token|
# do something with token
end
I have something similar to the above that I have created. However, I
wanted to check and see what is out there first before I invested too
much time in it. If no one knows of anything else and\or is interested
in what I have described let me know and I will clean it up and
publish it.
--Meador
If you look closely at what traditional tools like lex and flex do, you
could almost take the input to lex and execute it directly in Ruby! It's not
much more than a case statement from regular expressions. I'm not sure what
you mean by "lightweight" but if you mean "fast": a lot of people who wrote
language compilers back in the day (I'm one of them) would often write
scanners in hand-tuned C rather than using lex.
···
On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
Is there lightweight lexer for Ruby? I have seen a few lexer
generators out there, but I want something simple that can be used
without generating the lexer first. I really just need the basic
functionaliy of matching a pattern and associating an action with that
pattern. For example, I was thinking something like:
class MyLexer < Lexer
rule /a/ do
# action for pattern a
end
rule /b/ do
# action for pattern b
end
rule /c/ do
# action for pattern c
end
end
l = MyLexer.new(input_stream)
l.each_token do |token|
# do something with token
end
I have something similar to the above that I have created. However, I
wanted to check and see what is out there first before I invested too
much time in it. If no one knows of anything else and\or is interested
in what I have described let me know and I will clean it up and
publish it.
--Meador
Meador Inge wrote:
Is there lightweight lexer for Ruby?
There's a library called LittleLexer.
http://littlelexer.rubyforge.org/
Cheers,
Dave
If you look closely at what traditional tools like lex and flex do, you
could almost take the input to lex and execute it directly in Ruby!
They do look very similar.
It's not much more than a case statement from regular expressions.
It is a bit more that a case statement on regular expressions:). You have to
do a bit of work to manage the input buffer and ensure that the longest
match is made (e.g. greedy) for each token. Also, if you want some of the
other traditional features of lex and flex you have to add features such as
start states, putting characters back into the input stream, reject matches,
etc...
I'm not sure what you mean by "lightweight"
I more or less meant simple and easy to use (maybe I should have just said
so:)) without the full range of features of a typical lex\flex knockoff.
···
On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
If you look closely at what traditional tools like lex and flex do, you
could almost take the input to lex and execute it directly in Ruby! It's
not
much more than a case statement from regular expressions. I'm not sure
what
you mean by "lightweight" but if you mean "fast": a lot of people who
wrote
language compilers back in the day (I'm one of them) would often write
scanners in hand-tuned C rather than using lex.
On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
>
> Is there lightweight lexer for Ruby? I have seen a few lexer
> generators out there, but I want something simple that can be used
> without generating the lexer first. I really just need the basic
> functionaliy of matching a pattern and associating an action with that
> pattern. For example, I was thinking something like:
> class MyLexer < Lexer
> rule /a/ do
> # action for pattern a
> end
> rule /b/ do
> # action for pattern b
> end
> rule /c/ do
> # action for pattern c
> end
> end
>
> l = MyLexer.new(input_stream)
> l.each_token do |token|
> # do something with token
> end
>
> I have something similar to the above that I have created. However, I
> wanted to check and see what is out there first before I invested too
> much time in it. If no one knows of anything else and\or is interested
> in what I have described let me know and I will clean it up and
> publish it.
>
> --Meador
>
The fact he uses the word "cute" on his site puts me off!
Cheers,
Phil
···
On Sat, May 13, 2006 at 05:19:45 +0900, Dave Burt wrote:
Meador Inge wrote:
> Is there lightweight lexer for Ruby?
There's a library called LittleLexer.
http://littlelexer.rubyforge.org/
Well, yes, that's what I meant by "not much more." In my experience (writing
scanners for programming languages) the start states and greedy matching
were the most important features beyond the basic NFA. Unshifting input
seems like more of a job for the parser than the scanner, but different
strokes, I guess. Are you going to give this a try?
···
On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
> If you look closely at what traditional tools like lex and flex do, you
> could almost take the input to lex and execute it directly in Ruby!
They do look very similar.
> It's not much more than a case statement from regular expressions.
It is a bit more that a case statement on regular expressions:). You have
to
do a bit of work to manage the input buffer and ensure that the longest
match is made (e.g. greedy) for each token. Also, if you want some of the
other traditional features of lex and flex you have to add features such
as
start states, putting characters back into the input stream, reject
matches,
etc...
> I'm not sure what you mean by "lightweight"
I more or less meant simple and easy to use (maybe I should have just said
so:)) without the full range of features of a typical lex\flex knockoff.
On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
>
> If you look closely at what traditional tools like lex and flex do, you
> could almost take the input to lex and execute it directly in Ruby! It's
> not
> much more than a case statement from regular expressions. I'm not sure
> what
> you mean by "lightweight" but if you mean "fast": a lot of people who
> wrote
> language compilers back in the day (I'm one of them) would often write
> scanners in hand-tuned C rather than using lex.
>
> On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
> >
> > Is there lightweight lexer for Ruby? I have seen a few lexer
> > generators out there, but I want something simple that can be used
> > without generating the lexer first. I really just need the basic
> > functionaliy of matching a pattern and associating an action with that
> > pattern. For example, I was thinking something like:
> > class MyLexer < Lexer
> > rule /a/ do
> > # action for pattern a
> > end
> > rule /b/ do
> > # action for pattern b
> > end
> > rule /c/ do
> > # action for pattern c
> > end
> > end
> >
> > l = MyLexer.new(input_stream)
> > l.each_token do |token|
> > # do something with token
> > end
> >
> > I have something similar to the above that I have created. However, I
> > wanted to check and see what is out there first before I invested too
> > much time in it. If no one knows of anything else and\or is interested
> > in what I have described let me know and I will clean it up and
> > publish it.
> >
> > --Meador
> >
>
Are you going to give this a try?
I think I just may. I will be sure to post to ruby-talk if I come up with
anything.
···
On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
Well, yes, that's what I meant by "not much more." In my experience
(writing
scanners for programming languages) the start states and greedy matching
were the most important features beyond the basic NFA. Unshifting input
seems like more of a job for the parser than the scanner, but different
strokes, I guess. Are you going to give this a try?
On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
>
> > If you look closely at what traditional tools like lex and flex do,
you
> > could almost take the input to lex and execute it directly in Ruby!
> They do look very similar.
>
> > It's not much more than a case statement from regular expressions.
> It is a bit more that a case statement on regular expressions:). You
have
> to
> do a bit of work to manage the input buffer and ensure that the longest
> match is made (e.g. greedy) for each token. Also, if you want some of
the
> other traditional features of lex and flex you have to add features such
> as
> start states, putting characters back into the input stream, reject
> matches,
> etc...
>
> > I'm not sure what you mean by "lightweight"
> I more or less meant simple and easy to use (maybe I should have just
said
> so:)) without the full range of features of a typical lex\flex knockoff.
>
> On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
> >
> > If you look closely at what traditional tools like lex and flex do,
you
> > could almost take the input to lex and execute it directly in Ruby!
It's
> > not
> > much more than a case statement from regular expressions. I'm not sure
> > what
> > you mean by "lightweight" but if you mean "fast": a lot of people who
> > wrote
> > language compilers back in the day (I'm one of them) would often write
> > scanners in hand-tuned C rather than using lex.
> >
> > On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
> > >
> > > Is there lightweight lexer for Ruby? I have seen a few lexer
> > > generators out there, but I want something simple that can be used
> > > without generating the lexer first. I really just need the basic
> > > functionaliy of matching a pattern and associating an action with
that
> > > pattern. For example, I was thinking something like:
> > > class MyLexer < Lexer
> > > rule /a/ do
> > > # action for pattern a
> > > end
> > > rule /b/ do
> > > # action for pattern b
> > > end
> > > rule /c/ do
> > > # action for pattern c
> > > end
> > > end
> > >
> > > l = MyLexer.new(input_stream)
> > > l.each_token do |token|
> > > # do something with token
> > > end
> > >
> > > I have something similar to the above that I have created. However,
I
> > > wanted to check and see what is out there first before I invested
too
> > > much time in it. If no one knows of anything else and\or is
interested
> > > in what I have described let me know and I will clean it up and
> > > publish it.
> > >
> > > --Meador
> > >
> >
>
Might be fun to integrate it with racc.
···
On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
> Are you going to give this a try?
I think I just may. I will be sure to post to ruby-talk if I come up with
anything.
On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
>
> Well, yes, that's what I meant by "not much more." In my experience
> (writing
> scanners for programming languages) the start states and greedy matching
> were the most important features beyond the basic NFA. Unshifting input
> seems like more of a job for the parser than the scanner, but different
> strokes, I guess. Are you going to give this a try?
>
> On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
> >
> > > If you look closely at what traditional tools like lex and flex do,
> you
> > > could almost take the input to lex and execute it directly in Ruby!
> > They do look very similar.
> >
> > > It's not much more than a case statement from regular expressions.
> > It is a bit more that a case statement on regular expressions:). You
> have
> > to
> > do a bit of work to manage the input buffer and ensure that the
longest
> > match is made (e.g. greedy) for each token. Also, if you want some of
> the
> > other traditional features of lex and flex you have to add features
such
> > as
> > start states, putting characters back into the input stream, reject
> > matches,
> > etc...
> >
> > > I'm not sure what you mean by "lightweight"
> > I more or less meant simple and easy to use (maybe I should have just
> said
> > so:)) without the full range of features of a typical lex\flex
knockoff.
> >
> > On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
> > >
> > > If you look closely at what traditional tools like lex and flex do,
> you
> > > could almost take the input to lex and execute it directly in Ruby!
> It's
> > > not
> > > much more than a case statement from regular expressions. I'm not
sure
> > > what
> > > you mean by "lightweight" but if you mean "fast": a lot of people
who
> > > wrote
> > > language compilers back in the day (I'm one of them) would often
write
> > > scanners in hand-tuned C rather than using lex.
> > >
> > > On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
> > > >
> > > > Is there lightweight lexer for Ruby? I have seen a few lexer
> > > > generators out there, but I want something simple that can be used
> > > > without generating the lexer first. I really just need the basic
> > > > functionaliy of matching a pattern and associating an action with
> that
> > > > pattern. For example, I was thinking something like:
> > > > class MyLexer < Lexer
> > > > rule /a/ do
> > > > # action for pattern a
> > > > end
> > > > rule /b/ do
> > > > # action for pattern b
> > > > end
> > > > rule /c/ do
> > > > # action for pattern c
> > > > end
> > > > end
> > > >
> > > > l = MyLexer.new(input_stream)
> > > > l.each_token do |token|
> > > > # do something with token
> > > > end
> > > >
> > > > I have something similar to the above that I have created.
However,
> I
> > > > wanted to check and see what is out there first before I invested
> too
> > > > much time in it. If no one knows of anything else and\or is
> interested
> > > > in what I have described let me know and I will clean it up and
> > > > publish it.
> > > >
> > > > --Meador
> > > >
> > >
> >
>
There is rex, http://raa.ruby-lang.org/project/rex/ which takes a very
lex-like input and produces a ruby lexer that can be used with racc.
-Scott
···
On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
> Are you going to give this a try?
I think I just may. I will be sure to post to ruby-talk if I come up with
anything.