[ANN] rubylexer 0.7.3 Released

ruby-talk
1

Rubylexer version 0.7.3 has been released!

* <http://rubylexer.rubyforge.org/>
* <http://rubyforge.org/projects/rubylexer/>

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is meant
as a lexer for Ruby that's complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well. Just enough parsing
capability is included to give RubyLexer enough context to tokenize correctly
in all cases. (This turned out to be more parsing than I had thought or
wanted to take on at first.) RubyLexer handles the hard things like
complicated strings, the ambiguous nature of some punctuation characters and
keywords in ruby, and distinguishing methods and local variables.

Changes:

### 0.7.3/4-19-2009
* 9 Bugfixes:
  * remember whether comma was seen in paren context
  * reducing the warning load
  * remember whether we're in a method def when starting a recursive lexer
  * (so that class vars can be marked as being in a method if in #{})
  * make sure Token#to_s always outputs _something_ halfway sensible
  * make sure funclike keywords come out as a keyword
  * break/next/return method after . was not detected properly
  * need a NoWsToken before :: in names of compound modules
  * abort implicit paren contexts when elsif seen
  * all files should be world-readable now

* 9 Minor Enhancements:
  * move test data into lib/ dir so I can get to it better from redparse
  * split parse_keywords into separate methods for each keyword
  * (I had hoped to speed up keywords, but keyword_def is still slow)
  * more of the lists of keywords are now available as arrays as well
  * (parenthesis) KeywordToken needs to know if it's for grouping or call
  * minor speedups in newline and ident
  * don't panic if rubygems not available
  * make linenum publicly available in RubyCode
  * various other little helper methods needed by redparse in Tokens
  * hack Rakefile so 'rake test' will stay in 1 process (keeps netbeans happy)

* <http://rubylexer.rubyforge.org/>
* <http://rubyforge.org/projects/rubylexer/>

2

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is
meant as a lexer for Ruby that's complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well.

Would this allow one to create a "mini-irb", i.e. something that can
interprete ruby code like irb does, but without the need to reuse that
undercommented irb code?

···

--
Posted via http://www.ruby-forum.com/\.

3

So, irb has its own little lexer it uses internally, mostly so it can
tell whether you've typed a complete line of code or not. I presume
that's what you're referring to...? Irb's lexer could fairly easily be
replaced with RubyLexer, or you could write a whole new implementation
based on it.

I know irb's lexer is incomplete in various ways, but it seems to be
much better than it was
when I started writing RubyLexer. RubyLexer isn't totally perfect yet
either... it certainly isn't very readable... so now you've replaced
undercommented irb code with undercommented RubyLexer code... :wink:

Is this answering your question? I'm not sure.

···

On 5/2/09, Marc Heiler <shevegen@linuxmail.org> wrote:

RubyLexer is a lexer library for Ruby, written in Ruby. Rubylexer is
meant as a lexer for Ruby that's complete and correct; all legal Ruby
code should be lexed correctly by RubyLexer as well.

Would this allow one to create a "mini-irb", i.e. something that can
interprete ruby code like irb does, but without the need to reuse that
undercommented irb code?