SciTE/VIM Syntax highlighting

ruby-talk
1

I am learning Ruby via “Programming Ruby: A Pragmatic Programmer’s Guide”.
In one example, a regex is made with an apostrophe as part of the pattern.

aPhrase.scan /\w[-\w’]+/

The syntax highlighting in SciTE and Vim then interprets all code following
the apostrophe to be part of a single quoted string (making it pink). Is
there a way to make it “ignore” the single quote (besides switching to plain
vi)?? I tried escaping it (’) but that didn’t work. Thanks.

Jim.

2

Use one of the following alternatives:

aPhrase.scan %r/\w[-\w']+/
aPhrase.scan(/\w[-\w']+/)

Either of them should make syntax highlighting work with Vim (don’t
know about SciTE). The general problem is that it is non-trivial to
distinguish between / as the first character of a regular expression
and / as the division operator simply from local context. Therefore, a
heuristic is needed. And obviously, any heuristic will sometimes be
wrong.

[…]

		Reimer Behrends
···

Jim Bartlett (jimbart@magma.ca) wrote:

I am learning Ruby via “Programming Ruby: A Pragmatic Programmer’s Guide”.
In one example, a regex is made with an apostrophe as part of the pattern.

aPhrase.scan /\w[-\w’]+/

3

“Jim Bartlett” jimbart@magma.ca writes:

I am learning Ruby via “Programming Ruby: A Pragmatic Programmer’s Guide”.
In one example, a regex is made with an apostrophe as part of the pattern.

aPhrase.scan /\w[-\w’]+/

The syntax highlighting in SciTE and Vim then interprets all code following
the apostrophe to be part of a single quoted string (making it pink). Is
there a way to make it “ignore” the single quote (besides switching to plain
vi)?? I tried escaping it (’) but that didn’t work. Thanks.

Similar things happen in Emacs, too. Sometimes I have to include a comment
on that line or the next line just to un-confuse the fontifier.

aPhrase.scan /\w[-\w']+/		# ' <= un-confuse Emacs fontify code

Jim

···

–
Jim Menard, jimm@io.com, http://www.io.com/~jimm/
“I cannot articulate enough to express my dislike to people who think that
understanding spoils your experience… How would they know?”
– Marvin Minsky

4

Jim Bartlett:

aPhrase.scan /\w[-\w’]+/

The syntax highlighting in SciTE and Vim then interprets all code
following
the apostrophe to be part of a single quoted string (making it pink).

SciTE’s Ruby lexer does not understand regular expressions so there is no
way short of fixing the lexer to make this work.

Neil

5

In article slrnanj25f.gtd.behrends@ellington.cse.msu.edu, "Reimer

···

Behrends" behrends@cse.msu.edu wrote:

Use one of the following alternatives:

aPhrase.scan %r/\w[-\w’]+/
aPhrase.scan(/\w[-\w’]+/)

Either of them should make syntax highlighting work with Vim (don’t know
about SciTE). The general problem is that it is non-trivial to
distinguish between / as the first character of a regular expression and
/ as the division operator simply from local context. Therefore, a
heuristic is needed. And obviously, any heuristic will sometimes be
wrong.

[…]

  	Reimer Behrends

That worked like a charm. Thanks for your help.

Jim.

6

Thanks. I don’t use Emacs as much as SciTE or VIM (for me, Emacs is
overkill), but that is good to know. I appreciate the help.

Jim.

···

In article wsq1y86bbbt.fsf@io.com, “Jim Menard” jimm@io.com wrote:

Similar things happen in Emacs, too. Sometimes I have to include a
comment on that line or the next line just to un-confuse the fontifier.

aPhrase.scan /\w[-\w']+/		# ' <= un-confuse Emacs fontify code

Jim

7

I just spent the last six weeks trying to write an editor using the Ruby lexer
from irb to aid in syntax highlighting. So at the moment, I am of the
opinion that a proper lexer is probably not fast enough to be anything more
than an aggravation in syntax highlighting :slight_smile:

···

On Saturday 07 September 2002 04:41 am, Neil Hodgson wrote:

Jim Bartlett:

aPhrase.scan /\w[-\w’]+/

The syntax highlighting in SciTE and Vim then interprets all code

following

the apostrophe to be part of a single quoted string (making it pink).

SciTE’s Ruby lexer does not understand regular expressions so there is
no way short of fixing the lexer to make this work.

Neil

8

Albert Wagner:

I just spent the last six weeks trying to write an editor using the Ruby
lexer
from irb to aid in syntax highlighting. So at the moment, I am of the
opinion that a proper lexer is probably not fast enough to be anything
more
than an aggravation in syntax highlighting :slight_smile:

Lexing generally requires very little state compared to parsing and so
can be done well incrementally. For an editor, most modifications only
change the contents of one line and you often only need to relex that line.
Longer lexemes like multi-line comments may require more effort but only
when the modification creates a comment start or end or removes a comment
start or end.

Neil

9

Do you happen to know any good (into-level) web resources on lexing? The
current vim indenting rules for Ruby would really benefit from finding the
beginning of the current statement (which may be before the current line, of
course). I’m not eager to hack a solution … until I know how it should have
been done properly :slight_smile:

Gavin

···

----- Original Message -----
From: “Neil Hodgson” nhodgson@bigpond.net.au

Albert Wagner:

I just spent the last six weeks trying to write an editor using the Ruby
lexer
from irb to aid in syntax highlighting. So at the moment, I am of the
opinion that a proper lexer is probably not fast enough to be anything
more
than an aggravation in syntax highlighting :slight_smile:

Lexing generally requires very little state compared to parsing and so
can be done well incrementally. For an editor, most modifications only
change the contents of one line and you often only need to relex that line.
Longer lexemes like multi-line comments may require more effort but only
when the modification creates a comment start or end or removes a comment
start or end.

Neil

10

Gavin Sinclair:

Do you happen to know any good (into-level) web resources on lexing?

Not really anything introductory. I learned at university where the text
was the classic Aho and Ullman “Dragon” book “Principles of Compiler
Design”. For medium level interest (a sizeable investment in learning for a
good chunk of capability) I’d recommend downloading Terence Parr’s ANTLR and
experimenting with it.
http://www.antlr.org/
There is an incomplete book with an interesting chapter “Building
Translators By Hand”.
http://www.antlr.org/book/index.html
Dr. Parr also wrote an earlier book that is available online:
http://www.antlr.org/papers/pcctsbk.pdf
ANTLR includes functionality for writing lexers, parsers and tree
parsers.

The
current vim indenting rules for Ruby would really benefit from finding the
beginning of the current statement (which may be before the current line,
of
course).

Statement recognition requires parse level knowledge rather than just
lexical knowledge. Lexers take an input stream of characters and break it up
into tokens such as strings, identifiers, numbers and whitespace. A parser
can take this token stream and recognise the structure of the code such as
statements, blocks and declarations.

Neil