Converting a string to an array of tokens

ruby-talk
1

This is fairly general and easily extended …

Identifiers = %{[a-zA-Z_][a-zA-Z0-9_]*}
Numbers = %{[0-9]+}
Punct = %w{ << >> == += -= = /= != [ ] }
Single = %{[-+/<>{}=]}

TokenList = []
TokenList << Identifiers << Numbers
TokenList += Punct.collect{|op| Regexp.quote(op)}
TokenList << Single
TokenRe = Regexp.new(TokenList.join(’|’))

def tokenize(str)
str.scan(TokenRe)
end

···

–
– Jim Weirich / Compuware
– FWP Capture Services
– Phone: 859-386-8855