# How to lex javascript for an assert\_js system?
**URL:** https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106
**Category:** ruby-talk
**Created:** [30 December 2006 06:25 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106 "2006-12-30T06:25:07Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Phlip2](https://rubytalk.org/u/Phlip2)
#### Post date: [30 December 2006 06:25 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106/1 "2006-12-30T06:25:07Z")
Ruboids:
Someone recently posted this:
> o There's a difference between syntax checking and verification of
> functional correctness
That is indeed why a test case that spot-checks your syntax is less useful
than a test case that understands your needs. All unit testing
starts at the former and aims at the latter. Here's an example of
testing Javascript's syntax:
ondblclick = div.attributes['ondblclick']
assert\_match /^new Ajax.Updater\("hammy\_id"/, ondblclick
It trivially asserts that a DIV (somewhere) contains an ondblclick
handler, and that this has a Script.aculo.us Ajax.Updater in it. The
assertion naturally cannot test that the Updater will indeed update a DIV.
To get closer to the problem, we might decide to get closer to the
Javascript. We may need a mock-Javascript system, to evaluate that string.
It could return the list of nuances commonly called a "Log String Test".
Here's an example, using a slightly more verbose language:
public void testPaintGraphicsintint() {
Mock mockGraphics = new Mock(Graphics.class);
mockGraphics.expects(once()).method("setColor").with(eq(Color.decode("0x6491EE")));
mockGraphics.expects(once()).method("setColor").with(same(Color.black));
mockGraphics.expects(once()).method("drawPolygon");
mockGraphics.expects(once()).method("drawPolygon");
hex.paint((Graphics) mockGraphics.proxy());
mockGraphics.verify();
}
From the top, that mocks your graphics display driver, and retains its
non-retained graphics commands. Then the mockGraphics object
verifies a certain series of calls, with such-and-so parameters.
(That is a Log String Test because it's the equivalent of writing commands
like "setColor" and "drawPolygon" into a log file, and then reading this
to assert things.)
That test case indeed fits the ideal of moving away from testing raw
syntax, and closer to testing semantics. Such a test, for example, could
more easily ignore extraneous calls, and then check that two dynamic
polygons did not overlap.
Now suppose I envision this testage:
def ondblclick(ypath)
%(new Ajax.Updater("node",
"/ctrl/act",
{ asynchronous:true,
evalScripts:true,
method:"get",
parameters:"i\_b\_a=Parameter" })
).gsub("\n", '').squeeze(' ')
end
def test\_some\_js
js = ondblclick()
parse = assert\_js(js)
statement = parse.first
assert\_equal 'new Ajax.Updater', statement.get\_method
assert\_equal '"node"', statement.get\_param(0)
assert\_equal '"/ctrl/act"', statement.get\_param(1)
json = statement.get\_param(2)
assert\_equal true, json['evalScripts']
end
The goal is the target JS can flex easily - can reorder its Json, or
change fuzzy details, or add new features - without breaking the tests.
Ideally, only changes that break project requirements will break tests.
Now suppose I want to write that assert\_js() using less than seven billion
lines of code.
The first shortcut is to only parse code we expect. I'm aware that's
generally against the general philosophy of parsing, but I'm trying to
sell an application, not a JS parser. That's a private detail. I can
accept, for example, only parsing the JS emitted by Rails's standard
gizmos.
So before getting down to some actual questions, here's the code my
exquisite parsing skills have thrashed out so far:
def test\_assert\_js
source = 'new Ajax.Updater('+
'"node", '+
'"/controller/action", '+
'{ asynchronous:true, '+
'evalScripts:true, '+
'method:"get", '+
'parameters:"i\_b\_a=Parameter" })'
js = assert\_js(source)
assert\_equal 'new Ajax.Updater', js.keys.first
parameters = js.values.first['()']
assert\_equal '"node"', parameters[0]
assert\_equal '"/controller/action"', parameters[1]
json = parameters[2]['{}']
assert\_equal 'true', json['evalScripts']
assert\_equal '"get"', json['method']
assert\_equal '"i\_b\_a=Parameter"', json['parameters']
end
Now that's good enough for government work, and I could probably upgrade
the interface to look more like my idealized example...
...but the implementation is a mish-mash of redundant
Regexps and run-on methods:
Qstr = /^(["](?:(?:\\["])|(?:[^\\"]+))\*?["]),?\s\*/
def assert\_json(source)
js = {}
identifier = /([[:alnum:]\_]+)馃槙
while m = source.match(identifier)
source = m.post\_match
n = source.match(/^([[:alnum:]\_]+),?\s\*/)
n = source.match(Qstr) unless n
break unless n
js[m.captures[0]] = n.captures[0]
source = n.post\_match
end
return { '{}' =\> js }
end
def assert\_js(source)
js = {}
qstr = /^(["](?:(?:\\["])|(?:[^\\"]+))\*?["]),?\s\*/
json = /^(\{.\*\}),?\s\*/
if source =~ /^([^\("]+)(.\*)$/
js[$1] = assert\_js($2)
elsif source =~ /^\((.\*)\)$/
js['()'] = assert\_js($1)
else
index = 0
while (m = source.match(qstr)) or
(m = source.match(json))
break if m.size \< 1
if source =~ /^\{/
js[index] = assert\_json(m.captures[0])
else
js[index] = m.captures[0]
end
source = m.post\_match
index += 1
end
end
return js
end
Now the questions. Is there some...
...way to severely beautify that implementation?
...lexing library I could \_easily\_ throw in?
...robust JS Lexer library already out there?
...assert\_js already out there?
> **路路路**
>
> --
> Phlip
> [http://c2.com/cgi/wiki?ZeekLand](http://c2.com/cgi/wiki?ZeekLand) \<-- NOT a blog!!
---
### Author:  [@Luke\_Graham](https://rubytalk.org/u/Luke_Graham)
#### Post date: [2 January 2007 12:03 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106/2 "2007-01-02T12:03:01Z")
Parsing with regexps makes baby Jesus cry. Javascript itself can be
quite flexible, so it may be possible to do enough with a standard
interpreter's run-time. Alternatively, have a look at the Mozilla
projects repository for a real interpreter you could hack on.
> **路路路**
>
> On 12/30/06, Phlip \ wrote:
>
> > Now the questions. Is there some...
> >
> > ...way to severely beautify that implementation?
> > ...lexing library I could \_easily\_ throw in?
> > ...robust JS Lexer library already out there?
> > ...assert\_js already out there?
---
### Author:  [@Ryan\_Platte](https://rubytalk.org/u/Ryan_Platte)
#### Post date: [4 January 2007 16:45 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106/3 "2007-01-04T16:45:10Z")
Phlip wrote:
> The goal is the target JS can flex easily - can reorder its Json, or
> change fuzzy details, or add new features - without breaking the tests.
> Ideally, only changes that break project requirements will break tests.
>
> Now suppose I want to write that assert\_js() using less than seven billion
> lines of code.
>
> The first shortcut is to only parse code we expect. I'm aware that's
> generally against the general philosophy of parsing, but I'm trying to
> sell an application, not a JS parser. That's a private detail. I can
> accept, for example, only parsing the JS emitted by Rails's standard
> gizmos.
...
> Now the questions. Is there some...
>
> ...way to severely beautify that implementation?
For Rails apps, by way of not testing the library('s JavaScript
output): stub out RJS's JavaScriptGenerator ('page' object)? That was
the first thing I thought when I heard about RJS. Surfing the code, it
looks quite possible using Mocha if it was desirable. I haven't thought
through all the ramifications. The OP includes testing the source page
-- maybe also some work on the APIs to link to and submit forms to Ajax
actions would permit those to be stubbed out as well.
Has someone already created such a beast? (Besides Google's GWT?)
> **路路路**
>
> --
> Ryan Platte
> Obtiva Training and Consulting
> Agile, Ruby, Rails, Java Eclipse RCP
> [http://obtiva.com/](http://obtiva.com/)
---
### Author:  [@Phlip1](https://rubytalk.org/u/Phlip1)
#### Post date: [2 January 2007 12:40 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106/4 "2007-01-02T12:40:10Z")
spooq wrote:
> Parsing with regexps makes baby Jesus cry. Javascript itself can be
> quite flexible, so it may be possible to do enough with a standard
> interpreter's run-time. Alternatively, have a look at the Mozilla
> projects repository for a real interpreter you could hack on.
This question is for an academic paper, so it has even more ridiculous
constraints on the amount of fun I can have. (And why hasn't the
industry invented a Lex in a Bottle, using Regexp-like strings and a
BNF notation?)
Can I add a parser to the Syntax library? It only does Ruby, XML, and
YAML so far...
And, yes, JavaScript was designed to be parsed, unlike some other
languages...
> **路路路**
>
> --
> Phlip
---
### Author:  [@Phlip5](https://rubytalk.org/u/Phlip5)
#### Post date: [5 January 2007 05:45 UTC](https://rubytalk.org/t/how-to-lex-javascript-for-an-assert-js-system/34106/5 "2007-01-05T05:45:12Z")
Ryan Platte wrote:
> For Rails apps, by way of not testing the library('s JavaScript
> output): stub out RJS's JavaScriptGenerator ('page' object)?
Ultimately we are up against a "log string test". That's where you call an
emulator or a real deal of some type, it emits a log of its behavior, and
you parse into this log.
You could, for example, take some server program, turn its log level up,
call a high level function, read the log file as a string, and perform
Regular Expressions on it to pull out target data. (/Error (.\*)/ is a good
start!馃槈
Next, you might write a mock that records the calls sent to it as a sequence
of data items, like the MockGraphics example I started this thread with.
Here's the diagnostic when an assert\_rjs fails:
Content did not include:
$("moose\_panel").width = "50%";.
\<"$(\"wiki\_panel\").width = \"50%\";\n$(\"mouse\_panel\").width =
\"50%\";\nElement.update(\"mouse\_panel\", \"\