Ripper on array access operation

ruby-talk
1

Ripper.sexp('a[1]')

yeilds the following output

[:program, [[:aref, [:var_ref, [:@ident, "a", [1, 0]]],
[:args_add_block, [[:@int, "1", [1, 2]]], false]]]]

i am trying to figure out where the information that the methods :[] was
called is present.

But when i do

Ripper.sexp('a.[](0)')

[:program, [[:method_add_arg, [:call, [:var_ref, [:@ident, "a", [1,
0]]], :".", [:@op, "[]", [1, 2]]], [:arg_paren, [:args_add_block,
[[:@int, "0", [1, 5]]], false]]]]]

I get the information about which method was called.

···

--
Posted via http://www.ruby-forum.com/.

2

Quoting "Surya G." <sgaddipati@obtiva.com>:

Ripper.sexp('a[1]')

yeilds the following output

[:program, [[:aref, [:var_ref, [:@ident, "a", [1, 0]]],
[:args_add_block, [[:@int, "1", [1, 2]]], false]]]]

i am trying to figure out where the information that the methods : was
called is present.

But when i do

Ripper.sexp('a.(0)')

[:program, [[:method_add_arg, [:call, [:var_ref, [:@ident, "a", [1,
0]]], :".", [:@op, "", [1, 2]]], [:arg_paren, [:args_add_block,
[[:@int, "0", [1, 5]]], false]]]]]

I get the information about which method was called.

--
Posted via http://www.ruby-forum.com/\.

I am pretty sure it's the :aref part. Ripper is making a special case.

-Justin

3

unknown wrote in post #1019486:

Quoting "Surya G." <sgaddipati@obtiva.com>:

called is present.
I get the information about which method was called.

--
Posted via http://www.ruby-forum.com/\.

I am pretty sure it's the :aref part. Ripper is making a special
case.

-Justin

Thats what it looks like but why not treat aref as any-other method
call.

···

--
Posted via http://www.ruby-forum.com/\.

4

Quoting "Surya G." <sgaddipati@obtiva.com>:

unknown wrote in post #1019486:

Quoting "Surya G." <sgaddipati@obtiva.com>:

called is present.
I get the information about which method was called.

--
Posted via http://www.ruby-forum.com/\.

I am pretty sure it's the :aref part. Ripper is making a special
case.

-Justin

Thats what it looks like but why not treat aref as any-other method
call.

--
Posted via http://www.ruby-forum.com/\.

I don't know. Ripper's output looks like a mess to me when compared to RubyParser:

s(:call, s(:call, nil, :a, s(:arglist)), :, s(:arglist, s(:lit, 1)))

-Justin

5

Quoting "Surya G." <sgaddipati@obtiva.com>:

unknown wrote in post #1019486:

Quoting "Surya G." <sgaddipati@obtiva.com>:

I am pretty sure it's the :aref part. Ripper is making a special
case.

-Justin

Thats what it looks like but why not treat aref as any-other method
call.

--
Posted via http://www.ruby-forum.com/\.

Aref is parsed differently from other method calls. Ripper delivers a parse tree, and by providing aref as a different AST node, a program consuming the parse tree can distinguish between a.(1, 2) and a[1,2]. There are plenty of reasons (such as pretty-printing) that these two should not be conflated.

I don't know. Ripper's output looks like a mess to me when compared to RubyParser:

s(:call, s(:call, nil, :a, s(:arglist)), :, s(:arglist, s(:lit, 1)))

-Justin

That's because RubyParser doesn't provide an abstract syntax tree, it provides a tree interpretation of Ruby programs. While Ripper comes close to a concrete syntax tree at times, at least it sticks to syntax. t discuss my choice of Ripper over RubyParser in my undergraduate thesis [0]; while it is perhaps suitable for an interpreter, its output discards so much of the syntax. It even inserts nodes which have nothing to do with syntax and everything to do with semantics. In this example, if I want my static analyzer to find examples of a.(1), I can't using RubyParser (at least with the default settings).

It adds :scope nodes which don't have anything to do with the syntax, it evaluates constant literals, from integers to regular expressions, which is not the job of a general-purpose AST. It even inserts a local assignment node when it parses "rescue Foo => x" (x = $!).

RubyParser is a good library, and its value as a pure-ruby parser cannot be understated. It's not suitable for many uses because of the shortcuts it takes, the over-interpretation it does (this aref example is a clear one), and the overall focus on semantics and not on just *parsing*.

[0] PDF: http://www.cs.dartmouth.edu/cms_file/SYS_techReport/532/TR2011-686.pdf

···

On Aug 31, 2011, at 7:00 PM, justincollins@ucla.edu wrote:

6

I suppose it depends on what you want out of it. For what I use it for, I don't care about the exact syntax used when they mean the same thing, and the fewer node types I need to check for, the better. And I appreciate that RubyParser takes care of much of the semantics for me. In your case, you may have needed different information.

So, different strokes for different folks.

-Justin

···

On 08/31/2011 08:41 PM, Michael Edgar wrote:

On Aug 31, 2011, at 7:00 PM, justincollins@ucla.edu wrote:

Quoting "Surya G."<sgaddipati@obtiva.com>:

unknown wrote in post #1019486:

Quoting "Surya G."<sgaddipati@obtiva.com>:

I am pretty sure it's the :aref part. Ripper is making a special
case.

-Justin

Thats what it looks like but why not treat aref as any-other method
call.

--
Posted via http://www.ruby-forum.com/\.

Aref is parsed differently from other method calls. Ripper delivers a parse tree, and by providing aref as a different AST node, a program consuming the parse tree can distinguish between a.(1, 2) and a[1,2]. There are plenty of reasons (such as pretty-printing) that these two should not be conflated.

I don't know. Ripper's output looks like a mess to me when compared to RubyParser:

s(:call, s(:call, nil, :a, s(:arglist)), :, s(:arglist, s(:lit, 1)))

-Justin

That's because RubyParser doesn't provide an abstract syntax tree, it provides a tree interpretation of Ruby programs. While Ripper comes close to a concrete syntax tree at times, at least it sticks to syntax. t discuss my choice of Ripper over RubyParser in my undergraduate thesis [0]; while it is perhaps suitable for an interpreter, its output discards so much of the syntax. It even inserts nodes which have nothing to do with syntax and everything to do with semantics. In this example, if I want my static analyzer to find examples of a.(1), I can't using RubyParser (at least with the default settings).

It adds :scope nodes which don't have anything to do with the syntax, it evaluates constant literals, from integers to regular expressions, which is not the job of a general-purpose AST. It even inserts a local assignment node when it parses "rescue Foo => x" (x = $!).

RubyParser is a good library, and its value as a pure-ruby parser cannot be understated. It's not suitable for many uses because of the shortcuts it takes, the over-interpretation it does (this aref example is a clear one), and the overall focus on semantics and not on just *parsing*.

[0] PDF: http://www.cs.dartmouth.edu/cms_file/SYS_techReport/532/TR2011-686.pdf

7

I suspect this is actually only true at the margins, which is why parse trees are valuable and why I argue the corners RubyParser cuts are arbitrary and silly.

Would you prefer that RubyParser turn "@foo = bar" into a call to "instance_variable_set"?

Or "class A < B; end" into a series of conditions, constant lookups, assignments, "Class.new(B)", and so on?

Should "class << foo; end" turn into "foo.singleton_class.class_eval do.. end"?

Michael Edgar
adgar@carboni.ca
http://carboni.ca/

···

On Sep 1, 2011, at 9:37 PM, Justin Collins wrote:

For what I use it for, I don't care about the exact syntax used when they mean the same thing, and the fewer node types I need to check for, the better.

8

For what I use it for, I don't care about the exact syntax used when they mean the same thing, and the fewer node types I need to check for, the better.

I suspect this is actually only true at the margins, which is why parse trees are valuable and why I argue the corners RubyParser cuts are arbitrary and silly.

Would you prefer that RubyParser turn "@foo = bar" into a call to "instance_variable_set"?

Kind of... :slight_smile: But "instance_variable_set" might have a different meaning than "@foo = bar". Whereas, as far as I know, "a" and "a.()" will always be the same.

Or "class A< B; end" into a series of conditions, constant lookups, assignments, "Class.new(B)", and so on?

I think this might be going in the other direction (but you may also be correct). In "a" vs. "a.()", RubyParser makes it easier because I only have to look for a call to "". How the call is represented in the actual code is not important (to me!). If it were always valid to assume that the above-mentioned conditions, lookups, etc., would mean the same as "class A < B; end" then I (in my case!) would not mind if I just got back something representing "class A < B; end". In other words, the reverse of what you are suggesting.

Should "class<< foo; end" turn into "foo.singleton_class.class_eval do.. end"?

Michael Edgar
adgar@carboni.ca
http://carboni.ca/

Will they always mean exactly the same thing? There are very few cases I can think of where two bits of Ruby code will _always_ be the same thing, because you can override nearly anything.

Here's another case:

ruby-1.9.2-p290 :003 > RubyParser.new.parse "!a"
  => s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :004 > RubyParser.new.parse "not a"
  => s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :005 > Ripper.sexp "!a"
  => [:program, [[:unary, :!, [:var_ref, [:@ident, "a", [1, 1]]]]]]
ruby-1.9.2-p290 :006 > Ripper.sexp "not a"
  => [:program, [[:unary, :not, [:var_ref, [:@ident, "a", [1, 4]]]]]]

Again, if I want to check for a negative of "a", then with Ripper I must look for (at least) two alternatives, but with RubyParser there is just one. Semantically there is no difference. (There is a difference in the interpretation because RubyParser doesn't know about Ruby 1.9, but that is a different difference. What I mean is there is no difference between Ripper's "!" and "not", because they will do the same thing.)

Probably you can come up with some instances where RubyParser changes the semantics of the original code it is representing, which would be a flaw in RubyParser. It may or may not matter for my use case, though.

Whew. I'm only responding because you asked such direct questions :slight_smile: As I've said, RubyParser works better for me, while Ripper is a bit more difficult to manage. Clearly you needed more information about the program, so Ripper made sense to use. As you have pointed out before, Ripper has its own issues. There are use cases for either one depending on what your needs are. If Ripper were available for 1.8, I might have used it instead.

-Justin

···

On 09/01/2011 06:57 PM, Michael Edgar wrote:

On Sep 1, 2011, at 9:37 PM, Justin Collins wrote:

9

For what I use it for, I don't care about the exact syntax used when they

mean the same thing, and the fewer node types I need to check for, the
better.

I suspect this is actually only true at the margins, which is why parse
trees are valuable and why I argue the corners RubyParser cuts are arbitrary
and silly.

Would you prefer that RubyParser turn "@foo = bar" into a call to
"instance_variable_set"?

Kind of... :slight_smile: But "instance_variable_set" might have a different meaning
than "@foo = bar". Whereas, as far as I know, "a" and "a.()" will always
be the same.

Or "class A< B; end" into a series of conditions, constant lookups,

assignments, "Class.new(B)", and so on?

I think this might be going in the other direction (but you may also be
correct). In "a" vs. "a.()", RubyParser makes it easier because I only
have to look for a call to "". How the call is represented in the actual
code is not important (to me!). If it were always valid to assume that the
above-mentioned conditions, lookups, etc., would mean the same as "class A <
B; end" then I (in my case!) would not mind if I just got back something
representing "class A < B; end". In other words, the reverse of what you are
suggesting.

Should "class<< foo; end" turn into "foo.singleton_class.class_**eval

do.. end"?

Michael Edgar
adgar@carboni.ca
http://carboni.ca/

Will they always mean exactly the same thing? There are very few cases I
can think of where two bits of Ruby code will _always_ be the same thing,
because you can override nearly anything.

Here's another case:

ruby-1.9.2-p290 :003 > RubyParser.new.parse "!a"
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :004 > RubyParser.new.parse "not a"
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :005 > Ripper.sexp "!a"
=> [:program, [[:unary, :!, [:var_ref, [:@ident, "a", [1, 1]]]]]]
ruby-1.9.2-p290 :006 > Ripper.sexp "not a"
=> [:program, [[:unary, :not, [:var_ref, [:@ident, "a", [1, 4]]]]]]

Again, if I want to check for a negative of "a", then with Ripper I must
look for (at least) two alternatives, but with RubyParser there is just one.
Semantically there is no difference. (There is a difference in the
interpretation because RubyParser doesn't know about Ruby 1.9, but that is a
different difference. What I mean is there is no difference between Ripper's
"!" and "not", because they will do the same thing.)

They aren't quite the same, they bind differently:

!false && false # => false
not false && false # => true

Probably you can come up with some instances where RubyParser changes the
semantics of the original code it is representing, which would be a flaw in
RubyParser. It may or may not matter for my use case, though.

I think this is the ultimate point here. You have different needs, and
different libs to address those different needs.

···

On Fri, Sep 2, 2011 at 12:54 AM, Justin Collins <justincollins@ucla.edu>wrote:

On 09/01/2011 06:57 PM, Michael Edgar wrote:

On Sep 1, 2011, at 9:37 PM, Justin Collins wrote:

10

Quoting Josh Cheek <josh.cheek@gmail.com>:

For what I use it for, I don't care about the exact syntax used when they

mean the same thing, and the fewer node types I need to check for, the
better.

I suspect this is actually only true at the margins, which is why parse
trees are valuable and why I argue the corners RubyParser cuts are arbitrary
and silly.

Would you prefer that RubyParser turn "@foo = bar" into a call to
"instance_variable_set"?

Kind of... :slight_smile: But "instance_variable_set" might have a different meaning
than "@foo = bar". Whereas, as far as I know, "a" and "a.()" will always
be the same.

Or "class A< B; end" into a series of conditions, constant lookups,

assignments, "Class.new(B)", and so on?

I think this might be going in the other direction (but you may also be
correct). In "a" vs. "a.()", RubyParser makes it easier because I only
have to look for a call to "". How the call is represented in the actual
code is not important (to me!). If it were always valid to assume that the
above-mentioned conditions, lookups, etc., would mean the same as "class A <
B; end" then I (in my case!) would not mind if I just got back something
representing "class A < B; end". In other words, the reverse of what you are
suggesting.

Should "class<< foo; end" turn into "foo.singleton_class.class_**eval

do.. end"?

Michael Edgar
adgar@carboni.ca
http://carboni.ca/

Will they always mean exactly the same thing? There are very few cases I
can think of where two bits of Ruby code will _always_ be the same thing,
because you can override nearly anything.

Here's another case:

ruby-1.9.2-p290 :003 > RubyParser.new.parse "!a"
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :004 > RubyParser.new.parse "not a"
=> s(:not, s(:call, nil, :a, s(:arglist)))
ruby-1.9.2-p290 :005 > Ripper.sexp "!a"
=> [:program, [[:unary, :!, [:var_ref, [:@ident, "a", [1, 1]]]]]]
ruby-1.9.2-p290 :006 > Ripper.sexp "not a"
=> [:program, [[:unary, :not, [:var_ref, [:@ident, "a", [1, 4]]]]]]

Again, if I want to check for a negative of "a", then with Ripper I must
look for (at least) two alternatives, but with RubyParser there is just one.
Semantically there is no difference. (There is a difference in the
interpretation because RubyParser doesn't know about Ruby 1.9, but that is a
different difference. What I mean is there is no difference between Ripper's
"!" and "not", because they will do the same thing.)

They aren't quite the same, they bind differently:

!false && false # => false
not false && false # => true

Yes, but precedence is handled by the parser, so that difference doesn't matter once you have the s-expressions:

ruby-1.9.2-p290 :005 > RubyParser.new.parse "!false && false"
  => s(:and, s(:not, s(:false)), s(:false))
ruby-1.9.2-p290 :006 > RubyParser.new.parse "not false && false"
  => s(:not, s(:and, s(:false), s(:false)))
ruby-1.9.2-p290 :007 > Ripper.sexp "!false && false"
  => [:program, [[:binary, [:unary, :!, [:var_ref, [:@kw, "false", [1, 1]]]], :"&&", [:var_ref, [:@kw, "false", [1, 10]]]]]]
ruby-1.9.2-p290 :008 > Ripper.sexp "not false && false"
  => [:program, [[:unary, :not, [:binary, [:var_ref, [:@kw, "false", [1, 4]]], :"&&", [:var_ref, [:@kw, "false", [1, 13]]]]]]]

Probably you can come up with some instances where RubyParser changes the
semantics of the original code it is representing, which would be a flaw in
RubyParser. It may or may not matter for my use case, though.

I think this is the ultimate point here. You have different needs, and
different libs to address those different needs.

Yes.

-Justin

···

On Fri, Sep 2, 2011 at 12:54 AM, Justin Collins > <justincollins@ucla.edu>wrote:

On 09/01/2011 06:57 PM, Michael Edgar wrote:

On Sep 1, 2011, at 9:37 PM, Justin Collins wrote: