Match a pattern multiple times, returning matches, captures and offset?

ruby-talk
1

Hi,

I'm used to be able to use the following in PHP. What is basically does
is: return me all matches, including the captures, order by matching set
and provide me the offsets.

$ php -r 'preg_match_all("/_(\w+)_/", "_foo_ _bar_", $matches,
PREG_SET_ORDER|PREG_OFFSET_CAPTURE); var_dump($matches);'
array(2) {
  [0]=>
  array(2) {
    [0]=>
    array(2) {
      [0]=>
      string(5) "_foo_"
      [1]=>
      int(0)
    }
    [1]=>
    array(2) {
      [0]=>
      string(3) "foo"
      [1]=>
      int(1)
    }
  }
  [1]=>
  array(2) {
    [0]=>
    array(2) {
      [0]=>
      string(5) "_bar_"
      [1]=>
      int(6)
    }
    [1]=>
    array(2) {
      [0]=>
      string(3) "bar"
      [1]=>
      int(7)
    }
  }
}

I've found two ways in ruby getting in this direction, either use
String#match or String#scan, but both only provide me partial
information. I guess I can combine the knowledge of both, but before
attempting this I wanted to verify if I didn't overlook something. Here
are my ruby attempts:

ruby-1.9.2-p180 :001 > m = "_foo_ _bar_".match(/_(\w+)_/)
=> #<MatchData "_foo_" 1:"foo">
ruby-1.9.2-p180 :002 > [ m[0], m[1] ]
=> ["_foo_", "foo"]
ruby-1.9.2-p180 :003 > [ m.begin(0), m.begin(1) ]
=> [0, 1]

But here I'm missing the further possible matches, "_bar_" and "bar". Or
the #scan approach:

ruby-1.9.2-p180 :004 > m = "_foo_ _bar_".scan(/_(\w+)_/)
=> [["foo"], ["bar"]]

But in this case I've even less information, the match including _foo_
or _bar_ is not present and I can't get the offsets too.

I re-read the documentation for Regexp#match and found out that you can
pass an offset into the string as second parameter, so I guess I can
iterate over the string in a loop until I find no further matches ...?
Considering this I came up with:

$ cat test_match_all.rb
require 'pp'

class String
    def match_all(pattern)
        matches = []
        offset = 0
        while m = match(pattern, offset) do
            matches << m
            offset = m.begin(0) + m[0].length
        end
        matches
    end
end

pp "_foo_ _bar_ _baz_".match_all(/_(\w+)_/)

$ ruby test_match_all.rb
[#<MatchData "_foo_" 1:"foo">,
#<MatchData "_bar_" 1:"bar">,
#<MatchData "_baz_" 1:"baz">]

I've lots of data to parse so I could foresee that this approach can
become a bottleneck. Is there a more direct solution to it?

thanks,
- Markus

2

String#scan with a block may do what you want:

"_foo_ _bar_".scan(/_(\w+)_/) { |x| puts "Offset #{$`.size}, captures

#{x.inspect}" }
Offset 0, captures ["foo"]
Offset 6, captures ["bar"]
=> "_foo_ _bar_"

But it doesn't give you offsets to the individual captures, just to the
start of the whole match. (You also get the full match in $& and the
rest of the string after the match in $')

···

--
Posted via http://www.ruby-forum.com/\.

3

Markus Fischer wrote in post #991092:

But here I'm missing the further possible matches, "_bar_" and "bar". Or
the #scan approach:

ruby-1.9.2-p180 :004 > m = "_foo_ _bar_".scan(/_(\w+)_/)
=> [["foo"], ["bar"]]

But in this case I've even less information, the match including _foo_
or _bar_ is not present and I can't get the offsets too.

I re-read the documentation for Regexp#match

If you look at the preamble in the docs for the MatchData class, you can
retrieve a MatchData object using Regexp.last_match, which you can call
inside a scan() block:

str = "_foo_ _bar_"

str.scan(/_(\w+)_/) do |match|
  md = Regexp.last_match
  p [md[0], md[1], md.offset(1)]

end

--output:--
["_foo_", "foo", [1, 4]]
["_bar_", "bar", [7, 10]]

···

--
Posted via http://www.ruby-forum.com/\.

4

You can also get the relative offset like this:

str = "_foo_ _bar_"

str.scan(/_(\w+)_/) do |curr_match|
  md = Regexp.last_match
  whole_match = md[0]
  captures = md.captures
  captures.each do |capture|
    p [whole_match, capture, whole_match.index(capture)]
end

···

--
Posted via http://www.ruby-forum.com/.

5

When doing nested matching it may be better to use $~ because that is
local to the current stack frame which Regexp.last_match isn't.
Example with relative offsets as well:

irb(main):022:0> str.scan /_(\w+)_/ do
irb(main):023:1* 2.times {|i| p [$~[i], $~.offset(i), $~.offset(i).map
{|o| o - $~.offset(0)[0]}]}
irb(main):024:1> end
["_foo_", [0, 5], [0, 5]]
["foo", [1, 4], [1, 4]]
["_bar_", [6, 11], [0, 5]]
["bar", [7, 10], [1, 4]]
=> "_foo_ _bar_"

Kind regards

robert

···

On Wed, Apr 6, 2011 at 3:37 AM, 7stud -- <bbxx789_05ss@yahoo.com> wrote:

Markus Fischer wrote in post #991092:

But here I'm missing the further possible matches, "_bar_" and "bar". Or
the #scan approach:

ruby-1.9.2-p180 :004 > m = "_foo_ _bar_".scan(/_(\w+)_/)
=> [["foo"], ["bar"]]

But in this case I've even less information, the match including _foo_
or _bar_ is not present and I can't get the offsets too.

I re-read the documentation for Regexp#match

If you look at the preamble in the docs for the MatchData class, you can
retrieve a MatchData object using Regexp.last_match, which you can call
inside a scan() block:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

6

That's nice! I wasn't aware of this. Thanks for sharing!

I also just read this in the docs:

"Note that the last_match is local to the thread and method scope of the method
that did the pattern match."

So forget my point about $~ being safer.

Kind regards

robert

···

On Thu, Apr 7, 2011 at 1:58 AM, 7stud -- <bbxx789_05ss@yahoo.com> wrote:

You can also get the relative offset like this:

str = "_foo_ _bar_"

str.scan(/_(\w+)_/) do |curr_match|
md = Regexp.last_match
whole_match = md[0]
captures = md.captures
captures.each do |capture|
p [whole_match, capture, whole_match.index(capture)]
end

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

7

7stud -- wrote in post #991338:

You can also get relative beginning offsets like this:

str = "_foo_ _bar_"

str.scan(/_(\w+)_/) do |curr_match|
  md = Regexp.last_match
  whole_match = md[0]
  captures = md.captures

  captures.each do |capture|
    p [whole_match, capture, whole_match.index(capture)]
  end

end

Using 'index' doesn't work if you have multiple captures which have the
same pattern, or one is a substring of the other.

Use captures.begin and captures.end instead.

md = /(...)(...)/.match "foofoo"

=> #<MatchData "foofoo" 1:"foo" 2:"foo">

md.captures

=> ["foo", "foo"]

md.begin(1)

=> 0

md.begin(2)

=> 3

···

--
Posted via http://www.ruby-forum.com/\.

8

Brian Candler wrote in post #991406:

7stud -- wrote in post #991338:

You can also get relative beginning offsets like this:

str = "_foo_ _bar_"

str.scan(/_(\w+)_/) do |curr_match|
  md = Regexp.last_match
  whole_match = md[0]
  captures = md.captures

  captures.each do |capture|
    p [whole_match, capture, whole_match.index(capture)]
  end

end

Using 'index' doesn't work if you have multiple captures which have the
same pattern, or one is a substring of the other.

Use captures.begin and captures.end instead.

begin() and end() are the two elements of offset(), which we've already
discussed above:

The idea was to get the relative offsets within a match, not the
absolute offsets within the string.

···

--
Posted via http://www.ruby-forum.com/\.

9

7stud -- wrote in post #991546:

However, note that
begin() and end() are the two elements of offset(), which we've already
discussed above. The idea was to additionally provide the relative
offsets within a match, not just the absolute offsets within the string.

That's easy - subtract begin(0) which is the absolute offset of the
start of the match.

"foo bar" =~ /ba(.)/

=> 4

$~.captures

=> ["r"]

$~.begin(1)

=> 6

$~.begin(1) - $~.begin(0)

=> 2

···

--
Posted via http://www.ruby-forum.com/\.

10

Brian Candler wrote in post #991686:

7stud -- wrote in post #991546:

However, note that
begin() and end() are the two elements of offset(), which we've already
discussed above. The idea was to additionally provide the relative
offsets within a match, not just the absolute offsets within the string.

That's easy - subtract begin(0) which is the absolute offset of the
start of the match.

The "subtraction method" was thoroughly vetted earlier.

···

--
Posted via http://www.ruby-forum.com/\.