Regexp gotcha

ruby-talk
1

Hi, all. I was fixing a bug last night, and discovered some
"gotcha"-like behaviour in the process. Consider:

irb(main):173:0> s = "my string"
=> "my string"
irb(main):174:0> r1 = /my/
=> /my/
irb(main):175:0> r2 = /your/
=> /your/
irb(main):176:0> r3 = nil
=> nil
irb(main):177:0> s =~ r1
=> 0
irb(main):178:0> s =~ r2
=> nil
irb(main):179:0> s =~ r3
=> false

s =~ r1 .... That's cool, it gives me the index of the match.
s =~ r2 .... That's cool, it tells me there was no match.
s =~ r3 .... Whoa.

The reason this "got me" is that I had this code:

match_result = ( some_string =~ some_regexp )
if match_result != nil
    # Assume there was a match
end

But the problem is... I had an s =~ r3 case because some_regexp was nil,
and so it was entering my if block when I semantically did not want that
to occur. :frowning:

So now my code is

if match_result != nil and match_result != false
...
end

Note also that I can't even use Regexp.last_match != nil in my if block:

irb(main):224:0> s =~ r2
=> nil
irb(main):225:0> Regexp.last_match
=> nil
irb(main):226:0> s =~ r3
=> false
irb(main):227:0> Regexp.last_match
=> nil
irb(main):228:0> s =~ r1
=> 0
irb(main):229:0> Regexp.last_match
=> #<MatchData:0x406c40b4>
irb(main):230:0> s =~ r3
=> false
irb(main):231:0> Regexp.last_match
=> #<MatchData:0x406c40b4>

To be clear: Note how a "nil type" of non-match overwrites last_match,
but a "false type" of non-match doesn't.

So the question is... why are BOTH nil and false possible return values
of =~ ? Is there some benefit to this? Why not just one or the other?

I see that this behaviour is [documented][1] but I still feel that this
is unintuitive behaviour when people assume =~ only applies to Regexp
RHS's.

[1]: http://www.ruby-doc.org/core/classes/String.html#M001453

Thanks in advance for any and all clarifications and explanations.

Pistos

···

--
Posted via http://www.ruby-forum.com/.

2

So now my code is

if match_result != nil and match_result != false
...
end

AFAIK, it's an equivalent for simple

if match_result
...
end

because in Ruby only nil and false are "false", where any other value
(including 0, '' and ) are "true".

Victor.

3

Hi --

···

On Wed, 29 Mar 2006, Pistos Christou wrote:

Hi, all. I was fixing a bug last night, and discovered some
"gotcha"-like behaviour in the process. Consider:

irb(main):173:0> s = "my string"
=> "my string"
irb(main):174:0> r1 = /my/
=> /my/
irb(main):175:0> r2 = /your/
=> /your/
irb(main):176:0> r3 = nil
=> nil
irb(main):177:0> s =~ r1
=> 0
irb(main):178:0> s =~ r2
=> nil
irb(main):179:0> s =~ r3
=> false

s =~ r1 .... That's cool, it gives me the index of the match.
s =~ r2 .... That's cool, it tells me there was no match.
s =~ r3 .... Whoa.

The reason this "got me" is that I had this code:

match_result = ( some_string =~ some_regexp )
if match_result != nil
   # Assume there was a match
end

But the problem is... I had an s =~ r3 case because some_regexp was nil,
and so it was entering my if block when I semantically did not want that
to occur. :frowning:

So now my code is

if match_result != nil and match_result != false

You could shorten that to:

   if match_result

I'm not sure about the "why" part of the nil/false thing.

David

--
David A. Black (dblack@wobblini.net)
Ruby Power and Light, LLC (http://www.rubypowerandlight.com)

"Ruby for Rails" chapters now available
from Manning Early Access Program! Ruby for Rails

4

You could just do this...

if string =~ /(\w)/
  #do something with $1
end

anytime you try to match a string to a non-regexp object you get a
false, I think.

_Kevin

···

On Wednesday, March 29, 2006, at 12:24 AM, Pistos Christou wrote:

Hi, all. I was fixing a bug last night, and discovered some
"gotcha"-like behaviour in the process. Consider:

irb(main):173:0> s = "my string"
=> "my string"
irb(main):174:0> r1 = /my/
=> /my/
irb(main):175:0> r2 = /your/
=> /your/
irb(main):176:0> r3 = nil
=> nil
irb(main):177:0> s =~ r1
=> 0
irb(main):178:0> s =~ r2
=> nil
irb(main):179:0> s =~ r3
=> false

s =~ r1 .... That's cool, it gives me the index of the match.
s =~ r2 .... That's cool, it tells me there was no match.
s =~ r3 .... Whoa.

The reason this "got me" is that I had this code:

match_result = ( some_string =~ some_regexp )
if match_result != nil
   # Assume there was a match
end

But the problem is... I had an s =~ r3 case because some_regexp was nil,
and so it was entering my if block when I semantically did not want that
to occur. :frowning:

So now my code is

if match_result != nil and match_result != false
...
end

Note also that I can't even use Regexp.last_match != nil in my if block:

irb(main):224:0> s =~ r2
=> nil
irb(main):225:0> Regexp.last_match
=> nil
irb(main):226:0> s =~ r3
=> false
irb(main):227:0> Regexp.last_match
=> nil
irb(main):228:0> s =~ r1
=> 0
irb(main):229:0> Regexp.last_match
=> #<MatchData:0x406c40b4>
irb(main):230:0> s =~ r3
=> false
irb(main):231:0> Regexp.last_match
=> #<MatchData:0x406c40b4>

To be clear: Note how a "nil type" of non-match overwrites last_match,
but a "false type" of non-match doesn't.

So the question is... why are BOTH nil and false possible return values
of =~ ? Is there some benefit to this? Why not just one or the other?

I see that this behaviour is [documented][1] but I still feel that this
is unintuitive behaviour when people assume =~ only applies to Regexp
RHS's.

[1]: http://www.ruby-doc.org/core/classes/String.html#M001453

Thanks in advance for any and all clarifications and explanations.

Pistos

--
Posted via http://www.ruby-forum.com/\.

--
Posted with http://DevLists.com. Sign up and save your time!

5

Victor Shepelev wrote:

if match_result
...
end

because in Ruby only nil and false are "false", where any other value
(including 0, '' and ) are "true".

Yep, thank you to you and David. I forgot that I could rewrite it like
that.

Kevin wrote:

You could just do this...
if string =~ /(\w)/
  #do something with $1
end

Well, in this particular case, I am using the Fixnum returned, which is
why I am making the assignment. I normally otherwise do as you say,
using "if string =~ /regexp/".

Pistos

···

--
Posted via http://www.ruby-forum.com/\.

6

Are you asking why we can write

  if match_result

as equivalent to

  if match_result != nil and match_result != false

?

James H.

7

Pistos Christou wrote:

Victor Shepelev wrote:

if match_result
...
end

because in Ruby only nil and false are "false", where any other value
(including 0, '' and ) are "true".

Yep, thank you to you and David. I forgot that I could rewrite it like that.

Kevin wrote:

You could just do this...
if string =~ /(\w)/
  #do something with $1
end

Well, in this particular case, I am using the Fixnum returned, which is why I am making the assignment. I normally otherwise do as you say, using "if string =~ /regexp/".

Personally I prefer to use /rx/ =~ str over str =~ /rx/ - to me this makes it clearer that the RX is the one that does the matching. Just personal taste maybe but I think I also remember that that variant is a tad faster.

Kind regards

  robert

8

James H. wrote:

Are you asking why we can write

  if match_result

as equivalent to

  if match_result != nil and match_result != false

?

No, not at all. :slight_smile: I'm just whining a bit that =~ can return both nil
and false. It's not that big a deal, but this is something that could
catch unaware people who would follow down the same tracks as I did and
assume match_result != nil would cover all the bases, when it doesn't.

Robert Klemme wrote:

Personally I prefer to use /rx/ =~ str over str =~ /rx/ - to me this
makes it clearer that the RX is the one that does the matching. Just
personal taste maybe but I think I also remember that that variant is a
tad faster.

I didn't even realize this could be done :slight_smile: (though I see now that it is
documented, =~ being a synonym for Regexp#match). If I lived in a
vacuum for the last 15 years, and Ruby was the first and only
programming language I ever learned, then I would have done it that way,
too, from the very start. :slight_smile: Alas, I came from [other languages and
then] Perl, so it was just a carry over to continue using str =~
/regexp/.

FWIW, we still have the same problem:

irb(main):255:0> r1 =~ s
=> 0
irb(main):256:0> r2 =~ s
=> nil
irb(main):257:0> r3 =~ s
=> false

I've taken note that you say that r =~ s is faster. I (or someone else)
will have to do some benchmarking to see whether that's really true, and
how much speed gain can be had. Diakonos suffers when you use large and
many regexps for syntax highlighting, so I'd be interested in anything
that can speed that up.

Pistos

···

--
Posted via http://www.ruby-forum.com/\.

9

I do not want to enter into the discussion if it should be like that or not.
The beauty of ruby is that you can change a lot of things if you do not like
them.
In your case, maybe you like the following

···

On 3/29/06, Pistos Christou <jesusrubsyou.5.pistos@geoshell.com> wrote:

FWIW, we still have the same problem:

irb(main):255:0> r1 =~ s
=> 0
irb(main):256:0> r2 =~ s
=> nil
irb(main):257:0> r3 =~ s
=> false

I've taken note that you say that r =~ s is faster. I (or someone else)
will have to do some benchmarking to see whether that's really true, and
how much speed gain can be had. Diakonos suffers when you use large and
many regexps for syntax highlighting, so I'd be interested in anything
that can speed that up.

Pistos

--
Posted via http://www.ruby-forum.com/\.

Pistos

----------------------------------------------------------------
class String
    alias_method :__old_match, :=~
    def =~(obj)
        raise RuntimeError,
               "#{obj.nil? ? "nil" : obj.to_s} is not a Regexp but #{
obj.class}" unless Regexp === obj
        __old_match obj
    end # def =~(obj)
end # class String

puts "x" =~ %r{.}
puts "x" =~ nil

------------------------------

Hope this is helpfull

Cheers
Robert

--
Deux choses sont infinies : l'univers et la bêtise humaine ; en ce qui
concerne l'univers, je n'en ai pas acquis la certitude absolue.

- Albert Einstein

10

Robert Dober wrote:

class String
    alias_method :__old_match, :=~
    def =~(obj)
        raise RuntimeError,
               "#{obj.nil? ? "nil" : obj.to_s} is not a Regexp but #{
obj.class}" unless Regexp === obj
        __old_match obj
    end # def =~(obj)
end # class String

Thanks for the suggestion, Robert.

While I have no qualms about extending core classes, this sort of
adjustment feels like too... "brash"? of a modification for me. :slight_smile:

In this particular case, rewriting my if line is an acceptable solution
to the problem.

Pistos

···

--
Posted via http://www.ruby-forum.com/\.

11

Hi --

Robert Dober wrote:

class String
    alias_method :__old_match, :=~
    def =~(obj)
        raise RuntimeError,
               "#{obj.nil? ? "nil" : obj.to_s} is not a Regexp but #{
obj.class}" unless Regexp === obj
        __old_match obj
    end # def =~(obj)
end # class String

Thanks for the suggestion, Robert.

While I have no qualms about extending core classes, this sort of
adjustment feels like too... "brash"? of a modification for me. :slight_smile:

You should have *some* qualms :slight_smile: But in any case, this change is
certainly one I wouldn't make. I have no way of knowing whether
someone has done this somewhere:

   if str =~ re

and not tested for re being nil because it doesn't affect the outcome
of the test.

David

···

On Thu, 30 Mar 2006, Pistos Christou wrote:

--
David A. Black (dblack@wobblini.net)
Ruby Power and Light, LLC (http://www.rubypowerandlight.com)

"Ruby for Rails" chapters now available
from Manning Early Access Program! Ruby for Rails

12

You should have *some* qualms :slight_smile: But in any case, this change is
certainly one I wouldn't make. I have no way of knowing whether
someone has done this somewhere:

   if str =~ re

and not tested for re being nil because it doesn't affect the outcome
of the test.

In fact, my initial guess[1] was that String#=~ is defined something like:

  class String
    def =~( other )
      other =~ self
    end
  end

And of course the default Object#=~ is:

  class Object
    def =~( other )
      false
    end
  end

(observed by the fact that Object.new =~ Object.new returns false
without raising NoMethodException).

This is useful, because someone might come along and define #=~ for
their new class Foo which is close enough semantically to a Regexp to
merit the operator overload, but is not a Regexp. ri tells me that
REXML::Light::Node#=~ exists, for example. Doing this would cause that
to break.

-- Jacob Fugal

[1] The actual behavior of String#=~ is documented (from ri):

     If _obj_ is a +Regexp+, use it as a pattern to match
     against _str_. If _obj_ is a +String+, look for it in _str_
     (similar to +String#index+). Returns the position the match starts,
     or +nil+ if there is no match. Otherwise, invokes _obj.=~_, passing
     _str_ as an argument. The default +=~+ in +Object+ returns +false+.

However, trying string ~ another_string gives me an exception in irb:

  $ irb
  >> "looking for bob..." =~ "bob"
  TypeError: type mismatch: String given
        from (irb):1:in `=~'
        from (irb):1

Is this a documentation bug?

···

On 3/29/06, dblack@wobblini.net <dblack@wobblini.net> wrote:

13

>
> Thanks for the suggestion, Robert.
>
> While I have no qualms about extending core classes, this sort of
> adjustment feels like too... "brash"? of a modification for me. :slight_smile:

You should have *some* qualms :slight_smile: But in any case, this change is
certainly one I wouldn't make. I have no way of knowing whether
someone has done this somewhere:

   if str =~ re

and not tested for re being nil because it doesn't affect the outcome
of the test.

David

Now I am very much against extending core classes especally in commonly
used modules.
There was an interesting discussion about Rails doing so to an extreme
extend.
However I do not hesitate a second to do it if I feel it gives me consistent
behaviour throughout my applications especially when I change some rather
"strange" behaviour.
It is always a good idea to be ready to break "rules" if there are good
reasons to do so.
This might as well be such a case.
Please note, and that is important, that my modification will break code
clearly in a well defined manner.
It all comes down to the following decision you have to make:
(A) Somebody might rely on
   "a" =~ nil returning false
in that case abandon!!! (and ask yourself why you have posted that
question!)

(B) Matching with nil is probably a mistake that will break the code later,
in other words your astonishement about that behaviour will be "common
sense", in that case the above extension of a core class is just a great
idea.

Robert

···

--

David A. Black (dblack@wobblini.net)
Ruby Power and Light, LLC (http://www.rubypowerandlight.com)

"Ruby for Rails" chapters now available
from Manning Early Access Program! Ruby for Rails

--
Deux choses sont infinies : l'univers et la bêtise humaine ; en ce qui
concerne l'univers, je n'en ai pas acquis la certitude absolue.

- Albert Einstein

14

[snip]

However, trying string ~ another_string gives me an exception in irb:

  $ irb
>> "looking for bob..." =3D~ "bob"
  TypeError: type mismatch: String given
        from (irb):1:in `=3D~'
        from (irb):1

Is this a documentation bug?

Yes, I submitted a patch last week via the rubyforge bug form. Basically,
String =~ String was deprecated in 1.8.0 / 1.8.1 (generated obsolete
warning), and generates a TypeError since 1.8.2. I have no idea on the
status of the patch.

andrew

···

On Thu, 30 Mar 2006 02:24:41 +0900, Jacob Fugal <lukfugl@gmail.com> wrote:

--
Andrew L. Johnson http://www.siaris.net/
      It's kinda hard trying to remember Perl syntax *and* Occam's
      razor at the same time :slight_smile:
          -- Graham Patterson

15

(A) Somebody might rely on
   "a" =~ nil returning false
in that case abandon!!! (and ask yourself why you have posted that
question!)

I don't think anyone's code will rely on "a" =~ nil returning false.
However, it's possible and even likely that someone's code will rely
on "a" =~ not_a_regex not raising an exception For instance,
REXML::Light::Node supports #=~, as mentioned in my previous post. I
wouldn't take kindly to an "extension" that breaks REXML.

(B) Matching with nil is probably a mistake that will break the code later,
in other words your astonishement about that behaviour will be "common
sense", in that case the above extension of a core class is just a great
idea.

I agree matching with nil is probably never the intended usage.
Perhaps String#=~ should be patched specifically to not allow nil. The
current behavior is a side effect of nil being an Object and thus
inheriting Object#=~.

However, I disagree with the sweeping "extension" whereby anything
that's not a Regexp causes String#=~ to raise an exception. I take
this as an example of how tricky it can be to *safely* "extend" the
core modules. This doesn't mean you can't do it, just that you should
be *very careful.

Jacob Fugal

···

On 3/29/06, Robert Dober <robert.dober@gmail.com> wrote:

16

Hi --

···

On Thu, 30 Mar 2006, Robert Dober wrote:

It all comes down to the following decision you have to make:
(A) Somebody might rely on
  "a" =~ nil returning false
in that case abandon!!! (and ask yourself why you have posted that
question!)

(B) Matching with nil is probably a mistake that will break the code later,
in other words your astonishement about that behaviour will be "common
sense", in that case the above extension of a core class is just a great
idea.

(B) is where the dangers lie. You may think that someone who uses a
particular feature of Ruby is not programming well, but still, it's
only yourself that you punish by making your code incompatible with
the language :slight_smile:

David

--
David A. Black (dblack@wobblini.net)
Ruby Power and Light, LLC (http://www.rubypowerandlight.com)

"Ruby for Rails" chapters now available
from Manning Early Access Program! Ruby for Rails

17

Ok, thanks. Glad to know there's already been a patch submitted.

Jacob Fugal

···

On 3/29/06, Andrew Johnson <ajohnson@cpan.org> wrote:

On Thu, 30 Mar 2006 02:24:41 +0900, Jacob Fugal <lukfugl@gmail.com> wrote:
> Is this a documentation bug?

Yes, I submitted a patch last week via the rubyforge bug form. Basically,
String =~ String was deprecated in 1.8.0 / 1.8.1 (generated obsolete
warning), and generates a TypeError since 1.8.2. I have no idea on the
status of the patch.

18

> (A) Somebody might rely on
> "a" =~ nil returning false
> in that case abandon!!! (and ask yourself why you have posted that
> question!)

I don't think anyone's code will rely on "a" =~ nil returning false.
However, it's possible and even likely that someone's code will rely
on "a" =~ not_a_regex not raising an exception For instance,
REXML::Light::Node supports #=~, as mentioned in my previous post. I
wouldn't take kindly to an "extension" that breaks REXML.

So you strongly oppose Rails?

(B) Matching with nil is probably a mistake that will break the code
later,
> in other words your astonishement about that behaviour will be "common
> sense", in that case the above extension of a core class is just a great
> idea.

I agree matching with nil is probably never the intended usage.
Perhaps String#=~ should be patched specifically to not allow nil. The
current behavior is a side effect of nil being an Object and thus
inheriting Object#=~.

However, I disagree with the sweeping "extension" whereby anything
that's not a Regexp causes String#=~ to raise an exception. I take
this as an example of how tricky it can be to *safely* "extend" the
core modules. This doesn't mean you can't do it, just that you should
be *very careful.

That was just a toy extension to pass the idea of what ruby can do to the
poster.
Than I found myself mildly attacked about the extension stuff, so I got
mildely defensive.
It is *not* fair to judge the whole idea of extending or not extending core
objects by a simple example that was written to show the power of ruby.

I am however open for discussion if that power of ruby shall be "removed",
"restricted" or kept.
I think that -w could do things about it or SAVE levels.
Probably worth a different thread.
But I am not Matz :frowning:

Cheers
Robert

···

On 3/30/06, Jacob Fugal <lukfugl@gmail.com> wrote:

On 3/29/06, Robert Dober <robert.dober@gmail.com> wrote:

--
Deux choses sont infinies : l'univers et la bêtise humaine ; en ce qui
concerne l'univers, je n'en ai pas acquis la certitude absolue.

- Albert Einstein

19

> I don't think anyone's code will rely on "a" =~ nil returning false.
> However, it's possible and even likely that someone's code will rely
> on "a" =~ not_a_regex not raising an exception For instance,
> REXML::Light::Node supports #=~, as mentioned in my previous post. I
> wouldn't take kindly to an "extension" that breaks REXML.

So you strongly oppose Rails?

I wasn't aware Rails (or probably more particularly, ActiveSupport)
broke REXML. If so, then yes I'd strongly oppose that particular
extension within ActiveSupport/Rails that breaks REXML.

> However, I disagree with the sweeping "extension" whereby anything
> that's not a Regexp causes String#=~ to raise an exception. I take
> this as an example of how tricky it can be to *safely* "extend" the
> core modules. This doesn't mean you can't do it, just that you should
> be *very careful.

That was just a toy extension to pass the idea of what ruby can do to the
poster. Than I found myself mildly attacked about the extension stuff, so I got
mildely defensive. It is *not* fair to judge the whole idea of extending or not extending
core objects by a simple example that was written to show the power of ruby.

I understand that. I just thought it a good opportunity to point out
how easy it is to make mistakes like this. I make them all the time as
well. I was not judging the general idea of extending core objects --
I, like you, would be very put out if the power were removed. I like
to play/use it and I've seen some very nifty and useful things done
with it. I was just pointing out how easy it is to break other
people's code if you extend core classes without being careful.

And don't worry, we weren't attacking you or your coding ability by
pointing out the flaws in the code you proposed as an example. Just
trying to make sure the bases were covered. Please keep experimenting
and posting your ideas; and when you find flaws in my code, enlighten
me please! :slight_smile:

Jacob Fugal

···

On 3/30/06, Robert Dober <robert.dober@gmail.com> wrote:

On 3/30/06, Jacob Fugal <lukfugl@gmail.com> wrote: