# Ruby Regex **URL:** \ **Category:** ruby-talk\ **Created:** [13 January 2010 12:49 UTC](https://rubytalk.org/t/ruby-regex/56993 "2010-01-13T12:49:39Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Sriram\_Varahan](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@Sriram\_Varahan](https://rubytalk.org/u/Sriram_Varahan)\ **Post date:** [13 January 2010 12:49 UTC](https://rubytalk.org/t/ruby-regex/56993/1 "2010-01-13T12:49:39Z")
Hello, I have a string as a = "&0&1" I need to pass this value with the ampersand escaped to another command in my program. So I tried something like this: irb(main):038:0\> a.gsub(/&/,"\\&") =\> "&0&1" But if I replace the & with some other variable I get it properly working: irb(main):049:0\> a.gsub(/&/,"\\g") =\> "\\g0\\g1" Any explanation on why this is happening and how to go about escaping the &. Thanks. Sriram Varahan. > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [13 January 2010 13:11 UTC](https://rubytalk.org/t/ruby-regex/56993/2 "2010-01-13T13:11:25Z")
Sriram Varahan wrote: > Hello, > > I have a string as a = "&0&1" > > I need to pass this value with the ampersand escaped to another command > in my program. > > So I tried something like this: > > irb(main):038:0\> a.gsub(/&/,"\\&") > =\> "&0&1" That's because \& has a special meaning in a replacement string ("the matched string"). Either use a block to provide the replacement value (which doesn't do the backslash replacement), or put two backslashes in the replacement string. irb(main):002:0\> a.gsub(/&/) { "\\&" } =\> "\\&0\\&1" irb(main):003:0\> a.gsub(/&/, "\\\\&") =\> "\\&0\\&1" > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [13 January 2010 14:02 UTC](https://rubytalk.org/t/ruby-regex/56993/3 "2010-01-13T14:02:23Z")
But it's only special when preceded by a backslash, which is special in replacement strings. irb(main):002:0\> "123".gsub(/\d/, '\<&\>') =\> "\<&\>\<&\>\<&\>" irb(main):003:0\> "123".gsub(/\d/, '\<\\&\>') =\> "\<1\>\<2\>\<3\>" irb(main):004:0\> "123".gsub(/\d/, '\<\\\\&\>') =\> "\<\\&\>\<\\&\>\<\\&\>" Another example - backslash with group index: irb(main):011:0\> "abc".gsub(/\w(.)\w/, '\<1\>') =\> "\<1\>" irb(main):012:0\> "abc".gsub(/\w(.)\w/, '\<\\1\>') =\> "\" irb(main):013:0\> "abc".gsub(/\w(.)\w/, '\<\\\\1\>') =\> "\<\\1\>" So what you basically do here is you escape the escape so it looses its special meaning in the replacement string. 🙂 Kind regards robert > **···** > > 2010/1/13 Brian Candler \: > > > Sriram Varahan wrote: > > > > > Hello, > > > > > > I have a string as a = "&0&1" > > > > > > I need to pass this value with the ampersand escaped to another command > > > in my program. > > > > > > So I tried something like this: > > > > > > irb(main):038:0\> a.gsub(/&/,"\\&") > > > =\> "&0&1" > > > > That's because \& has a special meaning in a replacement string ("the > > matched string"). Either use a block to provide the replacement value > > (which doesn't do the backslash replacement), or put two backslashes in > > the replacement string. > > > > irb(main):002:0\> a.gsub(/&/) { "\\&" } > > =\> "\\&0\\&1" > > irb(main):003:0\> a.gsub(/&/, "\\\\&") > > =\> "\\&0\\&1" > > -- > remember.guy do |as, often| as.you\_can - without end > [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/) ---
**Author:** ![Sriram\_Varahan](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@Sriram\_Varahan](https://rubytalk.org/u/Sriram_Varahan)\ **Post date:** [14 January 2010 06:01 UTC](https://rubytalk.org/t/ruby-regex/56993/4 "2010-01-14T06:01:06Z")
Thanks Brian, Robert and Xavier for your explanation. It was very helpful. Regards, Sriram Varahan. > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Marnen\_Laibow-Koser](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@Marnen\_Laibow-Koser](https://rubytalk.org/u/Marnen_Laibow-Koser)\ **Post date:** [13 January 2010 15:56 UTC](https://rubytalk.org/t/ruby-regex/56993/5 "2010-01-13T15:56:35Z")
Robert Klemme wrote: > > > irb(main):038:0\> a.gsub(/&/,"\\&") > > > > =\> "\\&0\\&1" > > But it's only special when preceded by a backslash, which is special > in replacement strings. > > irb(main):002:0\> "123".gsub(/\d/, '\<&\>') > =\> "\<&\>\<&\>\<&\>" > irb(main):003:0\> "123".gsub(/\d/, '\<\\&\>') > =\> "\<1\>\<2\>\<3\>" > irb(main):004:0\> "123".gsub(/\d/, '\<\\\\&\>') > =\> "\<\\&\>\<\\&\>\<\\&\>" > > Another example - backslash with group index: > > irb(main):011:0\> "abc".gsub(/\w(.)\w/, '\<1\>') > =\> "\<1\>" > irb(main):012:0\> "abc".gsub(/\w(.)\w/, '\<\\1\>') > =\> "\" > irb(main):013:0\> "abc".gsub(/\w(.)\w/, '\<\\\\1\>') > =\> "\<\\1\>" > > So what you basically do here is you escape the escape so it looses > its special meaning in the replacement string. 🙂 ...and Ruby's stupid backslash handling strikes again. This is a completely brain-dead way to do it, and is one of the few things I really hate about Ruby. > Kind regards > > robert Best, > **···** > > > 2010/1/13 Brian Candler \: > > -- > Marnen Laibow-Koser > [http://www.marnen.org](http://www.marnen.org) > marnen@marnen.org > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Xavier\_Noria](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/xavier_noria/32/1867_2.png) [@Xavier\_Noria](https://rubytalk.org/u/Xavier_Noria)\ **Post date:** [13 January 2010 16:26 UTC](https://rubytalk.org/t/ruby-regex/56993/6 "2010-01-13T16:26:36Z")
String literals have a one-pass escaping at parse time, so that     "foo\\bar\nbaz" is an encoded way to express     foo\bar     baz And the result of that ordinary pass is what gsub receives. Then, at runtime gsub inspects its argument and looks in turn for occurrences of \1, \& and friends. That is gsub's contract, and has no relationship with string literals parsing. You need double-scaping for \1 and friends to skip both passes, one related to literals, and the other one related to how gsub works. ---
**Author:** ![Gary\_Wright](https://avatars.discourse-cdn.com/v4/letter/g/58956e/32.png) [@Gary\_Wright](https://rubytalk.org/u/Gary_Wright)\ **Post date:** [13 January 2010 17:35 UTC](https://rubytalk.org/t/ruby-regex/56993/7 "2010-01-13T17:35:46Z")
Is this really a Ruby snafu? It seems like it would be inherent in any sort of character escape sequence, of which there are many examples that have nothing at all to do with Ruby. Any pointers to alternative encoding schemes that avoid this problem? Gary Wright > **···** > > On Jan 13, 2010, at 10:56 AM, Marnen Laibow-Koser wrote: > > > ...and Ruby's stupid backslash handling strikes again. This is a > > completely brain-dead way to do it, and is one of the few things I > > really hate about Ruby. ---
**Author:** ![Marnen\_Laibow-Koser](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@Marnen\_Laibow-Koser](https://rubytalk.org/u/Marnen_Laibow-Koser)\ **Post date:** [13 January 2010 20:22 UTC](https://rubytalk.org/t/ruby-regex/56993/8 "2010-01-13T20:22:29Z")
Xavier Noria wrote: > String literals have a one-pass escaping at parse time, so that > >     "foo\\bar\nbaz" > > is an encoded way to express > >     foo\bar >     baz > > And the result of that ordinary pass is what gsub receives. > > Then, at runtime gsub inspects its argument and looks in turn for > occurrences of \1, \& and friends. That is gsub's contract, and has no > relationship with string literals parsing. > > You need double-scaping for \1 and friends to skip both passes, one > related to literals, and the other one related to how gsub works. Yes, I see that now. I wasn't aware that gsub did an extra parsing step. With that in mind, doubling backslashes makes sense. Best, > **···** > > -- > Marnen Laibow-Koser > [http://www.marnen.org](http://www.marnen.org) > marnen@marnen.org > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Marnen\_Laibow-Koser](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@Marnen\_Laibow-Koser](https://rubytalk.org/u/Marnen_Laibow-Koser)\ **Post date:** [13 January 2010 17:49 UTC](https://rubytalk.org/t/ruby-regex/56993/9 "2010-01-13T17:49:20Z")
Gary Wright wrote: > > ...and Ruby's stupid backslash handling strikes again. This is a > > completely brain-dead way to do it, and is one of the few things I > > really hate about Ruby. > > Is this really a Ruby snafu? Yes. The problem is that Ruby "helpfully" does another level of escaping, so that "\\&" is equivaIent to "\&", whereas it should simply take the escape at face value and consider it equivalent to the two characters \ and &. For real fun, try concatenating two strings, the first of which ends in a backslash. It's insane. > It seems like it would be inherent in > any sort of character escape sequence, of which there are many > examples that have nothing at all to do with Ruby. But Ruby has its own special brand of idiocy here. Even Perl and PHP get this right. > Any pointers to alternative encoding schemes that avoid this problem? It has nothing to do with encoding. It's a question of a particular point of stupidity in Ruby's parser and/or String class. > Gary Wright Best, > **···** > > > On Jan 13, 2010, at 10:56 AM, Marnen Laibow-Koser wrote: > > -- > Marnen Laibow-Koser > [http://www.marnen.org](http://www.marnen.org) > marnen@marnen.org > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Xavier\_Noria](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/xavier_noria/32/1867_2.png) [@Xavier\_Noria](https://rubytalk.org/u/Xavier_Noria)\ **Post date:** [13 January 2010 17:57 UTC](https://rubytalk.org/t/ruby-regex/56993/10 "2010-01-13T17:57:40Z")
I don't understand your point. The backslash is a special character in string literals. If you want to include one you need to escape it. That's pretty normal. What's your complain about parsing? This gotcha is related to gsub's contract, nor to rules for string literals themselves. > **···** > > On Wed, Jan 13, 2010 at 6:49 PM, Marnen Laibow-Koser \ wrote: > > > It has nothing to do with encoding. It's a question of a particular > > point of stupidity in Ruby's parser and/or String class. ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [13 January 2010 18:25 UTC](https://rubytalk.org/t/ruby-regex/56993/11 "2010-01-13T18:25:18Z")
Marnen Laibow-Koser wrote: > The problem is that Ruby "helpfully" does another level of > escaping It's necessary because it lets you use sequences like \n in double-quoted strings, and \" if you want a double-quote, and \#{expr} if you want literally # { expr } rather than interpolation. > so that "\\&" is equivaIent to "\&", whereas it should simply > take the escape at face value and consider it equivalent to the two > characters \ and &. Which is what happens in single-quoted strings. But you can't put any control character sequences like \n in those. > For real fun, try concatenating two strings, the first of which ends in > a backslash. It's insane. a = "abc\\" # that's a string ending with one backslash b = "def" c = a + b Looks OK to me. > But Ruby has its own special brand of idiocy here. Even Perl and PHP > get this right. Perl is \*exactly\* the same. #!/usr/bin/perl print "abc\\\n"; This prints abc\ - same as Ruby would. It was probably unfortunate that gsub uses sequences like \1 and the like in the substitution side though. But that's what perl does: #!/usr/bin/perl $\_ = "ab&de\n"; s/&/\&/; print; That prints ab&de, which is the problem the OP was grappling with. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Marnen\_Laibow-Koser](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@Marnen\_Laibow-Koser](https://rubytalk.org/u/Marnen_Laibow-Koser)\ **Post date:** [13 January 2010 20:24 UTC](https://rubytalk.org/t/ruby-regex/56993/12 "2010-01-13T20:24:32Z")
Brian Candler wrote: > Marnen Laibow-Koser wrote: > > > The problem is that Ruby "helpfully" does another level of > > escaping > > It's necessary because it lets you use sequences like \n in > double-quoted strings, and \" if you want a double-quote, and \#{expr} > if you want literally # { expr } rather than interpolation. > > > so that "\\&" is equivaIent to "\&", whereas it should simply > > take the escape at face value and consider it equivalent to the two > > characters \ and &. > > Which is what happens in single-quoted strings. But you can't put any > control character sequences like \n in those. I know that. > > For real fun, try concatenating two strings, the first of which ends in > > a backslash. It's insane. > > a = "abc\\" # that's a string ending with one backslash > b = "def" > c = a + b > > Looks OK to me. And to me too, when I just now tried it. I \*did\* run into a problem with this at one point, but I can't now reproduce it. Perhaps it was actually a gsub issue. I'm glad to know that Ruby's backslash handling is not as weird as I'd thought. Thanks for the correction. Best, > **···** > > -- > Marnen Laibow-Koser > [http://www.marnen.org](http://www.marnen.org) > marnen@marnen.org > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [14 January 2010 08:05 UTC](https://rubytalk.org/t/ruby-regex/56993/13 "2010-01-14T08:05:04Z")
🙂 For me there is actually something weird about Ruby's escape handling - but it's something else: in some circumstances Ruby allows you to \*omit\* a backslash which is meant to be convenient (I believe) but which leads to a certain inconsistency: irb(main):014:0\> '\1' # this might be seen as surprising =\> "\\1" irb(main):015:0\> '\\1' =\> "\\1" We can get a single backslash by just using one, but if we need more backslashes we need to escape: irb(main):027:0\> '\\1' =\> "\\1" irb(main):028:0\> '\\\1' =\> "\\\\1" irb(main):029:0\> '\\\\1' =\> "\\\\1" irb(main):030:0\> '\\\\\1' =\> "\\\\\\1" irb(main):031:0\> '\\\\\\1' =\> "\\\\\\1" For double quoted strings we always need to use two backslashes at least if followed by a digit: irb(main):016:0\> "\1" =\> "\x01" irb(main):017:0\> "\\1" =\> "\\1" and irb(main):018:0\> '\n' # this might be seen as surprising =\> "\\n" irb(main):019:0\> '\\n' =\> "\\n" but irb(main):020:0\> "\n" # a single newline =\> "\n" irb(main):021:0\> "\\n" # backslash and n =\> "\\n" Bottom line for me: I do not exploit the '\1' case and have made it a habit to use two backslashes whenever I need a literal backslash in a string. Kind regards robert > **···** > > 2010/1/13 Marnen Laibow-Koser \: > > > I'm glad to know that Ruby's backslash handling is not as weird as I'd > > thought. Thanks for the correction. > > -- > remember.guy do |as, often| as.you\_can - without end > [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/) ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [14 January 2010 10:15 UTC](https://rubytalk.org/t/ruby-regex/56993/14 "2010-01-14T10:15:43Z")
Robert Klemme wrote: > For me there is actually something weird about Ruby's escape handling > - but it's something else: in some circumstances Ruby allows you to > \*omit\* a backslash which is meant to be convenient (I believe) but > which leads to a certain inconsistency: > > irb(main):014:0\> '\1' # this might be seen as surprising > =\> "\\1" I think the principle is "single quoting does the absolute minimum amount of dequoting". However it \*has\* to support a way to get a single-quote within a single-quoted string, and they chose \'. As a consequence, it \*has\* to support \\ to get a single backslash within a single-quoted string. The question then is, should any other sequence like \1 raise an error, or return literal \ and 1 ? The alternative would have been to use two single quotes where you want a single quote within a string:    'It''s that time of day' I quite like that, but arguably it's just confusing in a different way. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [14 January 2010 11:43 UTC](https://rubytalk.org/t/ruby-regex/56993/15 "2010-01-14T11:43:22Z")
> Robert Klemme wrote: > > > For me there is actually something weird about Ruby's escape handling > > - but it's something else: in some circumstances Ruby allows you to > > \*omit\* a backslash which is meant to be convenient (I believe) but > > which leads to a certain inconsistency: > > > > irb(main):014:0\> '\1' # this might be seen as surprising > > =\> "\\1" > > I think the principle is "single quoting does the absolute minimum > amount of dequoting". Hmm, I never thought of it that way. I'm not sure I like this principle though. > However it \*has\* to support a way to get a single-quote within a > single-quoted string, and they chose \'. As a consequence, it \*has\* to > support \\ to get a single backslash within a single-quoted string. > > The question then is, should any other sequence like \1 raise an error, > or return literal \ and 1 ? I opt for raising a syntax error. I know, this is unlikely to happen anytime soon if only because of the large base of code that is potentially affected. With what I have seen over the past years, the number of backslashes needed for proper quoting (especially for #gsub and friends) has caused much confusion. I believe that could be avoided by disallowing the '\1'. > The alternative would have been to use two single quotes where you want > a single quote within a string: > > 'It''s that time of day' > > I quite like that, but arguably it's just confusing in a different way. I like the quoting approach better. 🙂 Kind regards robert > **···** > > 2010/1/14 Brian Candler \: > > -- > remember.guy do |as, often| as.you\_can - without end > [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/)