Regexp case-insensitive ligature bug?

Andrew_Vit · 30 January 2016 20:00

Looking at issues in ruby-core, something was reported to be within
spec. I'd like to confirm my understanding of why this is not a bug:

"fFfG".scan /[\W]+/i
#=> ["fF"]

"fFfG".downcase.scan /[\W]+/
#=> []

What I think is happening:

The /i switch matches all alternative cases/accented characters.

"ff" is a character pair that has a unicode ligature equivalent
LATIN SMALL LIGATURE FF ("\uFB00")

It looks like it compares ASCII characters as lowercase internally,
since otherwise it wouldn't see "fF" as a ligature for "ff"

I don't understand why it considers the character pair as a ligature
"non-word" when matching this. This doesn't happen with characters
like "oe" vs. "œ".

The result is that scanning words like "effluent" using /i means ruby
would tell you it contains "non-word" characters, and that is very
surprising.

Thoughts?
Andrew Vit

Eric_Christopherson · 30 January 2016 20:35

Wow. That surprises me. Under what circumstances do strings of
characters (ASCII or otherwise) get transparently mapped to ligatures of
those same characters? How about precomposed characters with diacritics?
And why does this happen with /[\W]/ but not /\W/?

···

On Sat, Jan 30, 2016, Andrew Vit wrote:

Looking at issues in ruby-core, something was reported to be within
spec. I'd like to confirm my understanding of why this is not a bug:

    "fFfG".scan /[\W]+/i
    #=> ["fF"]

    "fFfG".downcase.scan /[\W]+/
    #=>

What I think is happening:

The /i switch matches all alternative cases/accented characters.

"ff" is a character pair that has a unicode ligature equivalent
LATIN SMALL LIGATURE FF ("\uFB00")

It looks like it compares ASCII characters as lowercase internally,
since otherwise it wouldn't see "fF" as a ligature for "ff"

I don't understand why it considers the character pair as a ligature
"non-word" when matching this. This doesn't happen with characters
like "oe" vs. "œ".

The result is that scanning words like "effluent" using /i means ruby
would tell you it contains "non-word" characters, and that is very
surprising.

Thoughts?
Andrew Vit

--
Eric Christopherson

Eric_Christopherson · 30 January 2016 23:24

Oh! I missed this in the original email, the part about accented
characters being matched. I suppose that is part of why ff ligature
would be matched by /ff/i, the same way e-acute would be matched by /e/.
I don't understand why it's considered non-word, though (nor why the
Kelvin symbol is word, regardless of its graphical resemblance to the
normal K character).

···

On Sat, Jan 30, 2016, Andrew Vit wrote:

The /i switch matches all alternative cases/accented characters.

--
Eric Christopherson

Matthew_Kerwin · 30 January 2016 22:13

Every time I see this come up I think 'that seems wrong' and then I think
'who does that anyway?'

Here's my understanding:
* the 'ﬀ' ligature is a non-word character (??)
* it has a case conversion, so is affected by the //i flag
So:
* /ﬀ/ is a subset of /\W/
* /ﬀ/i matches 'ﬀ', 'FF', 'ff', 'fF', and 'Ff'
* therefore /\W/i should match all of the above

I believe there is some complication about using a character class
(/[\W]/i, or even /[^\W]/i) which may or may not be implementation detail,
bug, or deeper complexity. I dunno.

As I suggested here years ago: https://bugs.ruby-lang.org/issues/4044 this
issue would go away if \w and \W were based on unicode character properties
(i.e. if 'ﬀ' was treated as a word character) -- it seems silly allowing
case conversion on non-word characters.

Cheers

Bryce_Kerley · 31 January 2016 04:29

Here's my understanding:
* the 'ﬀ' ligature is a non-word character (??)

Let’s go to the Unicode data: http://www.unicode.org/Public/9.0.0/ucd/UnicodeData-9.0.0d4.txt (1.6mb long text file)

FB00;LATIN SMALL LIGATURE FF;Ll;0;L;<compat> 0066 0066;;;;N;;;;;

The first few fields, in order:

1. `FB00` code point
2. `LATIN SMALL LIGATURE FF` name
3. `Ll` general category, in this case “Letter, lowercase” c.f. Unicode character property - Wikipedia
4. `0` is it combining (c.f. https://en.wikipedia.org/wiki/Combining_character\) 0 means it’s not
5. `L` what direction? `L` means left-to-right language

This meets some standard for being a word character, since the `Ll` category means it can be treated like a lowercase letter.

* it has a case conversion, so is affected by the //i flag

Unicode’s special casing data http://www.unicode.org/Public/9.0.0/ucd/SpecialCasing-9.0.0d6.txt specifies how to upper-, lower-, and title-case it:

FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF

1. `FB00` code point
2. `FB00` lower case (same)
3. `0046 0066` title case it as code points \x46 and \x66, i.e. `Ff`
4. `0046 0046` upper case it as code points \x46 and \x46, i.e. `FF`

Ruby doesn’t upper-case the ligature correctly:

    "ﬀ".upcase.codepoints.map{|x| x.to_s 16} #=> ["fb00"]
    "ﬀ".codepoints #=> [64256]
    "ﬀ".upcase.codepoints #=> [64256]

So:
* /ﬀ/ is a subset of /\W/

Possibly incorrectly.

* /ﬀ/i matches 'ﬀ', 'FF', 'ff', 'fF', and 'Ff'

Makes perfect sense.

* therefore /\W/i should match all of the above

Nice proof!

I believe there is some complication about using a character class
(/[\W]/i, or even /[^\W]/i) which may or may not be implementation detail,
bug, or deeper complexity. I dunno.

All of the above, probably

···

On 30 Jan 2016, at 17:13, Matthew Kerwin wrote:

Matthew_Kerwin · 30 January 2016 22:22

IOW, /[\W]/i asks for anything that could, case-insensitively, match a
non-word character. Not: a case insensitive match including only non-word
characters.

I'm away from a PC for a bit; can you tell me how it interacts with: ss
(ß), k (K=Kelvin), å (Å=Angstrom), etc.?

···

On 31/01/2016 8:13 AM, "Matthew Kerwin" <matthew@kerwin.net.au> wrote:

* /ﬀ/ is a subset of /\W/
* /ﬀ/i matches 'ﬀ', 'FF', 'ff', 'fF', and 'Ff'
* therefore /\W/i should match all of the above

Matthew_Kerwin · 1 February 2016 00:15

Here's my understanding:
* the 'ﬀ' ligature is a non-word character (??)

Let’s go to the Unicode data:
http://www.unicode.org/Public/9.0.0/ucd/UnicodeData-9.0.0d4.txt (1.6mb
long text file)

<snip>

This meets some standard for being a word character, since the Ll
category means it can be treated like a lowercase letter.

Yep, to quote myself from 3 years ago:

"Can you just fall back on the Unicode categories? If we define "word
characters" as Letters and Numbers, U+212A is {Lu} and thus a word
character. Similary U+017F is {Ll}."

- it has a case conversion, so is affected by the //i flag

<snip>

Ruby doesn’t upper-case the ligature correctly:

Ruby and Oniguruma (the regular expression engine) are different beasts,

and very definitely don't always behave the same. Especially

w.r.t Unicode properties, case handling, etc.

So:
* /

ﬀ/ is a subset of /\W/

Possibly incorrectly.

- /ﬀ/i matches 'ﬀ', 'FF', 'ff', 'fF', and 'Ff'

Makes perfect sense.

- therefore /\W/i should match all of the above

Nice proof!

I believe there is some complication about using a character class
(/[\W]/i, or even /[^\W]/i) which may or may not be implementation detail,
bug, or deeper complexity. I dunno.

All of the above, probably

And if anyone ever wants to experience it in all its quirky glory:

gist.github.com

https://gist.github.com/phluid61/67953b6aeb01c82d0c21

output.log

$ ruby -v re.rb 
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
non-word:
  /\W/    => []
  /\W/i   => []
  /[\W]/  => []
  /[\W]/i => ["fF", "Ss"]

not word/leading word:
  /^\w/    => ["f"]

This file has been truncated. show original

output2.log

$ jruby -v re.rb 
jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa OpenJDK 64-Bit Server VM 24.91-b01 on 1.7.0_91-b02 +jit [linux-amd64]
non-word:
  /\W/    => []
  /\W/i   => []
  /[\W]/  => []
  /[\W]/i => ["fF", "S", "s", "k"]

not word/leading word:
  /^\w/    => ["f"]

This file has been truncated. show original

re.rb

$string = 'fFfGSsoEk'.freeze

def demo re, desc
  puts "#{desc}:"
  print "  /#{re}/    => "; p $string.scan(/#{re}/)
  print "  /#{re}/i   => "; p $string.scan(/#{re}/i)
  print "  /[#{re}]/  => "; p $string.scan(/[#{re}]/)
  print "  /[#{re}]/i => "; p $string.scan(/[#{re}]/i)
  puts ''
end

This file has been truncated. show original

Cheers

···

On 31 January 2016 at 14:29, Bryce Kerley <bkerley@brycekerley.net> wrote:

On 30 Jan 2016, at 17:13, Matthew Kerwin wrote:

--
Matthew Kerwin
http://matthew.kerwin.net.au/

Andrew_Vit · 30 January 2016 23:27

I hadn't noticed the difference between /\W/i and /[\W]/i
I guess I didn't expect there would be one!

Regexp documentation says:

The following metacharacters also behave like character classes:
...
/\W/ - A non-word character ([^a-zA-Z0-9_]). Please take a look at Bug
#4044 if using /\W/ with the /i modifier.

That equivalent pattern seems to be lying a little bit...

It seems there are subtle interactions between metacharacters and
character classes. My expectation was that using a metacharacter within
a character class just adds to the set by conjunction (OR).

     /[ab]/ # "a" OR "b"
     /[a\d]/ # "a" OR <digit>
     /[\W]/ # <non-word> ???

IOW, /[\W]/i asks for anything that could, case-insensitively, match a
non-word character. Not: a case insensitive match including only
non-word characters.

Yes, that seems to be a valid interpretation.

What surprises me is that some sequences of simple "word" characters (\w+) are not mutually exclusive with "non-words", but that might just
be because of the metacharacter-in-character-class syntax.

     "s".scan /\w/i #=> ["s"]
     "s".scan /\W/i #=>
     "s".scan /[\W]/i #=> ["s"]

I'm away from a PC for a bit; can you tell me how it interacts with: ss
(ß), k (K=Kelvin), å (Å=Angstrom), etc.?

(?a..?z).each { |c| puts c if c =~ /[\W]/i }
k
s

     (?a..?z).each { |c|
       (?a..?z).each { |d| puts "#{c}#{d}" if "#{c}#{d}" =~ /[\W]/i }
     }
ak
as
bk
bs
ck
cs
dk
ds
ek
es
ff
fi
fk
fl
fs ... (103 results)

But "ae" / "oe" are not in there... fascinating.

Andrew Vit

···

On 2016-01-30 2:22 PM, Matthew Kerwin wrote:

Topic		Replies	Views
Win32 ruby1.9 regexp and cyrillic string ruby-talk	6	168	10 May 2010
Regex boundaries+Umlauts ruby-talk	5	541	5 February 2016
Regex bug? ruby-talk	8	249	20 January 2015
Puzzling regex behaviour ruby-talk	23	310	16 February 2007
Unicode in Regex ruby-talk	32	552	7 December 2007

Regexp case-insensitive ligature bug?

Related topics