Remove non-ASCII characters in a string

ruby-talk
1

Hi,

i needed a method to convert a piece of text to plain ascii and
replace all non-ascii chars with a placeholder. I could not find
anything in the stdlib so I wrote one.

I'd love to hear your comments. (or pointers to existing libraries for
this task)

-Levin

#!/usr/bin/ruby

require 'iconv'

class String

  # removes all characters which are not part of ascii
  # and replaces them with +replacement+

···

#
  # +replacement+ is supposed to be the same encoding as +source+
  #
  def asciify(replacement = "?", target = "ASCII", source = "UTF-8")
    intermediate = "UCS-4"
    pack_format = "N*"
    i = Iconv.new(intermediate, source)

    u16s = i.iconv(self)
    repl = i.iconv(replacement).unpack(pack_format)

    s = u16s.unpack(pack_format).collect { |codepoint|
      codepoint < 128 ? codepoint : repl
    }.flatten.pack(pack_format)

    return Iconv.new(target, intermediate).iconv(s)
  end
end

if __FILE__ == $0
  require 'test/unit'

  class TestAsciify < Test::Unit::TestCase
    def test_asciify
      assert_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz?ti?n"
      assert_equal "Mötorhead".asciify("(removed)"), "M(removed)torhead"
    end
  end
end

2

I have a need for something like this as well. But I need to replace the chars with something plain ascii besides a placeholder. Any ideas how to do that?

  I ended up finding the escape codes for all the chars like "\322" and friends so I could replace say curly quotes with standard quotes and stuff like that.

  I will play with your code a bit and see if I can make it do what I want. Thanks for sharing it though.

Cheers-
-Ezra

···

On Jan 22, 2006, at 10:41 AM, Levin Alexander wrote:

Hi,

i needed a method to convert a piece of text to plain ascii and
replace all non-ascii chars with a placeholder. I could not find
anything in the stdlib so I wrote one.

I'd love to hear your comments. (or pointers to existing libraries for
this task)

-Levin

#!/usr/bin/ruby

require 'iconv'

class String

  # removes all characters which are not part of ascii
  # and replaces them with +replacement+
  #
  # +replacement+ is supposed to be the same encoding as +source+
  #
  def asciify(replacement = "?", target = "ASCII", source = "UTF-8")
    intermediate = "UCS-4"
    pack_format = "N*"
    i = Iconv.new(intermediate, source)

    u16s = i.iconv(self)
    repl = i.iconv(replacement).unpack(pack_format)

    s = u16s.unpack(pack_format).collect { |codepoint|
      codepoint < 128 ? codepoint : repl
    }.flatten.pack(pack_format)

    return Iconv.new(target, intermediate).iconv(s)
  end
end

if __FILE__ == $0
  require 'test/unit'

  class TestAsciify < Test::Unit::TestCase
    def test_asciify
      assert_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz?ti?n"
      assert_equal "Mötorhead".asciify("(removed)"), "M(removed)torhead"
    end
  end
end

-Ezra Zygmuntowicz
WebMaster
Yakima Herald-Republic Newspaper

ezra@yakima-herald.com
blog: http://brainspl.at

3

What is considered an ascii char? i used the
http://www.lookuptables.com/ chart from ! to ~ or 33 .. 127

class String
def remove_nonascii(replacement)
n=self.split("")
self.slice!(0..self.size)
n.each{|b|
if b[0].to_i< 33 || b[0].to_i>127 then
   self.concat(replacement)
   else
   self.concat(b)
end
}
self.to_s
end
end
require 'test/unit'

class TestAsciify < Test::Unit::TestCase
   def test_asciify
     assert_equal "Iñtërnâtiônàlizætiøn".remove_nonascii("?"),
"I?t?rn?ti?n?liz?ti?n"
     assert_equal "Mötorhead".remove_nonascii("(removed)"), "M(removed)torhead"
   end
end

···

On 1/22/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:

        I have a need for something like this as well. But I need to
replace the chars with something plain ascii besides a placeholder.
Any ideas how to do that?

        I ended up finding the escape codes for all the chars like "\322"
and friends so I could replace say curly quotes with standard quotes
and stuff like that.

        I will play with your code a bit and see if I can make it do what I
want. Thanks for sharing it though.

Cheers-
-Ezra

On Jan 22, 2006, at 10:41 AM, Levin Alexander wrote:

> Hi,
>
> i needed a method to convert a piece of text to plain ascii and
> replace all non-ascii chars with a placeholder. I could not find
> anything in the stdlib so I wrote one.
>
> I'd love to hear your comments. (or pointers to existing libraries for
> this task)
>
> -Levin
>
>
> #!/usr/bin/ruby
>
> require 'iconv'
>
> class String
>
> # removes all characters which are not part of ascii
> # and replaces them with +replacement+
> #
> # +replacement+ is supposed to be the same encoding as +source+
> #
> def asciify(replacement = "?", target = "ASCII", source = "UTF-8")
> intermediate = "UCS-4"
> pack_format = "N*"
> i = Iconv.new(intermediate, source)
>
> u16s = i.iconv(self)
> repl = i.iconv(replacement).unpack(pack_format)
>
> s = u16s.unpack(pack_format).collect { |codepoint|
> codepoint < 128 ? codepoint : repl
> }.flatten.pack(pack_format)
>
> return Iconv.new(target, intermediate).iconv(s)
> end
> end
>
> if __FILE__ == $0
> require 'test/unit'
>
> class TestAsciify < Test::Unit::TestCase
> def test_asciify
> assert_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz?
> ti?n"
> assert_equal "Mötorhead".asciify("(removed)"), "M(removed)
> torhead"
> end
> end
> end

-Ezra Zygmuntowicz
WebMaster
Yakima Herald-Republic Newspaper
http://yakimaherald.com
ezra@yakima-herald.com
blog: http://brainspl.at

4

What I have now looks like this:

  # basic usage
  "schön".asciify #=> "sch?n"
  # with mapping
  map = Asciify::Mapping.new(:default)
  "„foo"".asciify(map) #=> '"foo"'
  Asciify.new(map).convert("schön") #=> "schoen"
  Asciify.new(Asciify::HTMLEntities.new).convert("schön") #=> "sch&#246;n"

Mapping.new(:default) reads the mappings from a YAML file, you can use
Mapping.new("file.yaml") to load your own mappings or supply a Hash or
lambda to Asciify.new.

I have put it on rubyforge. This is my very first piece of released
code I'd very much like to hear your comments and criticism.

I'll try to work in the posted mappings.

Regards,
Levin

···

On 1/22/06, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:

        I have a need for something like this as well. But I need to
replace the chars with something plain ascii besides a placeholder.
Any ideas how to do that?

5

What is considered an ascii char? i used the
http://www.lookuptables.com/ chart from ! to ~ or 33 .. 127

I use everything <127 because I want to preserve tabs and linebreaks

def remove_nonascii(replacement)

This does not work if the source text is UTF-8 encoded. On my machine:

  str = "ö" #=> "\303\266"
  str.remove_nonascii #=> "??"

-Levin

···

On 1/22/06, ruby talk <rubytalk@gmail.com> wrote:

6

ruby talk wrote:

What is considered an ascii char? i used the
http://www.lookuptables.com/ chart from ! to ~ or 33 .. 127

Your range excludes ASCII character 32, space.
" ".remove_nonascii("?") #=> "?"

You also likely want to include characters like tabs and newlines, which are
in the 0-31 control range.

Levin's original version treats the original text as UTF-8. Is that part of
the requirements?

My version might look like this:

NON_ASCII = /[\x80-\xff]/
"Iñtërnâtiônàlizætiøn".gsub(NON_ASCII, "?") #=> "I?t?rn?ti?n?liz?ti?n"

Cheers,
Dave

7

i'd like not to remove no-ascii chars but replace all accentuated chars
(in an UTF-8 string) by them non-accentuated counterpart :

è => e
ä => a
ç => c

[...]

what is the best way to do that in Ruby?

···

Dave Burt <dave@burt.id.au> wrote:

My version might look like this:

NON_ASCII = /[\x80-\xff]/
"Iñtërnâtiônàlizætiøn".gsub(NON_ASCII, "?") #=> "I?t?rn?ti?n?liz?ti?n"

--
une bévue

8

"Une bévue" wrote:

My version might look like this:

NON_ASCII = /[\x80-\xff]/
"Iñtërnâtiônàlizætiøn".gsub(NON_ASCII, "?") #=> "I?t?rn?ti?n?liz?ti?n"

i'd like not to remove no-ascii chars but replace all accentuated chars
(in an UTF-8 string) by them non-accentuated counterpart :

è => e
ä => a
ç => c

[...]

what is the best way to do that in Ruby?

Try the code below, translated from

There may be a potential problem matching over character boundaries, but I
think UTF-8's unique starting bytes avoid the issue. So this should work.
For long strings, it could be slow. If I wanted speed, I'd probably do the
same thing in C and make it an extension.

Cheers,
Dave

class String
# Translate accented utf8 characters over to non-accented
def utf8_trans_unaccent
  tranmap = {
    "\xC3\x80" => "A", "\xC3\x81" => "A", "\xC3\x82" => "A", "\xC3\x83" =>
"A",
    "\xC3\x84" => "A", "\xC3\x85" => "A", "\xC3\x86" => "AE","\xC3\x87" =>
"C",
    "\xC3\x88" => "E", "\xC3\x89" => "E", "\xC3\x8A" => "E", "\xC3\x8B" =>
"E",
    "\xC3\x8C" => "I", "\xC3\x8D" => "I", "\xC3\x8E" => "I", "\xC3\x8F" =>
"I",
    "\xC3\x90" => "D", "\xC3\x91" => "N", "\xC3\x92" => "O", "\xC3\x93" =>
"O",
    "\xC3\x94" => "O", "\xC3\x95" => "O", "\xC3\x96" => "O", "\xC3\x98" =>
"O",
    "\xC3\x99" => "U", "\xC3\x9A" => "U", "\xC3\x9B" => "U", "\xC3\x9C" =>
"U",
    "\xC3\x9D" => "Y", "\xC3\x9E" => "P", "\xC3\x9F" => "ss",
    "\xC3\xA0" => "a", "\xC3\xA1" => "a", "\xC3\xA2" => "a", "\xC3\xA3" =>
"a",
    "\xC3\xA4" => "a", "\xC3\xA5" => "a", "\xC3\xA6" => "ae","\xC3\xA7" =>
"c",
    "\xC3\xA8" => "e", "\xC3\xA9" => "e", "\xC3\xAA" => "e", "\xC3\xAB" =>
"e",
    "\xC3\xAC" => "i", "\xC3\xAD" => "i", "\xC3\xAE" => "i", "\xC3\xAF" =>
"i",
    "\xC3\xB0" => "o", "\xC3\xB1" => "n", "\xC3\xB2" => "o", "\xC3\xB3" =>
"o",
    "\xC3\xB4" => "o", "\xC3\xB5" => "o", "\xC3\xB6" => "o", "\xC3\xB8" =>
"o",
    "\xC3\xB9" => "u", "\xC3\xBA" => "u", "\xC3\xBB" => "u", "\xC3\xBC" =>
"u",
    "\xC3\xBD" => "y", "\xC3\xBE" => "p", "\xC3\xBF" => "y",
    "\xC4\x80" => "A", "\xC4\x81" => "a", "\xC4\x82" => "A", "\xC4\x83" =>
"a",
    "\xC4\x84" => "A", "\xC4\x85" => "a", "\xC4\x86" => "C", "\xC4\x87" =>
"c",
    "\xC4\x88" => "C", "\xC4\x89" => "c", "\xC4\x8A" => "C", "\xC4\x8B" =>
"c",
    "\xC4\x8C" => "C", "\xC4\x8D" => "c", "\xC4\x8E" => "D", "\xC4\x8F" =>
"d",
    "\xC4\x90" => "D", "\xC4\x91" => "d", "\xC4\x92" => "E", "\xC4\x93" =>
"e",
    "\xC4\x94" => "E", "\xC4\x95" => "e", "\xC4\x96" => "E", "\xC4\x97" =>
"e",
    "\xC4\x98" => "E", "\xC4\x99" => "e", "\xC4\x9A" => "E", "\xC4\x9B" =>
"e",
    "\xC4\x9C" => "G", "\xC4\x9D" => "g", "\xC4\x9E" => "G", "\xC4\x9F" =>
"g",
    "\xC4\xA0" => "G", "\xC4\xA1" => "g", "\xC4\xA2" => "G", "\xC4\xA3" =>
"g",
    "\xC4\xA4" => "H", "\xC4\xA5" => "h", "\xC4\xA6" => "H", "\xC4\xA7" =>
"h",
    "\xC4\xA8" => "I", "\xC4\xA9" => "i", "\xC4\xAA" => "I", "\xC4\xAB" =>
"i",
    "\xC4\xAC" => "I", "\xC4\xAD" => "i", "\xC4\xAE" => "I", "\xC4\xAF" =>
"i",
    "\xC4\xB0" => "I", "\xC4\xB1" => "i", "\xC4\xB2" => "IJ","\xC4\xB3" =>
"ij",
    "\xC4\xB4" => "J", "\xC4\xB5" => "j", "\xC4\xB6" => "K", "\xC4\xB7" =>
"k",
    "\xC4\xB8" => "k", "\xC4\xB9" => "L", "\xC4\xBA" => "l", "\xC4\xBB" =>
"L",
    "\xC4\xBC" => "l", "\xC4\xBD" => "L", "\xC4\xBE" => "l", "\xC4\xBF" =>
"L",
    "\xC5\x80" => "l", "\xC5\x81" => "L", "\xC5\x82" => "l", "\xC5\x83" =>
"N",
    "\xC5\x84" => "n", "\xC5\x85" => "N", "\xC5\x86" => "n", "\xC5\x87" =>
"N",
    "\xC5\x88" => "n", "\xC5\x89" => "n", "\xC5\x8A" => "N", "\xC5\x8B" =>
"n",
    "\xC5\x8C" => "O", "\xC5\x8D" => "o", "\xC5\x8E" => "O", "\xC5\x8F" =>
"o",
    "\xC5\x90" => "O", "\xC5\x91" => "o", "\xC5\x92" => "CE","\xC5\x93" =>
"ce",
    "\xC5\x94" => "R", "\xC5\x95" => "r", "\xC5\x96" => "R", "\xC5\x97" =>
"r",
    "\xC5\x98" => "R", "\xC5\x99" => "r", "\xC5\x9A" => "S", "\xC5\x9B" =>
"s",
    "\xC5\x9C" => "S", "\xC5\x9D" => "s", "\xC5\x9E" => "S", "\xC5\x9F" =>
"s",
    "\xC5\xA0" => "S", "\xC5\xA1" => "s", "\xC5\xA2" => "T", "\xC5\xA3" =>
"t",
    "\xC5\xA4" => "T", "\xC5\xA5" => "t", "\xC5\xA6" => "T", "\xC5\xA7" =>
"t",
    "\xC5\xA8" => "U", "\xC5\xA9" => "u", "\xC5\xAA" => "U", "\xC5\xAB" =>
"u",
    "\xC5\xAC" => "U", "\xC5\xAD" => "u", "\xC5\xAE" => "U", "\xC5\xAF" =>
"u",
    "\xC5\xB0" => "U", "\xC5\xB1" => "u", "\xC5\xB2" => "U", "\xC5\xB3" =>
"u",
    "\xC5\xB4" => "W", "\xC5\xB5" => "w", "\xC5\xB6" => "Y", "\xC5\xB7" =>
"y",
    "\xC5\xB8" => "Y", "\xC5\xB9" => "Z", "\xC5\xBA" => "z", "\xC5\xBB" =>
"Z",
    "\xC5\xBC" => "z", "\xC5\xBD" => "Z", "\xC5\xBE" => "z", "\xC6\x8F" =>
"E",
    "\xC6\xA0" => "O", "\xC6\xA1" => "o", "\xC6\xAF" => "U", "\xC6\xB0" =>
"u",
    "\xC7\x8D" => "A", "\xC7\x8E" => "a", "\xC7\x8F" => "I",
    "\xC7\x90" => "i", "\xC7\x91" => "O", "\xC7\x92" => "o", "\xC7\x93" =>
"U",
    "\xC7\x94" => "u", "\xC7\x95" => "U", "\xC7\x96" => "u", "\xC7\x97" =>
"U",
    "\xC7\x98" => "u", "\xC7\x99" => "U", "\xC7\x9A" => "u", "\xC7\x9B" =>
"U",
    "\xC7\x9C" => "u",
    "\xC7\xBA" => "A", "\xC7\xBB" => "a", "\xC7\xBC" => "AE","\xC7\xBD" =>
"ae",
    "\xC7\xBE" => "O", "\xC7\xBF" => "o",
    "\xC9\x99" => "e",

    "\xC2\x82" => ",", # High code comma
    "\xC2\x84" => ",", # High code double comma
    "\xC2\x85" => "...", # Tripple dot
    "\xC2\x88" => "^", # High carat
    "\xC2\x91" => "\x27", # Forward single quote
    "\xC2\x92" => "\x27", # Reverse single quote
    "\xC2\x93" => "\x22", # Forward double quote
    "\xC2\x94" => "\x22", # Reverse double quote
    "\xC2\x96" => "-", # High hyphen
    "\xC2\x97" => "--", # Double hyphen
    "\xC2\xA6" => "|", # Split vertical bar
    "\xC2\xAB" => "<<", # Double less than
    "\xC2\xBB" => ">>", # Double greater than
    "\xC2\xBC" => "1/4", # one quarter
    "\xC2\xBD" => "1/2", # one half
    "\xC2\xBE" => "3/4", # three quarters

    "\xCA\xBF" => "\x27", # c-single quote
    "\xCC\xA8" => "", # modifier - under curve
    "\xCC\xB1" => "" # modifier - under line
  }

  tranmap.inject(self) do |str, (utf8, asc)|
    p [utf8, asc]
    str.gsub(utf8, asc)
  end
end
end

"Iñtërnâtiônàlizætiøn".utf8_trans_unaccent #=> "Internationalizaetion"

···

Dave Burt <dave@burt.id.au> wrote:

9

pere.noel@laponie.com.invalid (Une bévue) writes:

My version might look like this:

NON_ASCII = /[\x80-\xff]/
"Iñtërnâtiônàlizætiøn".gsub(NON_ASCII, "?") #=> "I?t?rn?ti?n?liz?ti?n"

i'd like not to remove no-ascii chars but replace all accentuated chars
(in an UTF-8 string) by them non-accentuated counterpart :

è => e
ä => a
ç => c

[...]

what is the best way to do that in Ruby?

How about this:

require 'iconv'
puts Iconv.open("ASCII//TRANSLIT", "ISO-8859-1").iconv("Iñtërnâtiônàlizætiøn")

#=> I~nt"ern^ati^on`alizaetion

···

Dave Burt <dave@burt.id.au> wrote:

une bévue

--
Christian Neukirchen <chneukirchen@gmail.com> http://chneukirchen.org

10

thanks a lot this works great even with ligatures, i don't need speed
because i'll use that only for file names...

···

Dave Burt <dave@burt.id.au> wrote:

ry the code below, translated from
Display UCS-2 and UTF-8 codes - About

There may be a potential problem matching over character boundaries, but I
think UTF-8's unique starting bytes avoid the issue. So this should work.
For long strings, it could be slow. If I wanted speed, I'd probably do the
same thing in C and make it an extension.

--
une bévue

11

# one time preprocessing
INTL, ASC = "", ""
tranmap.each {|k,v|
  INTL << k
  ASC << v
}

# quicker than repeated gsubs:
str.tr(INTL, ASC)

martin

···

Dave Burt <dave@burt.id.au> wrote:

  tranmap.inject(self) do |str, (utf8, asc)|
    p [utf8, asc]
    str.gsub(utf8, asc)
  end
end
end

"Iñtërnâtiônàlizætiøn".utf8_trans_unaccent #=> "Internationalizaetion"

12

Ǟćḉệñťś, you say? What about these (incomplete list, and w/o
ligatures) :))))

a, 69,
AaªÀÁÂÃÄÅàáâãäåĀāĂ㥹ǍǎǞǟǠǡǺǻȀȁȂȃȦȧᴬᵃḀḁẠạẢảẤấẦầẨẩẪẫẬậẮắẰằẲẳẴẵẶặₐÅⒶⓐＡａ
b, 15, BbᴮᵇḂḃḄḅḆḇℬⒷⓑＢｂ
c, 23, CcÇçĆćĈĉĊċČčᶜḈḉℂℭⅭⅽⒸⓒＣｃ
d, 29,
DdÐðĎďĐđᴰᵈᶞḊḋḌḍḎḏḐḑḒḓⅅⅆⅮⅾⒹⓓＤｄ
e, 62,
EeÈÉÊËèéêëĒēĔĕĖėĘęĚěȄȅȆȇȨȩᴱᵉḔḕḖḗḘḙḚḛḜḝẸẹẺẻẼẽẾếỀềỂểỄễỆệₑℯℰⅇⒺⓔＥｅ
f, 10, FfᶠḞḟℱⒻⓕＦｆ
g, 23, GgĜĝĞğĠġĢģǦǧǴǵᴳᵍḠḡℊⒼⓖＧｇ
h, 30,
HhĤĥĦħȞȟʰᴴḢḣḤḥḦḧḨḩḪḫẖℋℌℍℎℏⒽⓗＨｈ
i, 46,
IiÌÍÎÏìíîïĨĩĪīĬĭĮįİǏǐȈȉȊȋᴵᵢḬḭḮḯỈỉỊịⁱℐℑℹⅈⅠⅰⒾⓘＩｉ
j, 12, JjĴĵǰʲᴶⅉⒿⓙＪｊ
k, 19, KkĶķǨǩᴷᵏḰḱḲḳḴḵKⓀⓚＫｋ
l, 28,
LlĹĺĻļĽľŁłˡᴸḶḷḸḹḺḻḼḽℒℓⅬⅼⓁⓛＬｌ
m, 17, MmᴹᵐḾḿṀṁṂṃℳⅯⅿⓂⓜＭｍ
n, 27,
NnÑñŃńŅņŇňǸǹᴺṄṅṆṇṈṉṊṋⁿℕⓃⓝＮｎ
o, 83,
OoºÒÓÔÕÖØòóôõöøŌōŎŏŐőƠơǑǒǪǫǬǭǾǿȌȍȎȏȪȫȬȭȮȯȰȱᴼᵒṌṍṎṏṐṑṒṓỌọỎỏỐốỒồỔổỖỗỘộỚớỜờỞởỠỡỢợₒℴⓄⓞＯｏ
p, 13, PpᴾᵖṔṕṖṗℙⓅⓟＰｐ
q, 7, QqℚⓆⓠＱｑ
r, 30,
RrŔŕŖŗŘřȐȑȒȓʳᴿᵣṘṙṚṛṜṝṞṟℛℜℝⓇⓡＲｒ
s, 29,
SsŚśŜŝŞşŠšſȘșˢṠṡṢṣṤṥṦṧṨṩẛⓈⓢＳｓ
t, 23, TtŢţŤťȚțᵀᵗṪṫṬṭṮṯṰṱẗⓉⓣＴｔ
u, 69,
UuÙÚÛÜùúûüŨũŪūŬŭŮůŰűŲųƯưǓǔǕǖǗǘǙǚǛǜȔȕȖȗᵁᵘᵤṲṳṴṵṶṷṸṹṺṻỤụỦủỨứỪừỬửỮữỰựⓊⓤＵｕ
v, 14, VvᵛᵥṼṽṾṿⅤⅴⓋⓥＶｖ
w, 21, WwŴŵʷᵂẀẁẂẃẄẅẆẇẈẉẘⓌⓦＷｗ
x, 14, XxˣẊẋẌẍₓⅩⅹⓍⓧＸｘ
y, 26,
YyÝýÿŶŷŸȲȳʸẎẏẙỲỳỴỵỶỷỸỹⓎⓨＹｙ
z, 21, ZzŹźŻżŽžᶻẐẑẒẓẔẕℤℨⓏⓩＺｚ

13

Martin DeMello wrote:

···

Dave Burt <dave@burt.id.au> wrote:

  tranmap.inject(self) do |str, (utf8, asc)|
    p [utf8, asc]
    str.gsub(utf8, asc)
  end
end
end

"Iñtërnâtiônàlizætiøn".utf8_trans_unaccent #=> "Internationalizaetion"

# one time preprocessing
INTL, ASC = "", ""
tranmap.each {|k,v|
INTL << k
ASC << v
}

# quicker than repeated gsubs:
str.tr(INTL, ASC)

Except that won't work, because tr only matches bytes, not multi-byte
characters.

(It might work after applying one of the Unicode string extensions that have
been floating around recently. But not in standard Ruby.)

Cheers,
Dave

14

Oh - didn't know that! Pretty sad. Thanks for the correction.

martin

···

Dave Burt <dave@burt.id.au> wrote:

Except that won't work, because tr only matches bytes, not multi-byte
characters.

(It might work after applying one of the Unicode string extensions that have
been floating around recently. But not in standard Ruby.)