# Remove non-ASCII characters in a string **URL:** \ **Category:** ruby-talk\ **Created:** [22 January 2006 18:41 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239 "2006-01-22T18:41:02Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Levin\_Alexander1](https://avatars.discourse-cdn.com/v4/letter/l/97f17d/32.png) [@Levin\_Alexander1](https://rubytalk.org/u/Levin_Alexander1)\ **Post date:** [22 January 2006 18:41 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/1 "2006-01-22T18:41:02Z")
Hi, i needed a method to convert a piece of text to plain ascii and replace all non-ascii chars with a placeholder. I could not find anything in the stdlib so I wrote one. I'd love to hear your comments. (or pointers to existing libraries for this task) -Levin #!/usr/bin/ruby require 'iconv' class String   # removes all characters which are not part of ascii   # and replaces them with +replacement+ > **···** > > # >   # +replacement+ is supposed to be the same encoding as +source+ >   # >   def asciify(replacement = "?", target = "ASCII", source = "UTF-8") >     intermediate = "UCS-4" >     pack\_format = "N\*" >     i = Iconv.new(intermediate, source) > >     u16s = i.iconv(self) >     repl = i.iconv(replacement).unpack(pack\_format) > >     s = u16s.unpack(pack\_format).collect { |codepoint| >       codepoint \< 128 ? codepoint : repl >     }.flatten.pack(pack\_format) > >     return Iconv.new(target, intermediate).iconv(s) >   end > end > > if \_\_FILE\_\_ == $0 >   require 'test/unit' > >   class TestAsciify \< Test::Unit::TestCase >     def test\_asciify >       assert\_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz?ti?n" >       assert\_equal "Mötorhead".asciify("(removed)"), "M(removed)torhead" >     end >   end > end ---
**Author:** ![Ezra\_Zygmuntowicz](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@Ezra\_Zygmuntowicz](https://rubytalk.org/u/Ezra_Zygmuntowicz)\ **Post date:** [22 January 2006 22:20 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/2 "2006-01-22T22:20:27Z")
I have a need for something like this as well. But I need to replace the chars with something plain ascii besides a placeholder. Any ideas how to do that?   I ended up finding the escape codes for all the chars like "\322" and friends so I could replace say curly quotes with standard quotes and stuff like that.   I will play with your code a bit and see if I can make it do what I want. Thanks for sharing it though. Cheers- -Ezra > **···** > > On Jan 22, 2006, at 10:41 AM, Levin Alexander wrote: > > > Hi, > > > > i needed a method to convert a piece of text to plain ascii and > > replace all non-ascii chars with a placeholder. I could not find > > anything in the stdlib so I wrote one. > > > > I'd love to hear your comments. (or pointers to existing libraries for > > this task) > > > > -Levin > > > > #!/usr/bin/ruby > > > > require 'iconv' > > > > class String > > > >   # removes all characters which are not part of ascii > >   # and replaces them with +replacement+ > >   # > >   # +replacement+ is supposed to be the same encoding as +source+ > >   # > >   def asciify(replacement = "?", target = "ASCII", source = "UTF-8") > >     intermediate = "UCS-4" > >     pack\_format = "N\*" > >     i = Iconv.new(intermediate, source) > > > >     u16s = i.iconv(self) > >     repl = i.iconv(replacement).unpack(pack\_format) > > > >     s = u16s.unpack(pack\_format).collect { |codepoint| > >       codepoint \< 128 ? codepoint : repl > >     }.flatten.pack(pack\_format) > > > >     return Iconv.new(target, intermediate).iconv(s) > >   end > > end > > > > if \_\_FILE\_\_ == $0 > >   require 'test/unit' > > > >   class TestAsciify \< Test::Unit::TestCase > >     def test\_asciify > >       assert\_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz?ti?n" > >       assert\_equal "Mötorhead".asciify("(removed)"), "M(removed)torhead" > >     end > >   end > > end > > -Ezra Zygmuntowicz > WebMaster > Yakima Herald-Republic Newspaper > > > **[yakimaherald.com | Yakima, Washington - We Tell Your Stories](https://www.yakimaherald.com/)** > > ezra@yakima-herald.com > blog: [http://brainspl.at](http://brainspl.at) ---
**Author:** ![RubyTalk](https://avatars.discourse-cdn.com/v4/letter/r/9fc29f/32.png) [@RubyTalk](https://rubytalk.org/u/RubyTalk)\ **Post date:** [22 January 2006 22:43 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/3 "2006-01-22T22:43:37Z")
What is considered an ascii char? i used the [http://www.lookuptables.com/](http://www.lookuptables.com/) chart from ! to ~ or 33 .. 127 class String def remove\_nonascii(replacement) n=self.split("") self.slice!(0..self.size) n.each{|b| if b[0].to\_i\< 33 || b[0].to\_i\>127 then    self.concat(replacement)    else    self.concat(b) end } self.to\_s end end require 'test/unit' class TestAsciify \< Test::Unit::TestCase    def test\_asciify      assert\_equal "Iñtërnâtiônàlizætiøn".remove\_nonascii("?"), "I?t?rn?ti?n?liz?ti?n"      assert\_equal "Mötorhead".remove\_nonascii("(removed)"), "M(removed)torhead"    end end > **···** > > On 1/22/06, Ezra Zygmuntowicz \ wrote: > > >         I have a need for something like this as well. But I need to > > replace the chars with something plain ascii besides a placeholder. > > Any ideas how to do that? > > > >         I ended up finding the escape codes for all the chars like "\322" > > and friends so I could replace say curly quotes with standard quotes > > and stuff like that. > > > >         I will play with your code a bit and see if I can make it do what I > > want. Thanks for sharing it though. > > > > Cheers- > > -Ezra > > > > On Jan 22, 2006, at 10:41 AM, Levin Alexander wrote: > > > > \> Hi, > > \> > > \> i needed a method to convert a piece of text to plain ascii and > > \> replace all non-ascii chars with a placeholder. I could not find > > \> anything in the stdlib so I wrote one. > > \> > > \> I'd love to hear your comments. (or pointers to existing libraries for > > \> this task) > > \> > > \> -Levin > > \> > > \> > > \> #!/usr/bin/ruby > > \> > > \> require 'iconv' > > \> > > \> class String > > \> > > \> # removes all characters which are not part of ascii > > \> # and replaces them with +replacement+ > > \> # > > \> # +replacement+ is supposed to be the same encoding as +source+ > > \> # > > \> def asciify(replacement = "?", target = "ASCII", source = "UTF-8") > > \> intermediate = "UCS-4" > > \> pack\_format = "N\*" > > \> i = Iconv.new(intermediate, source) > > \> > > \> u16s = i.iconv(self) > > \> repl = i.iconv(replacement).unpack(pack\_format) > > \> > > \> s = u16s.unpack(pack\_format).collect { |codepoint| > > \> codepoint \< 128 ? codepoint : repl > > \> }.flatten.pack(pack\_format) > > \> > > \> return Iconv.new(target, intermediate).iconv(s) > > \> end > > \> end > > \> > > \> if \_\_FILE\_\_ == $0 > > \> require 'test/unit' > > \> > > \> class TestAsciify \< Test::Unit::TestCase > > \> def test\_asciify > > \> assert\_equal "Iñtërnâtiônàlizætiøn".asciify, "I?t?rn?ti?n?liz? > > \> ti?n" > > \> assert\_equal "Mötorhead".asciify("(removed)"), "M(removed) > > \> torhead" > > \> end > > \> end > > \> end > > > > -Ezra Zygmuntowicz > > WebMaster > > Yakima Herald-Republic Newspaper > > [http://yakimaherald.com](http://yakimaherald.com) > > ezra@yakima-herald.com > > blog: [http://brainspl.at](http://brainspl.at) ---
**Author:** ![Levin\_Alexander1](https://avatars.discourse-cdn.com/v4/letter/l/97f17d/32.png) [@Levin\_Alexander1](https://rubytalk.org/u/Levin_Alexander1)\ **Post date:** [24 January 2006 16:48 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/4 "2006-01-24T16:48:51Z")
What I have now looks like this:   # basic usage   "schön".asciify #=\> "sch?n"   # with mapping   map = Asciify::Mapping.new(:default)   "„foo"".asciify(map) #=\> '"foo"'   Asciify.new(map).convert("schön") #=\> "schoen"   Asciify.new(Asciify::HTMLEntities.new).convert("schön") #=\> "schön" Mapping.new(:default) reads the mappings from a YAML file, you can use Mapping.new("file.yaml") to load your own mappings or supply a Hash or lambda to Asciify.new. I have put it on rubyforge. This is my very first piece of released code I'd very much like to hear your comments and criticism. I'll try to work in the posted mappings. Regards, Levin > **···** > > On 1/22/06, Ezra Zygmuntowicz \ wrote: > > >         I have a need for something like this as well. But I need to > > replace the chars with something plain ascii besides a placeholder. > > Any ideas how to do that? ---
**Author:** ![Levin\_Alexander1](https://avatars.discourse-cdn.com/v4/letter/l/97f17d/32.png) [@Levin\_Alexander1](https://rubytalk.org/u/Levin_Alexander1)\ **Post date:** [22 January 2006 23:11 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/5 "2006-01-22T23:11:11Z")
> What is considered an ascii char? i used the > [http://www.lookuptables.com/](http://www.lookuptables.com/) chart from ! to ~ or 33 .. 127 I use everything \<127 because I want to preserve tabs and linebreaks > def remove\_nonascii(replacement) This does not work if the source text is UTF-8 encoded. On my machine:   str = "ö" #=\> "\303\266"   str.remove\_nonascii #=\> "??" -Levin > **···** > > On 1/22/06, ruby talk \ wrote: ---
**Author:** ![Dave\_Burt2](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/dave_burt2/32/2036_2.png) [@Dave\_Burt2](https://rubytalk.org/u/Dave_Burt2)\ **Post date:** [24 January 2006 15:46 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/6 "2006-01-24T15:46:11Z")
ruby talk wrote: > What is considered an ascii char? i used the > [http://www.lookuptables.com/](http://www.lookuptables.com/) chart from ! to ~ or 33 .. 127 Your range excludes ASCII character 32, space. " ".remove\_nonascii("?") #=\> "?" You also likely want to include characters like tabs and newlines, which are in the 0-31 control range. Levin's original version treats the original text as UTF-8. Is that part of the requirements? My version might look like this: NON\_ASCII = /[\x80-\xff]/ "Iñtërnâtiônàlizætiøn".gsub(NON\_ASCII, "?") #=\> "I?t?rn?ti?n?liz?ti?n" Cheers, Dave ---
**Author:** ![Pere\_Noel1](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@Pere\_Noel1](https://rubytalk.org/u/Pere_Noel1)\ **Post date:** [24 January 2006 15:47 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/7 "2006-01-24T15:47:01Z")
i'd like not to remove no-ascii chars but replace all accentuated chars (in an UTF-8 string) by them non-accentuated counterpart : è =\> e ä =\> a ç =\> c [...] what is the best way to do that in Ruby? > **···** > > Dave Burt \ wrote: > > > My version might look like this: > > > > NON\_ASCII = /[\x80-\xff]/ > > "Iñtërnâtiônàlizætiøn".gsub(NON\_ASCII, "?") #=\> "I?t?rn?ti?n?liz?ti?n" > > -- > une bévue ---
**Author:** ![Dave\_Burt2](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/dave_burt2/32/2036_2.png) [@Dave\_Burt2](https://rubytalk.org/u/Dave_Burt2)\ **Post date:** [24 January 2006 15:47 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/8 "2006-01-24T15:47:44Z")
"Une bévue" wrote: > > My version might look like this: > > > > NON\_ASCII = /[\x80-\xff]/ > > "Iñtërnâtiônàlizætiøn".gsub(NON\_ASCII, "?") #=\> "I?t?rn?ti?n?liz?ti?n" > > i'd like not to remove no-ascii chars but replace all accentuated chars > (in an UTF-8 string) by them non-accentuated counterpart : > > è =\> e > ä =\> a > ç =\> c > > [...] > > what is the best way to do that in Ruby? Try the code below, translated from > **[Display UCS-2 and UTF-8 codes - About](http://stuffofinterest.com/misc/utf8-about.html)** > > Discussion of comparison of Unicode UCS-2 and UTF-8 character encodings There may be a potential problem matching over character boundaries, but I think UTF-8's unique starting bytes avoid the issue. So this should work. For long strings, it could be slow. If I wanted speed, I'd probably do the same thing in C and make it an extension. Cheers, Dave class String # Translate accented utf8 characters over to non-accented def utf8\_trans\_unaccent   tranmap = {     "\xC3\x80" =\> "A", "\xC3\x81" =\> "A", "\xC3\x82" =\> "A", "\xC3\x83" =\> "A",     "\xC3\x84" =\> "A", "\xC3\x85" =\> "A", "\xC3\x86" =\> "AE","\xC3\x87" =\> "C",     "\xC3\x88" =\> "E", "\xC3\x89" =\> "E", "\xC3\x8A" =\> "E", "\xC3\x8B" =\> "E",     "\xC3\x8C" =\> "I", "\xC3\x8D" =\> "I", "\xC3\x8E" =\> "I", "\xC3\x8F" =\> "I",     "\xC3\x90" =\> "D", "\xC3\x91" =\> "N", "\xC3\x92" =\> "O", "\xC3\x93" =\> "O",     "\xC3\x94" =\> "O", "\xC3\x95" =\> "O", "\xC3\x96" =\> "O", "\xC3\x98" =\> "O",     "\xC3\x99" =\> "U", "\xC3\x9A" =\> "U", "\xC3\x9B" =\> "U", "\xC3\x9C" =\> "U",     "\xC3\x9D" =\> "Y", "\xC3\x9E" =\> "P", "\xC3\x9F" =\> "ss",     "\xC3\xA0" =\> "a", "\xC3\xA1" =\> "a", "\xC3\xA2" =\> "a", "\xC3\xA3" =\> "a",     "\xC3\xA4" =\> "a", "\xC3\xA5" =\> "a", "\xC3\xA6" =\> "ae","\xC3\xA7" =\> "c",     "\xC3\xA8" =\> "e", "\xC3\xA9" =\> "e", "\xC3\xAA" =\> "e", "\xC3\xAB" =\> "e",     "\xC3\xAC" =\> "i", "\xC3\xAD" =\> "i", "\xC3\xAE" =\> "i", "\xC3\xAF" =\> "i",     "\xC3\xB0" =\> "o", "\xC3\xB1" =\> "n", "\xC3\xB2" =\> "o", "\xC3\xB3" =\> "o",     "\xC3\xB4" =\> "o", "\xC3\xB5" =\> "o", "\xC3\xB6" =\> "o", "\xC3\xB8" =\> "o",     "\xC3\xB9" =\> "u", "\xC3\xBA" =\> "u", "\xC3\xBB" =\> "u", "\xC3\xBC" =\> "u",     "\xC3\xBD" =\> "y", "\xC3\xBE" =\> "p", "\xC3\xBF" =\> "y",     "\xC4\x80" =\> "A", "\xC4\x81" =\> "a", "\xC4\x82" =\> "A", "\xC4\x83" =\> "a",     "\xC4\x84" =\> "A", "\xC4\x85" =\> "a", "\xC4\x86" =\> "C", "\xC4\x87" =\> "c",     "\xC4\x88" =\> "C", "\xC4\x89" =\> "c", "\xC4\x8A" =\> "C", "\xC4\x8B" =\> "c",     "\xC4\x8C" =\> "C", "\xC4\x8D" =\> "c", "\xC4\x8E" =\> "D", "\xC4\x8F" =\> "d",     "\xC4\x90" =\> "D", "\xC4\x91" =\> "d", "\xC4\x92" =\> "E", "\xC4\x93" =\> "e",     "\xC4\x94" =\> "E", "\xC4\x95" =\> "e", "\xC4\x96" =\> "E", "\xC4\x97" =\> "e",     "\xC4\x98" =\> "E", "\xC4\x99" =\> "e", "\xC4\x9A" =\> "E", "\xC4\x9B" =\> "e",     "\xC4\x9C" =\> "G", "\xC4\x9D" =\> "g", "\xC4\x9E" =\> "G", "\xC4\x9F" =\> "g",     "\xC4\xA0" =\> "G", "\xC4\xA1" =\> "g", "\xC4\xA2" =\> "G", "\xC4\xA3" =\> "g",     "\xC4\xA4" =\> "H", "\xC4\xA5" =\> "h", "\xC4\xA6" =\> "H", "\xC4\xA7" =\> "h",     "\xC4\xA8" =\> "I", "\xC4\xA9" =\> "i", "\xC4\xAA" =\> "I", "\xC4\xAB" =\> "i",     "\xC4\xAC" =\> "I", "\xC4\xAD" =\> "i", "\xC4\xAE" =\> "I", "\xC4\xAF" =\> "i",     "\xC4\xB0" =\> "I", "\xC4\xB1" =\> "i", "\xC4\xB2" =\> "IJ","\xC4\xB3" =\> "ij",     "\xC4\xB4" =\> "J", "\xC4\xB5" =\> "j", "\xC4\xB6" =\> "K", "\xC4\xB7" =\> "k",     "\xC4\xB8" =\> "k", "\xC4\xB9" =\> "L", "\xC4\xBA" =\> "l", "\xC4\xBB" =\> "L",     "\xC4\xBC" =\> "l", "\xC4\xBD" =\> "L", "\xC4\xBE" =\> "l", "\xC4\xBF" =\> "L",     "\xC5\x80" =\> "l", "\xC5\x81" =\> "L", "\xC5\x82" =\> "l", "\xC5\x83" =\> "N",     "\xC5\x84" =\> "n", "\xC5\x85" =\> "N", "\xC5\x86" =\> "n", "\xC5\x87" =\> "N",     "\xC5\x88" =\> "n", "\xC5\x89" =\> "n", "\xC5\x8A" =\> "N", "\xC5\x8B" =\> "n",     "\xC5\x8C" =\> "O", "\xC5\x8D" =\> "o", "\xC5\x8E" =\> "O", "\xC5\x8F" =\> "o",     "\xC5\x90" =\> "O", "\xC5\x91" =\> "o", "\xC5\x92" =\> "CE","\xC5\x93" =\> "ce",     "\xC5\x94" =\> "R", "\xC5\x95" =\> "r", "\xC5\x96" =\> "R", "\xC5\x97" =\> "r",     "\xC5\x98" =\> "R", "\xC5\x99" =\> "r", "\xC5\x9A" =\> "S", "\xC5\x9B" =\> "s",     "\xC5\x9C" =\> "S", "\xC5\x9D" =\> "s", "\xC5\x9E" =\> "S", "\xC5\x9F" =\> "s",     "\xC5\xA0" =\> "S", "\xC5\xA1" =\> "s", "\xC5\xA2" =\> "T", "\xC5\xA3" =\> "t",     "\xC5\xA4" =\> "T", "\xC5\xA5" =\> "t", "\xC5\xA6" =\> "T", "\xC5\xA7" =\> "t",     "\xC5\xA8" =\> "U", "\xC5\xA9" =\> "u", "\xC5\xAA" =\> "U", "\xC5\xAB" =\> "u",     "\xC5\xAC" =\> "U", "\xC5\xAD" =\> "u", "\xC5\xAE" =\> "U", "\xC5\xAF" =\> "u",     "\xC5\xB0" =\> "U", "\xC5\xB1" =\> "u", "\xC5\xB2" =\> "U", "\xC5\xB3" =\> "u",     "\xC5\xB4" =\> "W", "\xC5\xB5" =\> "w", "\xC5\xB6" =\> "Y", "\xC5\xB7" =\> "y",     "\xC5\xB8" =\> "Y", "\xC5\xB9" =\> "Z", "\xC5\xBA" =\> "z", "\xC5\xBB" =\> "Z",     "\xC5\xBC" =\> "z", "\xC5\xBD" =\> "Z", "\xC5\xBE" =\> "z", "\xC6\x8F" =\> "E",     "\xC6\xA0" =\> "O", "\xC6\xA1" =\> "o", "\xC6\xAF" =\> "U", "\xC6\xB0" =\> "u",     "\xC7\x8D" =\> "A", "\xC7\x8E" =\> "a", "\xC7\x8F" =\> "I",     "\xC7\x90" =\> "i", "\xC7\x91" =\> "O", "\xC7\x92" =\> "o", "\xC7\x93" =\> "U",     "\xC7\x94" =\> "u", "\xC7\x95" =\> "U", "\xC7\x96" =\> "u", "\xC7\x97" =\> "U",     "\xC7\x98" =\> "u", "\xC7\x99" =\> "U", "\xC7\x9A" =\> "u", "\xC7\x9B" =\> "U",     "\xC7\x9C" =\> "u",     "\xC7\xBA" =\> "A", "\xC7\xBB" =\> "a", "\xC7\xBC" =\> "AE","\xC7\xBD" =\> "ae",     "\xC7\xBE" =\> "O", "\xC7\xBF" =\> "o",     "\xC9\x99" =\> "e",     "\xC2\x82" =\> ",", # High code comma     "\xC2\x84" =\> ",", # High code double comma     "\xC2\x85" =\> "...", # Tripple dot     "\xC2\x88" =\> "^", # High carat     "\xC2\x91" =\> "\x27", # Forward single quote     "\xC2\x92" =\> "\x27", # Reverse single quote     "\xC2\x93" =\> "\x22", # Forward double quote     "\xC2\x94" =\> "\x22", # Reverse double quote     "\xC2\x96" =\> "-", # High hyphen     "\xC2\x97" =\> "--", # Double hyphen     "\xC2\xA6" =\> "|", # Split vertical bar     "\xC2\xAB" =\> "\<\<", # Double less than     "\xC2\xBB" =\> "\>\>", # Double greater than     "\xC2\xBC" =\> "1/4", # one quarter     "\xC2\xBD" =\> "1/2", # one half     "\xC2\xBE" =\> "3/4", # three quarters     "\xCA\xBF" =\> "\x27", # c-single quote     "\xCC\xA8" =\> "", # modifier - under curve     "\xCC\xB1" =\> "" # modifier - under line   }   tranmap.inject(self) do |str, (utf8, asc)|     p [utf8, asc]     str.gsub(utf8, asc)   end end end "Iñtërnâtiônàlizætiøn".utf8\_trans\_unaccent #=\> "Internationalizaetion" > **···** > > > Dave Burt \ wrote: ---
**Author:** ![Christian\_Neukirche1](https://avatars.discourse-cdn.com/v4/letter/c/ee7513/32.png) [@Christian\_Neukirche1](https://rubytalk.org/u/Christian_Neukirche1)\ **Post date:** [24 January 2006 18:38 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/9 "2006-01-24T18:38:20Z")
pere.noel@laponie.com.invalid (Une bévue) writes: > > My version might look like this: > > > > NON\_ASCII = /[\x80-\xff]/ > > "Iñtërnâtiônàlizætiøn".gsub(NON\_ASCII, "?") #=\> "I?t?rn?ti?n?liz?ti?n" > > i'd like not to remove no-ascii chars but replace all accentuated chars > (in an UTF-8 string) by them non-accentuated counterpart : > > è =\> e > ä =\> a > ç =\> c > > [...] > > what is the best way to do that in Ruby? How about this: require 'iconv' puts Iconv.open("ASCII//TRANSLIT", "ISO-8859-1").iconv("Iñtërnâtiônàlizætiøn") #=\> I~nt"ern^ati^on`alizaetion > **···** > > > Dave Burt \ wrote: > > > une bévue > > -- > Christian Neukirchen \ [http://chneukirchen.org](http://chneukirchen.org) ---
**Author:** ![Pere\_Noel1](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@Pere\_Noel1](https://rubytalk.org/u/Pere_Noel1)\ **Post date:** [24 January 2006 15:47 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/10 "2006-01-24T15:47:55Z")
thanks a lot this works great even with ligatures, i don't need speed because i'll use that only for file names... > **···** > > Dave Burt \ wrote: > > > ry the code below, translated from > > [Display UCS-2 and UTF-8 codes - About](http://stuffofinterest.com/misc/utf8-about.html) > > > > There may be a potential problem matching over character boundaries, but I > > think UTF-8's unique starting bytes avoid the issue. So this should work. > > For long strings, it could be slow. If I wanted speed, I'd probably do the > > same thing in C and make it an extension. > > -- > une bévue ---
**Author:** ![Martin\_DeMello1](https://avatars.discourse-cdn.com/v4/letter/m/3be4f8/32.png) [@Martin\_DeMello1](https://rubytalk.org/u/Martin_DeMello1)\ **Post date:** [24 January 2006 15:48 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/11 "2006-01-24T15:48:52Z")
# one time preprocessing INTL, ASC = "", "" tranmap.each {|k,v|   INTL \<\< k   ASC \<\< v } # quicker than repeated gsubs: str.tr(INTL, ASC) martin > **···** > > Dave Burt \ wrote: > > >   tranmap.inject(self) do |str, (utf8, asc)| > >     p [utf8, asc] > >     str.gsub(utf8, asc) > >   end > > end > > end > > > > "Iñtërnâtiônàlizætiøn".utf8\_trans\_unaccent #=\> "Internationalizaetion" ---
**Author:** ![Dmitry\_Severin](https://avatars.discourse-cdn.com/v4/letter/d/73ab20/32.png) [@Dmitry\_Severin](https://rubytalk.org/u/Dmitry_Severin)\ **Post date:** [24 January 2006 15:48 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/12 "2006-01-24T15:48:21Z")
Ǟćḉệñťś, you say? What about these (incomplete list, and w/o ligatures) :)))) a, 69, AaªÀÁÂÃÄÅàáâãäåĀāĂ㥹ǍǎǞǟǠǡǺǻȀȁȂȃȦȧᴬᵃḀḁẠạẢảẤấẦầẨẩẪẫẬậẮắẰằẲẳẴẵẶặₐÅⒶⓐＡａ b, 15, BbᴮᵇḂḃḄḅḆḇℬⒷⓑＢｂ c, 23, CcÇçĆćĈĉĊċČčᶜḈḉℂℭⅭⅽⒸⓒＣｃ d, 29, DdÐðĎďĐđᴰᵈᶞḊḋḌḍḎḏḐḑḒḓⅅⅆⅮⅾⒹⓓＤｄ e, 62, EeÈÉÊËèéêëĒēĔĕĖėĘęĚěȄȅȆȇȨȩᴱᵉḔḕḖḗḘḙḚḛḜḝẸẹẺẻẼẽẾếỀềỂểỄễỆệₑℯℰⅇⒺⓔＥｅ f, 10, FfᶠḞḟℱⒻⓕＦｆ g, 23, GgĜĝĞğĠġĢģǦǧǴǵᴳᵍḠḡℊⒼⓖＧｇ h, 30, HhĤĥĦħȞȟʰᴴḢḣḤḥḦḧḨḩḪḫẖℋℌℍℎℏⒽⓗＨｈ i, 46, IiÌÍÎÏìíîïĨĩĪīĬĭĮįİǏǐȈȉȊȋᴵᵢḬḭḮḯỈỉỊịⁱℐℑℹⅈⅠⅰⒾⓘＩｉ j, 12, JjĴĵǰʲᴶⅉⒿⓙＪｊ k, 19, KkĶķǨǩᴷᵏḰḱḲḳḴḵKⓀⓚＫｋ l, 28, LlĹĺĻļĽľŁłˡᴸḶḷḸḹḺḻḼḽℒℓⅬⅼⓁⓛＬｌ m, 17, MmᴹᵐḾḿṀṁṂṃℳⅯⅿⓂⓜＭｍ n, 27, NnÑñŃńŅņŇňǸǹᴺṄṅṆṇṈṉṊṋⁿℕⓃⓝＮｎ o, 83, OoºÒÓÔÕÖØòóôõöøŌōŎŏŐőƠơǑǒǪǫǬǭǾǿȌȍȎȏȪȫȬȭȮȯȰȱᴼᵒṌṍṎṏṐṑṒṓỌọỎỏỐốỒồỔổỖỗỘộỚớỜờỞởỠỡỢợₒℴⓄⓞＯｏ p, 13, PpᴾᵖṔṕṖṗℙⓅⓟＰｐ q, 7, QqℚⓆⓠＱｑ r, 30, RrŔŕŖŗŘřȐȑȒȓʳᴿᵣṘṙṚṛṜṝṞṟℛℜℝⓇⓡＲｒ s, 29, SsŚśŜŝŞşŠšſȘșˢṠṡṢṣṤṥṦṧṨṩẛⓈⓢＳｓ t, 23, TtŢţŤťȚțᵀᵗṪṫṬṭṮṯṰṱẗⓉⓣＴｔ u, 69, UuÙÚÛÜùúûüŨũŪūŬŭŮůŰűŲųƯưǓǔǕǖǗǘǙǚǛǜȔȕȖȗᵁᵘᵤṲṳṴṵṶṷṸṹṺṻỤụỦủỨứỪừỬửỮữỰựⓊⓤＵｕ v, 14, VvᵛᵥṼṽṾṿⅤⅴⓋⓥＶｖ w, 21, WwŴŵʷᵂẀẁẂẃẄẅẆẇẈẉẘⓌⓦＷｗ x, 14, XxˣẊẋẌẍₓⅩⅹⓍⓧＸｘ y, 26, YyÝýÿŶŷŸȲȳʸẎẏẙỲỳỴỵỶỷỸỹⓎⓨＹｙ z, 21, ZzŹźŻżŽžᶻẐẑẒẓẔẕℤℨⓏⓩＺｚ ---
**Author:** ![Dave\_Burt2](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/dave_burt2/32/2036_2.png) [@Dave\_Burt2](https://rubytalk.org/u/Dave_Burt2)\ **Post date:** [24 January 2006 15:49 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/13 "2006-01-24T15:49:57Z")
Martin DeMello wrote: > **···** > > > Dave Burt \ wrote: > > > > >   tranmap.inject(self) do |str, (utf8, asc)| > > >     p [utf8, asc] > > >     str.gsub(utf8, asc) > > >   end > > > end > > > end > > > > > > "Iñtërnâtiônàlizætiøn".utf8\_trans\_unaccent #=\> "Internationalizaetion" > > > > # one time preprocessing > > INTL, ASC = "", "" > > tranmap.each {|k,v| > > INTL \<\< k > > ASC \<\< v > > } > > > > # quicker than repeated gsubs: > > str.tr(INTL, ASC) > > Except that won't work, because tr only matches bytes, not multi-byte > characters. > > (It might work after applying one of the Unicode string extensions that have > been floating around recently. But not in standard Ruby.) > > Cheers, > Dave ---
**Author:** ![Martin\_DeMello1](https://avatars.discourse-cdn.com/v4/letter/m/3be4f8/32.png) [@Martin\_DeMello1](https://rubytalk.org/u/Martin_DeMello1)\ **Post date:** [24 January 2006 15:51 UTC](https://rubytalk.org/t/remove-non-ascii-characters-in-a-string/24239/14 "2006-01-24T15:51:46Z")
Oh - didn't know that! Pretty sad. Thanks for the correction. martin > **···** > > Dave Burt \ wrote: > > > Except that won't work, because tr only matches bytes, not multi-byte > > characters. > > > > (It might work after applying one of the Unicode string extensions that have > > been floating around recently. But not in standard Ruby.)