Ruby unicode/string explosion (0xFF in utf-8)

ruby-talk
1

Hi, I'm using ruby 1.9.2

I'm reading a CSV file that has some non US-ASCII characters. I want
to parse each value in each row and strip out any leading/lagging
potential whitespace.
However, when I come across some unusual characters, I get invalid
byte sequence in UTF-8

here is an example:

irb(main):041:0* a = "\xFF"
=> "\xFF"
irb(main):042:0> a.encoding
=> #<Encoding:UTF-8>
irb(main):043:0> a.strip
ArgumentError: invalid byte sequence in UTF-8
        from (irb):43:in `strip'
        from (irb):43
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:44:in `start'
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:8:in `start'
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands.rb:23:in `<top (required)>'
        from script/rails:6:in `require'
        from script/rails:6:in `<main>'

# so now I'm going to try and change encoding, but this doesn't work
either

irb(main):044:0> a.encode!("ASCII-8BIT", undef: :replace)
Encoding::InvalidByteSequenceError: "\xFF" on UTF-8
        from (irb):44:in `encode!'
        from (irb):44
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:44:in `start'
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands/console.rb:8:in `start'
        from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.0.3/lib/
rails/commands.rb:23:in `<top (required)>'
        from script/rails:6:in `require'
        from script/rails:6:in `<main>'

Is there any way to strip out these characters while staying with
utf-8 encoding?

2

see Gray Soft / Not Found

eg,

require 'iconv'
#=> true
a="abcde \xFF ghi"
#=> "abcde \xFF ghi"
a.encoding
#=> #<Encoding:UTF-8>
ic = Iconv.new 'UTF-8//IGNORE', 'UTF-8'
#=> #<Iconv:0x944f468>
ic.iconv a
#=> "abcde ghi"

kind regards -botp

···

On Sat, Dec 11, 2010 at 12:40 PM, Sylvester T Cat <sylvestertcat1@gmail.com> wrote:

Is there any way to strip out these characters while staying with
utf-8 encoding?

3

Sylvester T Cat wrote in post #967790:

I'm reading a CSV file that has some non US-ASCII characters. I want
to parse each value in each row and strip out any leading/lagging
potential whitespace.
However, when I come across some unusual characters, I get invalid
byte sequence in UTF-8

I guess it's not genuinely UTF-8.

If you think it *is* sort of broken UTF-8 which includes FF characters
for some reason, then you could force encoding to binary, remove the FF
characters, then force back to UTF-8.

More likely I'd have thought it was a single-byte encoding (like
ISO-8859-1 perhaps). But in any case, if you're just doing CSV parsing,
you can quite legitimately treat UTF-8 as binary - since all you need to
do is recognise commas and double quotes, and the rest just gets passed
through.

More info at

Or just use ruby 1.8.

···

--
Posted via http://www.ruby-forum.com/\.