# Unicode string conversion **URL:** \ **Category:** ruby-talk\ **Created:** [5 May 2007 00:01 UTC](https://rubytalk.org/t/unicode-string-conversion/37405 "2007-05-05T00:01:07Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Alexandre\_Rosenfeld](https://avatars.discourse-cdn.com/v4/letter/a/9f8e36/32.png) [@Alexandre\_Rosenfeld](https://rubytalk.org/u/Alexandre_Rosenfeld)\ **Post date:** [5 May 2007 00:01 UTC](https://rubytalk.org/t/unicode-string-conversion/37405/1 "2007-05-05T00:01:07Z")
I'm reading a binary file in my program. It contains strings in the Windows Unicode format, which it says is stored as little-endian in the spefication. I'm loading it and trying to convert using Iconv, but I'm getting a invalid character exception, on any string. Now I'm just stripping the \000 character from it and it works, but I know it's not an ideal solution and it only works in some cases. So, how can I get the string in a format Ruby can understand? By the way, I'll load these string in GTK (with Ruby bindings), anyone knows if it can show Unicode strings? > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![James\_Britt](https://avatars.discourse-cdn.com/v4/letter/j/22d042/32.png) [@James\_Britt](https://rubytalk.org/u/James_Britt)\ **Post date:** [5 May 2007 04:50 UTC](https://rubytalk.org/t/unicode-string-conversion/37405/2 "2007-05-05T04:50:47Z")
Stripping the BOM? (byte order mark) Should be fine. Unicode works just as well w/ no BOM, actually better with no BOM. The first thing you should check for though is the presence of the BOM and read the BOM. > **···** > > On May 5, 2007, at 9:01 AM, Alexandre Rosenfeld wrote: > > > I'm reading a binary file in my program. It contains strings in the > > Windows Unicode format, which it says is stored as little-endian in the > > spefication. I'm loading it and trying to convert using Iconv, but I'm > > getting a invalid character exception, on any string. Now I'm just > > stripping the \000 character from it and it works, but I know it's not > > an ideal solution and it only works in some cases. > > So, how can I get the string in a format Ruby can understand? By the > > way, I'll load these string in GTK (with Ruby bindings), anyone knows if > > it can show Unicode strings? > > > > -- > > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Alexandre\_Rosenfeld](https://avatars.discourse-cdn.com/v4/letter/a/9f8e36/32.png) [@Alexandre\_Rosenfeld](https://rubytalk.org/u/Alexandre_Rosenfeld)\ **Post date:** [6 May 2007 14:05 UTC](https://rubytalk.org/t/unicode-string-conversion/37405/3 "2007-05-06T14:05:37Z")
John Joyce wrote: > **···** > > > On May 5, 2007, at 9:01 AM, Alexandre Rosenfeld wrote: > > > > > it can show Unicode strings? > > > > > > -- > > > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). > > > > Stripping the BOM? (byte order mark) > > Should be fine. Unicode works just as well w/ no BOM, actually better > > with no BOM. > > The first thing you should check for though is the presence of the > > BOM and read the BOM. > > There is no BOM. The specifications clearly states it "uses UTF-16, > little endian, and the Byte-Order Marker (BOM) character is not present" > > What I'm confused is to why Iconv coudlnt convert it. Does Iconv expects > for the BOM, even when I specify UTF16LE, which would make it explicit > the byte order? > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Nobuyoshi\_Nakada1](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nobuyoshi\_Nakada1](https://rubytalk.org/u/Nobuyoshi_Nakada1)\ **Post date:** [8 May 2007 23:51 UTC](https://rubytalk.org/t/unicode-string-conversion/37405/4 "2007-05-08T23:51:57Z")
Hi, At Sun, 6 May 2007 23:05:37 +0900, Alexandre Rosenfeld wrote in [ruby-talk:250503]: > What I'm confused is to why Iconv coudlnt convert it. Does Iconv expects > for the BOM, even when I specify UTF16LE, which would make it explicit > the byte order? BOM is a "ZERO WIDTH NON-BREAKING SPACE" at the beginning of a text. Almost iconv(3) should be possible to deal with it. Can't you show minimal data to reproduce the error? > **···** > > -- > Nobu Nakada