Basically, the to_yaml method seems to use some strange hex escape
sequences which do not correspond to ANSI, UTF-8 or windows-1252...
The funny part is that when I load the same string from YAML, it is
displayed correctly in the console. This would be fine, except that
when I tried to save it to a file the international characters are not
displayed properly (or better, they are converted to the corresponding
ANSI/UTF-8 characters). What's going on here? What encoding does
to_yaml use to escape international characters?
According to the docs it should be UTF-8, but apparently it is not.
IIRC the various YAML implementations in each language can choose
to output UTF-8, or unicode-escaped ASCII. I think a YAML implementation
has to be able to read either.
···
On 10/23/07, h3raLd <h3rald@gmail.com> wrote:
Hello,
I noticed some weird behavior when converting a string containing
international characters to YAML:
I figured out the other half, unfortunately: the reason why I can't
view the characters in ANSI or UTF8 is because I'm inputting from DOS,
which means, unfortunately, "Code Page 437" (http://en.wikipedia.org/
wiki/Code_page_437).
···
On Oct 23, 3:05 pm, "Luis Parravicini" <lparr...@gmail.com> wrote:
On 10/23/07, h3raLd <h3r...@gmail.com> wrote:
> I noticed some weird behavior when converting a string containing
> international characters to YAML:
é gets translated to \202
è gets translated to \212
à gets translated to \205
ç gets translated to \207
â gets translated to \203
ê gets translated to \210
î gets translated to \214
ô gets translated to \223
û gets translated to \226
ä gets translated to \204
ë gets translated to \211
ï gets translated to \213
ö gets translated to \224
ù gets translated to \227
But why?
The app I am working on gets its data from different sources (yaml
files, dBaseIV files, MS Access files) and then produces xml files (via
builder).
When using print you get the original character. When using p, you get
the escaped equivalent.
And that's only the start of your problems! When trying to get those
characters into utf-8
é gets translated to \202 that then gets translated to ‚
è gets translated to \212 that then gets translated to Š
à gets translated to \205 that then gets translated to …
ç gets translated to \207 that then gets translated to ‡
â gets translated to \203 that then gets translated to ƒ
ê gets translated to \210 that then gets translated to ˆ
î gets translated to \214 that then gets translated to Œ
ô gets translated to \223 that then gets translated to “
û gets translated to \226 that then gets translated to –
ä gets translated to \204 that then gets translated to „
ë gets translated to \211 that then gets translated to ‰
ï gets translated to \213 that then gets translated to ‹
ö gets translated to \224 that then gets translated to ”
ù gets translated to \227 that then gets translated to —
Does someone have an explanation?
Does anyone know how to get those characters into the final xml files?
Any help would be greatly appreciated.
Jamal
Luis Parravicini wrote:
···
On 10/23/07, h3raLd <h3rald@gmail.com> wrote:
=> "test \225\227\212"
\225\227\212 is the same as \x95\x97\x8A, the former in octal, and the
latter in hex.
The app I am working on gets its data from different sources (yaml
files, dBaseIV files, MS Access files) and then produces xml files (via
builder).
When using print you get the original character. When using p, you get
the escaped equivalent.
And that's only the start of your problems! When trying to get those
characters into utf-8
...
Does someone have an explanation?
Does anyone know how to get those characters into the final xml files?
Any help would be greatly appreciated.
Jamal
In short, you're asking what the difference between "\303\251", "é",
and "‚" are.
The first is an octal sequence embedded in a string (it happens to be the
same as utf-8 'é'). The second is also utf-8 'é'. These two are the same
string ("\303\251" == "é"). The last, '‚' is the html-escaped notation
for a 'é' (I'm trusting your email for the correct number here). That is,
literally "‚" != "é", but they should render the same to a browser
capable of displaying utf-8.
é gets translated to \202
è gets translated to \212
à gets translated to \205
ç gets translated to \207
â gets translated to \203
ê gets translated to \210
î gets translated to \214
ô gets translated to \223
û gets translated to \226
ä gets translated to \204
ë gets translated to \211
ï gets translated to \213
ö gets translated to \224
ù gets translated to \227
But why?
I guess that your understanding is just wrong. I'm not really sure from
where your program gets those accented chars that are translated to
those specific escaped octal sequences. But if you're specifying them in
string constants in your program, then it all depends on according to
what encodig your editor displays it.
For instance, I usually edit my scripts as UTF-8 text files, and I treat
my sting constants that way too. In that case, if I put an é in a string
constant, it gets interpreted as \303\251, and not as \202. It's just
the octal representation of the byte(s) your editor displays as a
specific accented character.
The program gets the accented characters from a dBaseIV file, a MS
Access File and some YAML files.
I use Komodo Edit as my editor and it does handle UTF-8 correctly.
I know! That's why I did not understand why I got \202. I do not know
which charset ruby uses to convert the characters. I tried iconv and
jcode but ended up with the same results. At first I thought it was
because of the library I used (builder for example). The only
explanation I found was on that python tutorial.
Thanks.
Jamal
mortee wrote:
···
Jamal Bengeloun wrote:
à gets translated to \205
ù gets translated to \227
But why?
I guess that your understanding is just wrong. I'm not really sure from
where your program gets those accented chars that are translated to
those specific escaped octal sequences. But if you're specifying them in
string constants in your program, then it all depends on according to
what encodig your editor displays it.
For instance, I usually edit my scripts as UTF-8 text files, and I treat
my sting constants that way too. In that case, if I put an é in a string
constant, it gets interpreted as \303\251, and not as \202. It's just
the octal representation of the byte(s) your editor displays as a
specific accented character.
Thanks a lot for your help. I thought I will be going mad with this. I
thought it had something to do with ruby being C based (I saw something
on the internet about the difference between Python and JPython and the
accented characters were encoded in UTF-8 and not html escaped).
What if the end rendering engine is not a browser (I checked and you're
absolutely right, it does work in a browser)? How to get true UTF-8
encoded characters instead of HTML escaped ones? I am using builder to
generate XML files from the data I get.
Thanks a lot for your explanation (it really did enlighten me) and your
help.
Jamal
Konrad Meyer wrote:
···
Quoth Jamal Bengeloun:
characters into utf-8
...
Does someone have an explanation?
Does anyone know how to get those characters into the final xml files?
Any help would be greatly appreciated.
Jamal
In short, you're asking what the difference between "\303\251", "é",
and "‚" are.
The first is an octal sequence embedded in a string (it happens to be
the
same as utf-8 'é'). The second is also utf-8 'é'. These two are the same
string ("\303\251" == "é"). The last, '‚' is the html-escaped
notation
for a 'é' (I'm trusting your email for the correct number here). That
is,
literally "‚" != "é", but they should render the same to a browser
capable of displaying utf-8.
Jamal Bengeloun wrote:
Thanks a lot for your help. I thought I will be going mad with this. I
thought it had something to do with ruby being C based (I saw something
on the internet about the difference between Python and JPython and the
accented characters were encoded in UTF-8 and not html escaped).
What if the end rendering engine is not a browser (I checked and you're
absolutely right, it does work in a browser)? How to get true UTF-8
encoded characters instead of HTML escaped ones? I am using builder to
generate XML files from the data I get.
Thanks a lot for your explanation (it really did enlighten me) and your
help.
Thanks a lot for your help. I thought I will be going mad with this. I
thought it had something to do with ruby being C based (I saw something
on the internet about the difference between Python and JPython and the
accented characters were encoded in UTF-8 and not html escaped).
What if the end rendering engine is not a browser (I checked and you're
absolutely right, it does work in a browser)? How to get true UTF-8
encoded characters instead of HTML escaped ones? I am using builder to
generate XML files from the data I get.
Thanks a lot for your explanation (it really did enlighten me) and your
help.
Jamal
Konrad Meyer wrote:
> Quoth Jamal Bengeloun:
>> characters into utf-8
>>
>> ...
>>
>> Does someone have an explanation?
>>
>> Does anyone know how to get those characters into the final xml files?
>>
>> Any help would be greatly appreciated.
>>
>> Jamal
>
> In short, you're asking what the difference between "\303\251", "é",
> and "‚" are.
>
> The first is an octal sequence embedded in a string (it happens to be
> the
> same as utf-8 'é'). The second is also utf-8 'é'. These two are the same
> string ("\303\251" == "é"). The last, '‚' is the html-escaped
> notation
> for a 'é' (I'm trusting your email for the correct number here). That
> is,
> literally "‚" != "é", but they should render the same to a browser
> capable of displaying utf-8.
>
> HTH,
If I'm not mistaken, HTML and XML encoding is the same. So you're good for
those &#xxxxxx; chars.
The program gets the accented characters from a dBaseIV file, a MS
Access File and some YAML files.
I use Komodo Edit as my editor and it does handle UTF-8 correctly.
I know! That's why I did not understand why I got \202. I do not know
which charset ruby uses to convert the characters.
When converting some accented characters to \nnn then it doesn't use any
encoding. It just represents the verbatim non-ascii bytes it sees in the
string it gets. Encodig/decoding happens mainly when you input accented
chars on your keyboard, and they get converted to some byte (-sequence)
to be stored in a string, and when those strings are displayed, and they
are converted back to some printable characters.
Problems arise when the displaying code interprets the same string
according to a different charset than what it was encoded according to.
For example, when you puts a string in irb, then it's your terminal's
current charset which determines how the bytes in the string are
actually displayed. In contrast, when you use p (or, for that matter,
inspect), then non-ascii characters get ascaped as \nnn.
when the accented character comes from à YAML file, it is correctly HTML
encoded, however when it comes from the dBase file, it goes again
through:
é gets translated to \202 that then gets translated to ‚
è gets translated to \212 that then gets translated to Š
à gets translated to \205 that then gets translated to …
ç gets translated to \207 that then gets translated to ‡
â gets translated to \203 that then gets translated to ƒ
ê gets translated to \210 that then gets translated to ˆ
î gets translated to \214 that then gets translated to Œ
ô gets translated to \223 that then gets translated to “
û gets translated to \226 that then gets translated to –
ä gets translated to \204 that then gets translated to „
ë gets translated to \211 that then gets translated to ‰
ï gets translated to \213 that then gets translated to ‹
ö gets translated to \224 that then gets translated to ”
ù gets translated to \227 that then gets translated to —
[dBase > XML] é gets translated to \202 that then gets translated to
‚ (single low-9 quotation mark)
[YAML > XML] é gets translated to é
Thanks for your help!
Jamal
Jimmy Kofler wrote:
···
Jamal Bengeloun wrote:
Thanks a lot for your help. I thought I will be going mad with this. I
thought it had something to do with ruby being C based (I saw something
on the internet about the difference between Python and JPython and the
accented characters were encoded in UTF-8 and not html escaped).
What if the end rendering engine is not a browser (I checked and you're
absolutely right, it does work in a browser)? How to get true UTF-8
encoded characters instead of HTML escaped ones? I am using builder to
generate XML files from the data I get.
Thanks a lot for your explanation (it really did enlighten me) and your
help.
But at the end I simply used the ADODB wrapper to open my dbf file and I
did not have any character encoding problems after that.
Thanks a lot.
Jamal Abdou-Karim Bengeloun
mortee wrote:
···
When converting some accented characters to \nnn then it doesn't use any
encoding. It just represents the verbatim non-ascii bytes it sees in the
string it gets. Encodig/decoding happens mainly when you input accented
chars on your keyboard, and they get converted to some byte (-sequence)
to be stored in a string, and when those strings are displayed, and they
are converted back to some printable characters.
Problems arise when the displaying code interprets the same string
according to a different charset than what it was encoded according to.
For example, when you puts a string in irb, then it's your terminal's
current charset which determines how the bytes in the string are
actually displayed. In contrast, when you use p (or, for that matter,
inspect), then non-ascii characters get ascaped as \nnn.
I am not sure about that, I'll have to check. What I noticed though is
that builder converted the accented characters correctly when coming
from yaml files, but had problems (it did convert them but... See my
previous post) getting those coming from the dbf file to hit the target.
It seems that the problem was coming from the page code encoding.
Thanks
Jamal Abdou-Karim Bengeloun
Konrad Meyer wrote:
···
If I'm not mistaken, HTML and XML encoding is the same. So you're good
for
those &#xxxxxx; chars.