I was trying to convert UTF-8 content into a series of entities like
剛 so that whatever the page encoding is, the characters would
show...
so I used something like this:
<%
begin
t = ''
s = Iconv.conv("UTF-32", "UTF-8", some_utf8_string)
s.scan(/(.)(.)(.)(.)/) do |b1, b2, b3, b4|
t += ("&#x" + "%02X" % b3.ord) + ("%02X" % b4.ord) + ";"
end
rescue => details
t = "exception " + details
end
%>
<%= t %>
but some characters get converted, and some don't. Is it true that
(.)(.)(.)(.) will not necessarily match 4 bytes at a time?
At first, I was going to use
s = Iconv.conv("UTF-16", "UTF-8", some_utf8_string)
but then i found that utf-16 is also variable length... so I used UTF-32
instead which is fixed length. The UTF-8 string I have is just the
Basic Plane... so should be all in the 0x0000 to 0xFFFF range in
unicode.
--
Si tu veux construire un bateau ...
Ne rassemble pas des hommes pour aller chercher du bois, préparer des
outils, répartir les tâches, alléger le travail… mais enseigne aux
gens la nostalgie de l’infini de la mer.
If you want to build a ship, don’t herd people together to collect
wood and don’t assign them tasks and work, but rather teach them to
long for the endless immensity of the sea.
On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
sorry for a quite superficial answer, but can you use the unicode
switch for regexen in your Ruby Version. This seems to be the problem.
Robert
it really might be the 0 that is choking the regular expression match...
if i use
s.scan(/(.)(.)(.)(.)/s)
then it works better but still not all characters are converted...
but the way i have a solution using the byte processing ... in next post
On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
sorry for a quite superficial answer, but can you use the unicode
switch for regexen in your Ruby Version. This seems to be the problem.
Robert
by the way... Robert... what is the regexen? is it the regular
expression modifier? I'd like it to match absolutely anything
(newline, 0, etc)... but seems like there is no match
--
Posted via http://www.ruby-forum.com/\.
I'm pretty sure that Robert used regexen as the geeky way of pluralizing regex.
The unicode switch (a u regular expression option) forces the use of
unicode to interpret the string being matched, otherwise it uses
whatever the encoding of the source file containing the regular
expression.
e.g. /./u
If you want . to match newlines you want the m (multi-line) option.
Normally . will match anything BUT a new line, m changes this.
On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
Robert Dober wrote:
On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
sorry for a quite superficial answer, but can you use the unicode
switch for regexen in your Ruby Version. This seems to be the problem.
Robert
by the way... Robert... what is the regexen? is it the regular
expression modifier? I'd like it to match absolutely anything
(newline, 0, etc)... but seems like there is no match
On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
(newline, 0, etc)... but seems like there is no match
I'm pretty sure that Robert used regexen as the geeky way of pluralizing
regex.
The unicode switch (a u regular expression option) forces the use of
unicode to interpret the string being matched, otherwise it uses
whatever the encoding of the source file containing the regular
expression.
e.g. /./u
If you want . to match newlines you want the m (multi-line) option.
Normally . will match anything BUT a new line, m changes this.
aha... here i just want to match 4 bytes at a time, no matter what the
bytes are. Using "m" won't do it... the "u" would be helpful if i
match one UTF-8 character at a time and then process it... right now i
actually convert it all at once to UTF-32 and then process it... so I
wonder if there is a way to match 4 bytes at a time.
(newline, 0, etc)... but seems like there is no match
I'm pretty sure that Robert used regexen as the geeky way of pluralizing
regex.
The unicode switch (a u regular expression option) forces the use of
unicode to interpret the string being matched, otherwise it uses
whatever the encoding of the source file containing the regular
expression.
e.g. /./u
If you want . to match newlines you want the m (multi-line) option.
Normally . will match anything BUT a new line, m changes this.
aha... here i just want to match 4 bytes at a time, no matter what the
bytes are. Using "m" won't do it... the "u" would be helpful if i
match one UTF-8 character at a time and then process it... right now i
actually convert it all at once to UTF-32 and then process it... so I
wonder if there is a way to match 4 bytes at a time.
So what's the problem? A dot matches any byte (with the 'm' switch).
Make a regex with four dots:
/..../
or
/.{4}/
···
On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
the problem is that some characters are converted to the correct
骼 etc, but some characters are not... you can try if you want...
just go to Google News and get a China, taiwan, or hk news headline.
the problem is that some characters are converted to the correct
骼 etc, but some characters are not... you can try if you want...
just go to Google News and get a China, taiwan, or hk news headline.
Then why do you insist that you are trying to match any 4 bytes?
the problem is that some characters are converted to the correct
骼 etc, but some characters are not... you can try if you want...
just go to Google News and get a China, taiwan, or hk news headline.
Then why do you insist that you are trying to match any 4 bytes?
no... the program converts the UTF-8 string into UTF-32, so that each
character (code point) is 4 bytes long. And then the program process
the end result, 4 bytes at a time, so that's why scanning 4 bytes at a
time.
