How to parse a unicode url?

I would really like to be able to do the following. Is this even
possible?

Thanks,
nerdytenor

uri = URI.parse('http://www.hören.com') # not a real url (that I know
of)
URI::InvalidURIError: bad URI(is not URI?): http://www.hören.com
        from /usr/lib/ruby/1.8/uri/common.rb:432:in `split'
        from /usr/lib/ruby/1.8/uri/common.rb:481:in `parse'
        from (irb):26

···

--
Posted via http://www.ruby-forum.com/.

Dan The man wrote:

I would really like to be able to do the following. Is this even
possible?

Thanks,
nerdytenor

uri = URI.parse('http://www.hören.com') # not a real url (that I know
of)
URI::InvalidURIError: bad URI(is not URI?): http://www.hören.com
        from /usr/lib/ruby/1.8/uri/common.rb:432:in `split'
        from /usr/lib/ruby/1.8/uri/common.rb:481:in `parse'
        from (irb):26

You can do this:

require "uri"

url = "http://www.hören.co
enc_url = URI.encode(url)
puts enc_url

to get this:

http://www.h%C3%B6ren.co

which according to wikipedia here:

is a legal uri. But when I do this:

require "uri"

url = "http://www.hören.co"
enc_url = URI.encode(url)
puts enc_url

uri = URI.parse(enc_url)

I get this:

http://www.h%C3%B6ren.co
/usr/lib/ruby/1.8/uri/generic.rb:194:in `initialize': the scheme http
does not accept registry part: www.h%C3%B6ren.co (or bad hostname?)
(URI::InvalidURIError)
        from /usr/lib/ruby/1.8/uri/http.rb:46:in `initialize'
        from /usr/lib/ruby/1.8/uri/common.rb:484:in `new'
        from /usr/lib/ruby/1.8/uri/common.rb:484:in `parse'
        from r3test.rb:7

which as far as I can tell means that URI.parse() is broken.

···

--
Posted via http://www.ruby-forum.com/\.

Dan The man wrote:

I would really like to be able to do the following. Is this even
possible?

Thanks,
nerdytenor

uri = URI.parse('http://www.hören.com') # not a real url (that I know
of)
URI::InvalidURIError: bad URI(is not URI?): http://www.hören.com
        from /usr/lib/ruby/1.8/uri/common.rb:432:in `split'
        from /usr/lib/ruby/1.8/uri/common.rb:481:in `parse'
        from (irb):26

There is no such thing as a Unicode URL. The RFC for URI and URL specify the charset as 7Bit ASCII AFAIK.

The legal form of that URL is this: http://www.hören.com/

See IDNA for details, for example:

Quick searching revealed this - maybe it can help:
http://rubyforge.org/pipermail/idn-discuss/2005-September/000000.html

You can do this:

require "uri"

url = "http://www.hören.co
enc_url = URI.encode(url)
puts enc_url

to get this:

http://www.h%C3%B6ren.co

which according to wikipedia here:

Percent-encoding - Wikipedia

is a legal uri. But when I do this:

require "uri"

url = "http://www.hören.co"
enc_url = URI.encode(url)
puts enc_url

uri = URI.parse(enc_url)

I get this:

http://www.h%C3%B6ren.co
/usr/lib/ruby/1.8/uri/generic.rb:194:in `initialize': the scheme http does not accept registry part: www.h%C3%B6ren.co (or bad hostname?) (URI::InvalidURIError)
        from /usr/lib/ruby/1.8/uri/http.rb:46:in `initialize'
        from /usr/lib/ruby/1.8/uri/common.rb:484:in `new'
        from /usr/lib/ruby/1.8/uri/common.rb:484:in `parse'
        from r3test.rb:7

which as far as I can tell means that URI.parse() is broken.

I don't think so. There are invalid characters in the domain name (as the exception indicates).

Kind regards

  robert

···

On 20.09.2007 07:19, 7stud -- wrote:

Robert Klemme wrote:

which as far as I can tell means that URI.parse() is broken.

I don't think so. There are invalid characters in the domain name (as
the exception indicates).

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

···

--
Posted via http://www.ruby-forum.com/\.

In article <aa11611ac34b5cc87350f239c546818f@ruby-forum.com>,

···

7stud -- <dolgun@excite.com> wrote:

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

They are but host & domain names do not accept Unicode characters at all and are limited to 7 bits ASCII. Search for IDN for more information.
--
Ollivier ROBERT -=- EEC/RIF/SEU -=-
Systems Engineering Unit

It says that what characters are valid for each piece of a URI is dependent on the URI scheme. The characters valid for the hostname part of the http URI scheme is goverened by the DNS system, so you need to use an IDN.

I believe there is a ruby wrapper for libidn.

···

On Sep 20, 2007, at 24:31 , 7stud -- wrote:

Robert Klemme wrote:

which as far as I can tell means that URI.parse() is broken.

I don't think so. There are invalid characters in the domain name (as
the exception indicates).

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

Ollivier Robert wrote:

In article <aa11611ac34b5cc87350f239c546818f@ruby-forum.com>,

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

They are but host & domain names do not accept Unicode characters at all
and are limited to 7 bits ASCII. Search for IDN for more information.

I thought this might be the case. However, typing the following into
firefox gets me a real live page (after trying a few random domains)

Hmmm...

···

7stud -- <dolgun@excite.com> wrote:

--
Posted via http://www.ruby-forum.com/\.

I thought this might be the case. However, typing the following into
firefox gets me a real live page (after trying a few random domains)

http://www.hören.at/

Hmmm...

That's because Firefox automatically translates into the equivalent IDN;
attempting to lookup the domain www.hören.at (or hören.at) gives
NXDOMAIN - it will never work. www.hören.at is translated into the
punycode name www.xn–hren-5qa.at. Try visiting that in Firefox - it
actually rewrote the URL as www.hören.at for me, but rest assured the
domain is one and the same.

7 bit ASCII is to be used in domain names only; please use IDN to
represent international characters, as it's the accepted way to do it,
and implemented as you see in Firefox.

Cheers.

Arlen