# Gsub: invalid byte sequence in US-ASCII **URL:** \ **Category:** ruby-talk\ **Created:** [15 June 2010 09:29 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892 "2010-06-15T09:29:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![R\_Kumar](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@R\_Kumar](https://rubytalk.org/u/R_Kumar)\ **Post date:** [15 June 2010 09:29 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/1 "2010-06-15T09:29:06Z")
I download the page [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) using wget. Then i cat the file and pipe to gsub. I get: -e:1:in `gsub': invalid byte sequence in US-ASCII (ArgumentError) wget -q -k -O index11.html [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) cat index11.html | ruby -pe 'gsub(/href=a\/"/,"href=\"'${base}'")' \> ofile (The value of base is [http://www.ruby-forum.com/](http://www.ruby-forum.com/)) So what must i do so this command can run. It runs fine with another site. If i replace ruby with perl -pe 's|....|g' that works fine. I actually run this in a loop with various URLS from cron. > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [15 June 2010 15:56 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/2 "2010-06-15T15:56:21Z")
R.. Kumar wrote: > If i replace ruby with perl -pe 's|....|g' that works fine. Replacing ruby 1.9.x with ruby 1.8.x is just as effective, and I would recommend this for maintaining your sanity. I can only guess that the external encoding picked up from your platform's environment is US-ASCII (are you using cygwin by any chance?) You probably need to set the external encoding to UTF-8 or BINARY for your regexp not to crash. Try adding -Ku or -Kn to your ruby command line. If you want to attempt to understand String encoding in ruby 1.9, then good luck to you. I tried, documented what I found here: > and gave up after about 200 rules. There is no official documentation. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Caleb\_Clausen1](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@Caleb\_Clausen1](https://rubytalk.org/u/Caleb_Clausen1)\ **Post date:** [15 June 2010 21:06 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/3 "2010-06-15T21:06:44Z")
Handling this kind of thing right means tracking encodings right.... which means you'd have to extract the encoding from the http session and then mark the input as that encoding in your ruby script... and then deal with the inevitable incompatible encoding errors that would crop up. It sounds to me, tho, like in this case what you have a just some hacky little scripts and it would be acceptable for them to be imperfect. So, in that case, I suggest trying to set the encoding for your source file(s) to BINARY. That's a hack, but it ought to be effective. Alternately, you could drop back to the 1.8 interpreter, like Brian suggests, which more or less uses BINARY as the default source encoding. > **···** > > On 6/15/10, R.. Kumar \ wrote: > > > I download the page [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) using wget. Then i > > cat the file and pipe to gsub. > > > > I get: -e:1:in `gsub': invalid byte sequence in US-ASCII (ArgumentError) > > > > wget -q -k -O index11.html [http://www.ruby-forum.com/forum/4](http://www.ruby-forum.com/forum/4) > > > > cat index11.html | ruby -pe 'gsub(/href=a\/"/,"href=\"'${base}'")' \> > > ofile > > > > (The value of base is [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C)) > > > > So what must i do so this command can run. It runs fine with another > > site. > > If i replace ruby with perl -pe 's|....|g' that works fine. > > > > I actually run this in a loop with various URLS from cron. ---
**Author:** ![Bill\_Kelly](https://avatars.discourse-cdn.com/v4/letter/b/cc9497/32.png) [@Bill\_Kelly](https://rubytalk.org/u/Bill_Kelly)\ **Post date:** [15 June 2010 21:19 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/4 "2010-06-15T21:19:21Z")
Caleb Clausen wrote: > Handling this kind of thing right means tracking encodings right.... > which means you'd have to extract the encoding from the http session > and then mark the input as that encoding in your ruby script... and > then deal with the inevitable incompatible encoding errors that would > crop up. > > It sounds to me, tho, like in this case what you have a just some > hacky little scripts and it would be acceptable for them to be > imperfect. So, in that case, I suggest trying to set the encoding for > your source file(s) to BINARY. That's a hack, but it ought to be > effective. Additional info on the source, external, and internal encodings: > **[Gray Soft / Not Found](http://graysoftinc.com/articles/ruby_19s_three_default_encodings)** > > Something has gone wrong. For the OP, I'd expect `ruby -EBINARY ...` or `ruby -EASCII-8BIT ...` should work. Regards, Bill ---
**Author:** ![R\_Kumar](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@R\_Kumar](https://rubytalk.org/u/R_Kumar)\ **Post date:** [16 June 2010 04:20 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/5 "2010-06-16T04:20:16Z")
Brian Candler wrote: > R.. Kumar wrote: > > > If i replace ruby with perl -pe 's|....|g' that works fine. > > Replacing ruby 1.9.x with ruby 1.8.x is just as effective, and I would > recommend this for maintaining your sanity. > > I can only guess that the external encoding picked up from your > platform's environment is US-ASCII (are you using cygwin by any chance?) > > You probably need to set the external encoding to UTF-8 or BINARY for > your regexp not to crash. Try adding -Ku or -Kn to your ruby command > line. > > If you want to attempt to understand String encoding in ruby 1.9, then > good luck to you. I tried, documented what I found here: > [string19/string19.rb at master · candlerb/string19 · GitHub](http://github.com/candlerb/string19/blob/master/string19.rb) > and gave up after about 200 rules. There is no official documentation. 1. I have moved to 1.9 long back. Don't want to move back. 2. I am on OSX. I think I had \_probably\_ (?) solved this issue on my previous laptop (PPC) -- now I;ve migrated my user to a new machine (Snow Leopard). All my settings, should have moved. (I say this since I had commented out the perl line). LC\_ALL=en\_US.UTF-8 LC\_CTYPE=en\_US.UTF-8 LANG=C 3. Thanks for the link, i will read it. But NO, i have already read up enough a few months back, and do not have the energy to do it again :-(. Thanks for the tip on -Ku / -Kn > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![R\_Kumar](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@R\_Kumar](https://rubytalk.org/u/R_Kumar)\ **Post date:** [16 June 2010 04:31 UTC](https://rubytalk.org/t/gsub-invalid-byte-sequence-in-us-ascii/58892/6 "2010-06-16T04:31:52Z")
Brian Candler wrote: > I can only guess that the external encoding picked up from your > platform's environment is US-ASCII (are you using cygwin by any chance?) > > You probably need to set the external encoding to UTF-8 or BINARY for > your regexp not to crash. Try adding -Ku or -Kn to your ruby command > line. Ok, I've got it. The problem occured when the program was run by cron. My user setting is UTF and it ran fine in terminal. So now in the program itself I have set LC\_CTYPE and LC\_ALL to en\_US.UTF-8. Hopefully, it should work fine now. Thanks. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).