How to use the encoding option :universal_newline while working with File::new?

Arup_Rakshit1 · 27 January 2015 08:30

Hi,
If do use the option universal_newline with simple string object it works :
"foo \r \r\n".encode(universal_newline: true)
# => "foo \n \nNow, File::new documentation says it supports all options that String#encode does. That's why I tired the below code :-File.open("#{__dir__}/out.txt", universal_newline: true) do |file|
  file.each_line do |line|
    p line
  end
end

# gives output

# "I am a good boy \\r \\r\\n\n"
# "I am a good girl \\r\\n \\r\n"
As you can see, it didn't convert all the \r \r\n to \n only. It seems, I am using the option incorrect way. Can anyone just give a pointer how to use the option as I said above while dealing with File _encoding_ ? I applied the code to the text file :
I am a good boy \r \r\n
I am a good girl \r\n \r
Regards,
Arup Rakshit

Robert_K1 · 27 January 2015 15:24

Where exactly do you see that? When looking at

I see no such statement. When following the reference to

It does not mention universal_newline. What am I missing?

Kind regards

robert

···

On Tue, Jan 27, 2015 at 9:30 AM, Arup Rakshit <aruprakshit@rocketmail.com> wrote:

"foo \r \r\n".encode(universal_newline: true)
# => "foo \n \n

Now, File::new documentation says it supports all options that String#encode does. That's why I tired the below code :-

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can -
without end}
http://blog.rubybestpractices.com/

abinoam · 27 January 2015 18:38

Hi Robert Klemme,

I was luckier than you!

I have steped on...

I have no idea how to help on this!

Abinoam Jr.

···

On Tue, Jan 27, 2015 at 12:24 PM, Robert Klemme <shortcutter@googlemail.com> wrote:

On Tue, Jan 27, 2015 at 9:30 AM, Arup Rakshit <aruprakshit@rocketmail.com> > wrote:

"foo \r \r\n".encode(universal_newline: true)
# => "foo \n \n

Now, File::new documentation says it supports all options that
String#encode does. That's why I tired the below code :-

Where exactly do you see that? When looking at

Class: File (Ruby 2.2.0)

I see no such statement. When following the reference to

Class: IO (Ruby 2.2.0)

It does not mention universal_newline. What am I missing?

Kind regards

robert

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can -
without end}
http://blog.rubybestpractices.com/

Arup_Rakshit1 · 28 January 2015 01:56

Please look the doc : Class: IO (Ruby 2.2.0) and the end line :-

Also, `opt` can have same keys in `String#encode` for controlling conversion between the external encoding and the internal encoding.

···

On Tuesday, January 27, 2015 04:24:21 PM you wrote:

On Tue, Jan 27, 2015 at 9:30 AM, Arup Rakshit <aruprakshit@rocketmail.com> > wrote:

> "foo \r \r\n".encode(universal_newline: true)
> # => "foo \n \n
>
> Now, File::new documentation says it supports all options that String#encode does. That's why I tired the below code :-
>
>
Where exactly do you see that? When looking at

Class: File (Ruby 2.2.0)

I see no such statement. When following the reference to

Class: IO (Ruby 2.2.0)

It does not mention universal_newline. What am I missing?

Kind regards

robert

--

Regards,
Arup Rakshit

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

--Brian Kernighan

Robert_K1 · 28 January 2015 09:49

Thanks for helping my feeble eyes!

The explanation is pretty simple: the option is only effective if there is
an encoding conversion going on:

gist.github.com

https://gist.github.com/rklemme/9cc94b65df923dd9615e

encoding-line-break-test.rb

#!/usr/bin/ruby -w

FILE_NAME = "x"

printf "%-20s %p\n%-20s %p\n",
  "Default external", Encoding.default_external,
  "Default internal", Encoding.default_internal

# p File.instance_methods.grep(/enc|opt/)

This file has been truncated. show original

output.txt

$ ./encoding-line-break-test.rb
Default external     #<Encoding:UTF-8>
Default internal     nil

external             #<Encoding:UTF-8>
internal             nil
UTF-8                "1\n"
UTF-8                "2\r3\rn4\n"

external             #<Encoding:UTF-8>

This file has been truncated. show original

If one thinks about it this is pretty logical.

Kind regards

robert

···

On Wed, Jan 28, 2015 at 2:56 AM, Arup Rakshit <aruprakshit@rocketmail.com> wrote:

Please look the doc :
http://www.ruby-doc.org/core-2.2.0/IO.html#method-c-new-label-IO+Encoding
and the end line :-

Also, `opt` can have same keys in `String#encode` for controlling
conversion between the external encoding and the internal encoding.

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can -
without end}
http://blog.rubybestpractices.com/

Arup_Rakshit1 · 28 January 2015 16:30

Thanks Robert.

Ohh! That's how I need to use. Then it seems I wouldn't be able to take the advantage of it. Actually I am trying to replace such characters for a big file:

text = File.open('file/to/path/foo.txt').read
text.gsub!(/\r\n?/, "\n")

What is the most efficient way to do this, without reading the whole file ?

···

On Wednesday, January 28, 2015 10:49:35 AM Robert Klemme wrote:

On Wed, Jan 28, 2015 at 2:56 AM, Arup Rakshit <aruprakshit@rocketmail.com> > wrote:

> Please look the doc :
> Class: IO (Ruby 2.2.0)
> and the end line :-
>
> Also, `opt` can have same keys in `String#encode` for controlling
> conversion between the external encoding and the internal encoding.
>

Thanks for helping my feeble eyes!

The explanation is pretty simple: the option is only effective if there is
an encoding conversion going on:

encoding-line-break-test.rb · GitHub

robert

--

Regards,
Arup Rakshit

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

--Brian Kernighan

abinoam · 28 January 2015 20:28

Hi Arup,

text = File.open('file/to/path/foo.txt').read
text.gsub!(/\r\n?/, "\n")

What is the most efficient way to do this, without reading the whole file ?

Dear Robert Klemme in person has told us some time ago!

"Use File.foreach for large files"

http://blog.rubybestpractices.com/posts/rklemme/001-Using_blocks_for_Robustness.html

Abinoam Jr.

Robert_K1 · 29 January 2015 08:46

Hi Arup,

> text = File.open('file/to/path/foo.txt').read
> text.gsub!(/\r\n?/, "\n")
>
> What is the most efficient way to do this, without reading the whole
file ?

Dear Robert Klemme in person has told us some time ago!

:-))

"Use File.foreach for large files"

http://blog.rubybestpractices.com/posts/rklemme/001-Using_blocks_for_Robustness.html

In this case you could also work with a fixed buffer, i.e. use IO#read.

But even more efficient is probably this:

$ dos2unix your-large-file

Kind regards

robert

···

On Wed, Jan 28, 2015 at 9:28 PM, Abinoam Jr. <abinoam@gmail.com> wrote:

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can -
without end}
http://blog.rubybestpractices.com/

Topic		Replies	Views
Newline conversion on reading? ruby-talk	3	111	31 October 2012
File.new and encoding ruby-talk	4	82	29 November 2005
Copy file into new without dups, eol problem ruby-talk	8	184	30 March 2012
Ruby 1.9.1: Encoding trouble: broken US-ASCII String ruby-talk	21	199	16 December 2008
1.8.7 String#lines keeps new-line chars (say it ain't so in 1.9) ruby-talk	17	159	24 August 2009

How to use the encoding option :universal_newline while working with File::new?

--

Regards, Arup Rakshit

--

Regards, Arup Rakshit

Related topics

Regards,
Arup Rakshit

Regards,
Arup Rakshit