An issue piping to "head"

ruby-talk
1

Hi Rubysts !

I found this little issue, I woud like to know what you think about it.

This simple little script

···

-----------
ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read; puts s ' | head -n 1
------------
gives error :
"`write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)"

I never saw errors coming out from "head".
As far as i understand "head" is closing the "pipe" and "Ruby" complains
it has not where to write to.

In my opinion an error should not be given, in line with what
happens in other Unix tools as
#> cat foofile | head

Bye
Nicola

2

Might be related to your not closing the file. Try either closing it
explicitly, or using a block.

···

On Sat, Dec 16, 2017 at 10:42 AM, Nicola Mingotti <nmingotti@gmail.com> wrote:

This simple little script
-----------
ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read; puts
s ' | head -n 1
------------
gives error :
"`write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)"

--
Dave Aronson, Software Development Consultant
T. Rex of Codosaurus, LLC (http://www.Codosaur.us)
What can we evolve for you today?

3

Umm, no Dave, the problem is not there

I tried:
----------- these both give the same error --------------
[1] ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read; puts s; f.close ' | head -n 1

[2] ruby -e 'File.open("./datiTLX/aggregatoTLX.csv", "r") { |f| s = f.read; puts s } ' | head -n 1

···

----------------------------------------------------------

then, out of curiosity i tried to make a dumb file to repeat the experiment:
-------- this works: does not give out an error ------------------
ruby -e '100.times { puts "hello world" } ' > hellos.txt
ruby -e 'File.open("hellos.txt", "r") { |f| s = f.read; puts s } ' | head -n 1
----------------------------------------------------------------------------

Since my file "aggregatoTXT.csv" is not secret data I can give you
so you can try as well, it is here:
http://euriscom.it/aggregatoTLX.csv

Bye
Nicola

On 16/12/2017 16:47, Dave Aronson wrote:

On Sat, Dec 16, 2017 at 10:42 AM, Nicola Mingotti <nmingotti@gmail.com> wrote:

This simple little script
-----------
ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read; puts
s ' | head -n 1
------------
gives error :
"`write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)"

Might be related to your not closing the file. Try either closing it
explicitly, or using a block.

4

Quoting Nicola Mingotti (nmingotti@gmail.com):

Umm, no Dave, the problem is not there

I tried:
----------- these both give the same error --------------
[1] ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read;
puts s; f.close ' | head -n 1

[2] ruby -e 'File.open("./datiTLX/aggregatoTLX.csv", "r") { |f| s = f.read;
puts s } ' | head -n 1
----------------------------------------------------------

then, out of curiosity i tried to make a dumb file to repeat the experiment:
-------- this works: does not give out an error ------------------
ruby -e '100.times { puts "hello world" } ' > hellos.txt
ruby -e 'File.open("hellos.txt", "r") { |f| s = f.read; puts s } ' | head
-n 1
----------------------------------------------------------------------------

Since my file "aggregatoTXT.csv" is not secret data I can give you
so you can try as well, it is here:
http://euriscom.it/aggregatoTLX.csv

It's a question of size. Cut your size to 655 lines:

head -n 655 aggregatoTLX.csv > a.csv

and this new file will not give the message. With 656 lines, the
message is there (at least on my machine). What happens is that head
is able to parse the first line, print it and exit before the ruby
script has finished writing to its stdout.

The message can be safely ignored like this:
ruby -e 'f = File.open("./datiTLX/aggregatoTLX.csv", "r"); s = f.read; puts s; f.close ' 2>/dev/null | head -n 1

but I would just get completely rid of 'head':

ruby -e 'f=File.open("./datiTLX/aggregatoTLX.csv","r");puts f.readlines[0];f.close'

Carlo

···

Subject: Re: An issue piping to "head"
  Date: sab 16 dic 17 05:31:56 +0100

--
  * Se la Strada e la sua Virtu' non fossero state messe da parte,
* K * Carlo E. Prelz - fluido@fluido.as che bisogno ci sarebbe
  * di parlare tanto di amore e di rettitudine? (Chuang-Tzu)

5

On my phone.

How is opening a file relevant? The error is related to the pipe right? Can
you reproduce simply with puts “foo\nbar\nbaz” * 100 ?

···

--
Sent from Gmail Mobile

6

Yes Carlo, indeed that was also my hypothesis.

In my opinion that should be changed in Ruby. Because it is not the expected behaviour
of a command in a shell pipe line.

Even if It is not so a big issue in general to have an un-interesting
error message, it can be annoying. Imagine i have this stuff in a Cron line,
than annoying error message would be mailed to me (boring^2).

In conclusion, I don't think this is an error to send to STDERR.
But before mailing the issue to ruby-core-dev-people I would like to
know if you agree with me this is a legitimate and useful change
to ask.

Bye
Nicola

···

On 16/12/2017 18:48, Carlo E. Prelz wrote:

and this new file will not give the message. With 656 lines, the
message is there (at least on my machine). What happens is that head
is able to parse the first line, print it and exit before the ruby
script has finished writing to its stdout.

7

Quoting Nicola Mingotti (nmingotti@gmail.com):

>and this new file will not give the message. With 656 lines, the
>message is there (at least on my machine). What happens is that head
>is able to parse the first line, print it and exit before the ruby
>script has finished writing to its stdout.

Yes Carlo, indeed that was also my hypothesis.

In my opinion that should be changed in Ruby. Because it is not the expected
behaviour
of a command in a shell pipe line.

Even if It is not so a big issue in general to have an un-interesting
error message, it can be annoying. Imagine i have this stuff in a Cron line,
than annoying error message would be mailed to me (boring^2).

In conclusion, I don't think this is an error to send to STDERR.
But before mailing the issue to ruby-core-dev-people I would like to
know if you agree with me this is a legitimate and useful change
to ask.

Well, you must think in perspective of the task you are giving to the
interpreter. You are asking Ruby to write out a bunch of bytes to
stdout regardless. It tries to do that, but the unit it writes to
(which is managed by the shell, thus completely out of control from
its point of view) is suddenly slammed close before it can complete
its task.

In the vast majority of cases in which this sort of circumstance takes
place, I would be quite upset if the Ruby interpreter did not draw my
attention to the event. Which it does by raising an exception.

In special cases, you could always write a little more complex script,
that would catch the exception. For example:

f=File.open("./aggregatoTLX.csv", "r")
s=f.read
begin
  puts s
rescue Errno::EPIPE
  # Here take whatever measure you wish, or simply do nothing...
end

Carlo

···

Subject: Re: An issue piping to "head"
  Date: sab 16 dic 17 07:09:27 +0100

On 16/12/2017 18:48, Carlo E. Prelz wrote:

--
  * Se la Strada e la sua Virtu' non fossero state messe da parte,
* K * Carlo E. Prelz - fluido@fluido.as che bisogno ci sarebbe
  * di parlare tanto di amore e di rettitudine? (Chuang-Tzu)

8

True,

But we can check if the STDOUT is bound to a pipe and
change the Ruby behaviour accordingly.

On the other side, i checked in the good old Stevens book
"Advanced Programming in the Unix Environment"
and it states that
----- When one end of the pipe is closed ---
1. ..
2. If we write to a pipe whose read end has been closed
the Signal SIGPIPE is generated....

···

On 16/12/2017 19:31, Carlo E. Prelz wrote:

Well, you must think in perspective of the task you are giving to the
interpreter. You are asking Ruby to write out a bunch of bytes to
stdout regardless. It tries to do that, but the unit it writes to
(which is managed by the shell, thus completely out of control from
its point of view) is suddenly slammed close before it can complete
its task.

-----

=> Ruby behaviour is expected, what i want is out of the
standard => not a good idea => I dismissi it :slight_smile:

   puts s
rescue Errno::EPIPE
   # Here take whatever measure you wish, or simply do nothing...

True as well.

I have a final question, how can i check if STDOUT is bound to
a pipe in Ruby ?

I found "STDOUT.isatty" on StackOverflow but it is not exactly what I want,
According to Stevens book I should check "S_ISFIFO" in C.

Bye
Nicola

9

Quoting Nicola Mingotti (nmingotti@gmail.com):

I have a final question, how can i check if STDOUT is bound to
a pipe in Ruby ?

I found "STDOUT.isatty" on StackOverflow but it is not exactly what I want,
According to Stevens book I should check "S_ISFIFO" in C.

File::Stat::ftype returns the string 'fifo' in that case:

$ ruby -e "puts(\"Stdout is a #{STDOUT.stat.ftype}\")"
Stdout is a characterSpecial
$ ruby -e "puts(\"Stdout is a #{STDOUT.stat.ftype}\")" | tail -1
Stdout is a fifo

Carlo

···

Subject: Re: An issue piping to "head"
  Date: sab 16 dic 17 08:35:19 +0100

--
  * Se la Strada e la sua Virtu' non fossero state messe da parte,
* K * Carlo E. Prelz - fluido@fluido.as che bisogno ci sarebbe
  * di parlare tanto di amore e di rettitudine? (Chuang-Tzu)

10

Thansk a lot for the fruitful discussion !

Good evening
Nicola

···

On 16/12/17 20:55, Carlo E. Prelz wrote:

  Subject: Re: An issue piping to "head"
  Date: sab 16 dic 17 08:35:19 +0100

Quoting Nicola Mingotti (nmingotti@gmail.com):

I have a final question, how can i check if STDOUT is bound to
a pipe in Ruby ?

I found "STDOUT.isatty" on StackOverflow but it is not exactly what I want,
According to Stevens book I should check "S_ISFIFO" in C.

File::Stat::ftype returns the string 'fifo' in that case:

$ ruby -e "puts(\"Stdout is a #{STDOUT.stat.ftype}\")"
Stdout is a characterSpecial
$ ruby -e "puts(\"Stdout is a #{STDOUT.stat.ftype}\")" | tail -1
Stdout is a fifo

Carlo

11

In this case there is a shorter solution with rescue

$ seq 1 1000000 >|x
$ wc x
1000000 1000000 6888896 x
$ ruby -e 'f = File.open("x", "r"); s = f.read; puts s' | head -n 1
1
-e:1:in `write': Broken pipe @ io_write - <STDOUT> (Errno::EPIPE)
from -e:1:in `puts'
from -e:1:in `puts'
from -e:1:in `<main>'
$ ruby -e 'f = File.open("x", "r"); s = f.read; puts s rescue nil' | head -n 1
1

One more remark: many tools will report the same error if cutting off
their output prematurely. Others just silently ignore it.

Cheers

robert

12

Thank you for your contribution Robert,

$ ruby -e 'f = File.open("x", "r"); s = f.read; puts s rescue nil' | head -n 1
1

*] Carlo also proposed to catch the exception, that is interesting,
I will take this into consideration for future.

*] In the end, my idea was not good because against the Unix standard
(I sent a reference about it in a previous mail in this thread) so I abandoned it.

One more remark: many tools will report the same error if cutting off
their output prematurely. Others just silently ignore it.

*] Yes, and this also depends of buffering, indeed with
different file-sizes also Ruby has been shown not to complain,
without catching the exception.

*] I have seen you use the ">|" redirection, that is
interesting, I red about it in the Bash manual and I am quite
preplexed. ">" has always worked to me without setting
the "noclobber" option !! Mistery or bug;)

Bye
Nicola

13

to list as well

···

---------- Forwarded message ----------
From: Robert Klemme <shortcutter@googlemail.com>
Date: Mon, Dec 18, 2017 at 10:18 AM
Subject: Re: An issue piping to "head"
To: Nicola Mingotti <nmingotti@gmail.com>

On Mon, Dec 18, 2017 at 9:31 AM, Nicola Mingotti <nmingotti@gmail.com> wrote:

Thank you for your contribution Robert,

You're welcome!

One more remark: many tools will report the same error if cutting off
their output prematurely. Others just silently ignore it.

*] Yes, and this also depends of buffering, indeed with
different file-sizes also Ruby has been shown not to complain,
without catching the exception.

Well, if the program has written all the content to the pipe's buffer
before the other side closes down there is no way to complain because
the sender does not learn of the premature closing.

*] I have seen you use the ">|" redirection, that is
interesting, I red about it in the Bash manual and I am quite
preplexed. ">" has always worked to me without setting
the "noclobber" option !! Mistery or bug;)

There is no mystery: the point is, you only need >| if "noclobber" is
set. The, quite obvious, purpose of that option is to avoid accidental
overwriting of files that you might still need.

Cheers

robert

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

14

to list as well

Didn't my prevoius messege go to the list as well ?
I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

*] I have seen you use the ">|" redirection, that is
interesting, I red about it in the Bash manual and I am quite
preplexed. ">" has always worked to me without setting
the "noclobber" option !! Mistery or bug;)

There is no mystery: the point is, you only need >| if "noclobber" is
set. The, quite obvious, purpose of that option is to avoid accidental
overwriting of files that you might still need.

You are Right !
I confirm, that was my bug in parsing the manual :wink:

Thanks

Nicola

···

On 18/12/2017 10:19, Robert Klemme wrote:

15

to list as well

Didn't my prevoius messege go to the list as well ?

Your's did, but my reply did not.

I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

Well, you can always look up your sent message in "Sent Mail" and
check the addressee.

I confirm, that was my bug in parsing the manual :wink:

:slight_smile:

Cheers

robert

···

On Mon, Dec 18, 2017 at 10:56 AM, Nicola Mingotti <nmingotti@gmail.com> wrote:

On 18/12/2017 10:19, Robert Klemme wrote:

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

16

It is what i usually do, but with this impractical "solution" I am never 100%
sure what i sent has been received and correctly placed in
the discussion thread.

For a while I will keep it this way, too busy to drop gmail and now :wink:

bye
Nicola

···

On 18/12/2017 14:35, Robert Klemme wrote:

I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

Well, you can always look up your sent message in "Sent Mail" and
check the addressee.

17

Nicola,

In gmail:
If you add yourself to mail “to:”,
you will send the mail also to yourself

thus

P.

⁂

···

to: other@mail.adress your@mail.adress
from: your@mail.adress

On 18 Dec 2017, at 17:50, Nicola Mingotti <nmingotti@gmail.com> wrote:

On 18/12/2017 14:35, Robert Klemme wrote:

I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

Well, you can always look up your sent message in "Sent Mail" and
check the addressee.

It is what i usually do, but with this impractical "solution" I am never 100%
sure what i sent has been received and correctly placed in
the discussion thread.

For a while I will keep it this way, too busy to drop gmail and now :wink:

bye
Nicola

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

18

Peter, ahahhaha, right, nice trick, thank you !
I make a test right now ...

···

On 18/12/2017 18:08, Peter Giltay wrote:

Nicola,

In gmail:
If you add yourself to mail “to:”,
you will send the mail also to yourself

thus
to: other@mail.adress your@mail.adress
from: your@mail.adress

P.

⁂

On 18 Dec 2017, at 17:50, Nicola Mingotti <nmingotti@gmail.com> wrote:

On 18/12/2017 14:35, Robert Klemme wrote:

I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

Well, you can always look up your sent message in "Sent Mail" and
check the addressee.

It is what i usually do, but with this impractical "solution" I am never 100%
sure what i sent has been received and correctly placed in
the discussion thread.

For a while I will keep it this way, too busy to drop gmail and now :wink:

bye
Nicola

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

19

Peter method works !

I reccomend it to other people using gmail.

bye
Nicola

···

On 20/12/2017 16:34, Nicola Mingotti wrote:

Peter, ahahhaha, right, nice trick, thank you !
I make a test right now ...

On 18/12/2017 18:08, Peter Giltay wrote:

Nicola,

In gmail:
If you add yourself to mail “to:”,
you will send the mail also to yourself

thus
to: other@mail.adress your@mail.adress
from: your@mail.adress

P.

⁂

On 18 Dec 2017, at 17:50, Nicola Mingotti <nmingotti@gmail.com> wrote:

On 18/12/2017 14:35, Robert Klemme wrote:

I am using gmail so I can't see easily what i post to the list,
I see only other people posts.
(I red somewhere this is a gmail "thing" )

Well, you can always look up your sent message in "Sent Mail" and
check the addressee.

It is what i usually do, but with this impractical "solution" I am never 100%
sure what i sent has been received and correctly placed in
the discussion thread.

For a while I will keep it this way, too busy to drop gmail and now :wink:

bye
Nicola

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

20

Robert, for whatever reason, Gmail is telling me *your* message is spam.

···

On Mon, Dec 18, 2017 at 2:06 PM, Robert Klemme <shortcutter@googlemail.com> wrote:

On Mon, Dec 18, 2017 at 5:50 PM, Nicola Mingotti <nmingotti@gmail.com> > wrote:
>
> On 18/12/2017 14:35, Robert Klemme wrote:

>> Well, you can always look up your sent message in "Sent Mail" and
>> check the addressee.
>
> It is what i usually do, but with this impractical "solution" I am never
> 100%
> sure what i sent has been received and correctly placed in
> the discussion thread.

Note, you can even look this up via this thread as GMail will group
accordingly.

> For a while I will keep it this way, too busy to drop gmail and now :wink:

I find it actually pretty usable. And the spam filter is also pretty
decent.

Cheers

robert

--
        Eric Christopherson