Kenneth Kalmer has brought up a HighLine issue and I'm trying to look into it. Oddly, it seems to happen when interacting with the Net::HTTP library. I've narrowed it done to the following example on my box:
$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true
Can anyone explain why $stdin is closed after the page read?
James Edward Gray II
$stdin isn't closed, its at the end of file. Use #closed? to test if an IO has been closed or not.
$ cat test.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.closed?
p $stdin.eof?
$ ruby test.rb
false # hit ^D to close $stdin on the sending side.
true
$
$ echo hi | ruby test.rb
false
···
On Jan 27, 2007, at 17:13, James Edward Gray II wrote:
Kenneth Kalmer has brought up a HighLine issue and I'm trying to look into it. Oddly, it seems to happen when interacting with the Net::HTTP library. I've narrowed it done to the following example on my box:
$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true
Can anyone explain why $stdin is closed after the page read?
--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
As Eric Hodel pointed out, EOF on a terminal device is detected by trying to read from
the device. It appears that *prior* to the creation of a thread, a read on a terminal
device will simply block waiting for some activity on the device; either actual data
or indication of an eof condition (someone typing control-d).
*After* a thread has been created, Ruby's green thread implementation kicks in. This
means that the read caused by IO#eof? on the terminal device is not allowed to block.
Instead it fails as an interrupted system call, which is interpreted by IO#eof? as an
end of file condition. This is true even if the main thread is the only running thread.
James' original example used Net::HTTP, which uses Timeout, which creates a thread to
gain control of code that may block (i.e., an HTTP connection to a remote host).
I don't know enough about Ruby's thread implementation to know if this is a bug or
a feature. My hunch is that IO#eof? needs to be fixed so that an interrupted system
call doesn't get interpreted as an end of file condition. Another change would be to
turn off the software interrupt that is used to multiplex IO activity by threads when
only the main thread is running.
Gary Wright
···
On Jan 27, 2007, at 8:13 PM, James Edward Gray II wrote:
Kenneth Kalmer has brought up a HighLine issue and I'm trying to look into it. Oddly, it seems to happen when interacting with the Net::HTTP library. I've narrowed it done to the following example on my box:
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves differently (stalls and prints false). Why does it not behave the same after that page read?
James Edward Gray II
···
On Jan 27, 2007, at 7:39 PM, Eric Hodel wrote:
$stdin isn't closed, its at the end of file. Use #closed? to test if an IO has been closed or not.
sounds quite reasonable. however, on my box:
harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?
harp:~ > ruby a.rb # and then type ctrl-d
true
harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
-a
···
On Wed, 31 Jan 2007 gwtmp01@mac.com wrote:
On Jan 27, 2007, at 8:13 PM, James Edward Gray II wrote:
Kenneth Kalmer has brought up a HighLine issue and I'm trying to look into it. Oddly, it seems to happen when interacting with the Net::HTTP library. I've narrowed it done to the following example on my box:
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
As Eric Hodel pointed out, EOF on a terminal device is detected by trying to
read from the device. It appears that *prior* to the creation of a thread,
a read on a terminal device will simply block waiting for some activity on
the device; either actual data or indication of an eof condition (someone
typing control-d).
*After* a thread has been created, Ruby's green thread implementation kicks
in. This means that the read caused by IO#eof? on the terminal device is
not allowed to block. Instead it fails as an interrupted system call, which
is interpreted by IO#eof? as an end of file condition. This is true even if
the main thread is the only running thread.
James' original example used Net::HTTP, which uses Timeout, which creates a
thread to gain control of code that may block (i.e., an HTTP connection to a
remote host).
I don't know enough about Ruby's thread implementation to know if this is a
bug or a feature. My hunch is that IO#eof? needs to be fixed so that an
interrupted system call doesn't get interpreted as an end of file condition.
Another change would be to turn off the software interrupt that is used to
multiplex IO activity by threads when only the main thread is running.
Gary Wright
--
we can deny everything, except that we have the possibility of being better.
simply reflect on that.
- the dalai lama
If there are other threads and the IO isn't readable, ruby will run those until they complete. Your thread immediately stopped.
Since the thread in your example has completed there's no threads to switch to, so the main thread blocks waiting for a character (on a BSD-ish libc). If there was a thread running, the main thread would remain blocked (as if it were blocked on getc(3)) and the other threads were running.
To figure out what your platform is doing use a tool that gives system-call information like ktrace.
···
On Jan 30, 2007, at 13:31, gwtmp01@mac.com wrote:
On Jan 27, 2007, at 8:13 PM, James Edward Gray II wrote:
Kenneth Kalmer has brought up a HighLine issue and I'm trying to look into it. Oddly, it seems to happen when interacting with the Net::HTTP library. I've narrowed it done to the following example on my box:
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
As Eric Hodel pointed out, EOF on a terminal device is detected by trying to read from the device. It appears that *prior* to the creation of a thread, a read on a terminal device will simply block waiting for some activity on the device; either actual data or indication of an eof condition (someone typing control-d).
*After* a thread has been created, Ruby's green thread implementation kicks in. This means that the read caused by IO#eof? on the terminal device is not allowed to block.
$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
require 'net/http'
require 'io/wait'
Net::HTTP.start('localhost', 80) do |http|
body = http.get('/').body
end
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$
For the first #eof? no data written, so Ruby waits until something's been flushed. I typed some text and hit return to flush the terminal's stdout (Ruby's $stdin). At the second #eof? no input on $stdin has been consumed, so Ruby doesn't need to check for #eof? again by waiting.
$ cat test.rb
puts "eof? %p" % $stdin.eof?
gets
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
aoeu
eof? false
aoeu
eof? false
···
On Jan 27, 2007, at 18:38, James Edward Gray II wrote:
On Jan 27, 2007, at 7:39 PM, Eric Hodel wrote:
$stdin isn't closed, its at the end of file. Use #closed? to test if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves differently (stalls and prints false). Why does it not behave the same after that page read?
--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
On my box:
lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?
lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686
So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just
notice that my ruby is compiled i486-linux -- that *may* have
something to do with it, but I don't think so.
Jacob Fugal
···
On 1/30/07, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
... on my box:
harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?
harp:~ > ruby a.rb # and then type ctrl-d
true
harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
My post was based on ruby 1.8.5 on Darwin 8.8.0 Power Macintosh
The anomaly does not occur with ruby-1.9 on the same OS/hardware.
Gary Wright
···
On Jan 30, 2007, at 5:35 PM, ara.t.howard@noaa.gov wrote:
On Wed, 31 Jan 2007 gwtmp01@mac.com wrote:
Here is a simpler example that illustrates the issue:
Thread.new { }
p $stdin.eof?
sounds quite reasonable. however, on my box:
harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?
harp:~ > ruby a.rb # and then type ctrl-d
true
harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
so looks to be a bsd/mac issue?
If there are other threads and the IO isn't readable, ruby will run those until they complete. Your thread immediately stopped.
I coded it that way on purpose to see if the issue depended on whether another thread was running. On my platform it doesn't matter. It also doesn't matter if the main thread joins the now dead thread. It is the initiation of
another thread that changes the behavior of IO#eof? on certain platforms. The two platforms that have been reported
are:
ruby 1.8.5 Linux 2.6.17-2-686 i686
ruby 1.8.5 Darwin 8.8.0 Power Macintosh (Mac OS X 10.4.8)
Since the thread in your example has completed there's no threads to switch to, so the main thread blocks waiting for a character (on a BSD-ish libc). If there was a thread running, the main thread would remain blocked (as if it were blocked on getc(3)) and the other threads were running.
But my point (and James' point) was that on my (his) platform it *doesn't* block when it logically should.
To figure out what your platform is doing use a tool that gives system-call information like ktrace.
I did that and came up with the simpler example in the process. If you run just:
p $stdin.eof?
Ruby blocks waiting for input. If you first create a thread
which immediately terminates (my simple example) then Ruby doesn't
block (ruby 1.8.4 on Mac OS X) instead the read call is interrupted
with a software interrupt, which I interpreted as being party of Ruby's
thread implementation.
Gary Wright
···
On Jan 30, 2007, at 6:17 PM, Eric Hodel wrote:
OK, that makes sense. However, why does moving that eof?() check below a Net::HTTP page fetch change this? In that case, why does Ruby not need for something to be flushed in that case and why is eof?() then +true+?
James Edward Gray II
···
On Jan 27, 2007, at 8:48 PM, Eric Hodel wrote:
On Jan 27, 2007, at 18:38, James Edward Gray II wrote:
On Jan 27, 2007, at 7:39 PM, Eric Hodel wrote:
$stdin isn't closed, its at the end of file. Use #closed? to test if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves differently (stalls and prints false). Why does it not behave the same after that page read?
$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
require 'net/http'
require 'io/wait'
Net::HTTP.start('localhost', 80) do |http|
body = http.get('/').body
end
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$
For the first #eof? no data written, so Ruby waits until something's been flushed.
strange:
[nrt@anchor ~]$ ruby -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.8.5 (2006-12-11 patchlevel 5000) [i686-linux]
true
[nrt@anchor ~]$ uname -srm
Linux 2.6.9-42.0.3.ELsmp i686
and:
harp:~ > ruby19 -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.9.0 (2006-07-26) [i686-linux]
true
harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686
this could get tricky to debug! none of my linux versions or ruby versions
show this...
just to be clear (to everyone): have you all compiled ruby from source? all
my installs are from source using nothing but '--prefix' given to ./configure.
if anyone has a package (cough) manager or vendor (cough cough) install i'd
reccomend compiling from source without any options and testing again.
kind regards.
-a
···
On Wed, 31 Jan 2007, Jacob Fugal wrote:
On my box:
lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?
lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686
So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just notice
that my ruby is compiled i486-linux -- that *may* have something to do with
it, but I don't think so.
--
we can deny everything, except that we have the possibility of being better.
simply reflect on that.
- the dalai lama
Thank you. I was beginning to think I was going crazy. 
James Edward Gray II
···
On Jan 30, 2007, at 5:40 PM, gwtmp01@mac.com wrote:
On Jan 30, 2007, at 6:17 PM, Eric Hodel wrote:
Since the thread in your example has completed there's no threads to switch to, so the main thread blocks waiting for a character (on a BSD-ish libc). If there was a thread running, the main thread would remain blocked (as if it were blocked on getc(3)) and the other threads were running.
But my point (and James' point) was that on my (his) platform it *doesn't* block when it logically should.
$stdin isn't closed, its at the end of file. Use #closed? to test if an IO has been closed or not.
Right, good point.
However, when I call eof?() before the Net::HTTP call it behaves differently (stalls and prints false). Why does it not behave the same after that page read?
$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$
For the first #eof? no data written, so Ruby waits until something's been flushed.
OK, that makes sense. However, why does moving that eof?() check below a Net::HTTP page fetch change this?
I don't see this behavior. With your original example I have to hit ^D to get a prompt back.
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
In that case, why does Ruby not need for something to be flushed
Because #eof? reads a character if feof(3) is not true and there's no data pending:
rb_io_eof(io)
VALUE io;
{
OpenFile *fptr;
int ch;
GetOpenFile(io, fptr);
rb_io_check_readable(fptr);
if (feof(fptr->f)) return Qtrue;
if (READ_DATA_PENDING(fptr->f)) return Qfalse;
READ_CHECK(fptr->f);
clearerr(fptr->f);
TRAP_BEG;
ch = getc(fptr->f);
^^^^^^^^^^^^^^^^^^^
Analysis of sampling pid 15640 every 10.000000 milliseconds
Call graph:
[...]
300 rb_io_eof
300 getc
[...]
in that case and why is eof?() then +true+?
#eof? will only be true when there's no more data to read.
···
On Jan 27, 2007, at 22:01, James Edward Gray II wrote:
On Jan 27, 2007, at 8:48 PM, Eric Hodel wrote:
On Jan 27, 2007, at 18:38, James Edward Gray II wrote:
On Jan 27, 2007, at 7:39 PM, Eric Hodel wrote:
--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
I LIT YOUR GEM ON FIRE!
this could get tricky to debug!
Bingo. It's actually a pretty significant problem for HighLine because after you do something like fetching a web page, HighLine believes $stdin to e closed and won't allow you to ask the user any more questions. I fear we will need to disable HighLine's eof?() check the work around this.
just to be clear (to everyone): have you all compiled ruby from source?
I compiled from source. I did --enable-pthread as well.
James Edward Gray II
···
On Jan 30, 2007, at 5:28 PM, ara.t.howard@noaa.gov wrote:
I just upgraded to Ruby 1.8.5 to see if it would make a difference, but I still don't get prompted for input and it still tells me $stdin is closed:
Firefly:~/Desktop$ ruby -v
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Firefly:~/Desktop$ cat stdin_eof_issue.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
Firefly:~/Desktop$ ruby stdin_eof_issue.rb
true
James Edward Gray II
···
On Jan 29, 2007, at 3:54 AM, Eric Hodel wrote:
On Jan 27, 2007, at 22:01, James Edward Gray II wrote:
OK, that makes sense. However, why does moving that eof?() check below a Net::HTTP page fetch change this?
I don't see this behavior. With your original example I have to hit ^D to get a prompt back.
$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]
Then it sounds like HighLine needs to check #closed? or continue reading until IOError. If #eof? returns true you've read everything you could up to now. Nothing prevents more data from appearing on the socket after you've checked for #eof?.
···
On Jan 30, 2007, at 19:08, James Edward Gray II wrote:
On Jan 30, 2007, at 5:28 PM, ara.t.howard@noaa.gov wrote:
this could get tricky to debug!
Bingo. It's actually a pretty significant problem for HighLine because after you do something like fetching a web page, HighLine believes $stdin to e closed and won't allow you to ask the user any more questions. I fear we will need to disable HighLine's eof?() check the work around this.
I'm sorry, I meant it still claims $stdin is at eof?().
James Edward Gray II
···
On Jan 29, 2007, at 12:20 PM, James Edward Gray II wrote:
I still don't get prompted for input and it still tells me $stdin is closed:
I can confirm on my machine:
lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ cat test.rb
require 'net/http'
require 'io/wait'
Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
lukfugl@hephaestus:~$ ruby test.rb
true
One note: the first time I ran it, I thought it was waiting for input
too, but it was really just a few second delay while fetching
http://www.ruby-lang.org/en/license.txt\.
Jacob Fugal
···
On 1/29/07, James Edward Gray II <james@grayproductions.net> wrote:
I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is [eof]:
But, as you pointed out, closed?() is not the same as eof?().
Watching for an IOError also isn't very reliable since HighLine's input stream may be anything from STDIN, to a Socket, with IO and StringIO in between.
For now, I've added a workaround so user code can disable HighLine's EOF checks as needed.
Thank to all for helping me to understand this issue.
James Edward Gray II
···
On Jan 31, 2007, at 12:30 AM, Eric Hodel wrote:
On Jan 30, 2007, at 19:08, James Edward Gray II wrote:
On Jan 30, 2007, at 5:28 PM, ara.t.howard@noaa.gov wrote:
this could get tricky to debug!
Bingo. It's actually a pretty significant problem for HighLine because after you do something like fetching a web page, HighLine believes $stdin to e closed and won't allow you to ask the user any more questions. I fear we will need to disable HighLine's eof?() check the work around this.
Then it sounds like HighLine needs to check #closed? or continue reading until IOError.