Well, yes, you'd have to implement the method (or something like it) yourself.
Note that I'm using a fifo file (created with "mkfifo file") so it is not
"stored" in the filesystem. Instead it's just a communication between two
processes at SO level via SO's buffers.
Yeah, I gathered that from your other posts. The general point, though, still applies: check the pipe's size, and if it grows too large, spin off a new reading thread.
That's something different than you proposed initially, isn't it? This approach (increasing the number of readers if the pipe fills too fast) is better because it regulates read performance according to load.
pipe.write unless pipe.full?
i.e. check if your pipe hits a set limit on disk, and generate an exception if the pipe_file reaches (or is close to reaching) the limit.
You could then buffer the data to be written until an additional (or new) reading thread has started.
IMHO this approach (local buffering if the pipe cannot be written to) is not really helping because the pipe *is* a buffer already. In other words, the same effect will happen - only later. The only argument in favor of additional buffering I can see is less lock contention: if every writer process has multiple threads that want to write to the buffer, they could instead write to a Queue internally and a single reader could read from that local queue and write to the global queue. That would reduce the number of writers that compete for locks on the global queue. Whether that is performant or not would need to be tested.
Nevertheless I would start with a simple solution, monitor its performance and change the implementation if it does not scale well enough. Often simple solutions work surprisingly well...
Yes, pipes allow atomic operation but just in case the message is less than 4
or 8 bytes, no more. In C it can be useful if you pass a pointer via pipe.
I tested it (in ruby) by sending strings of 10 bytes size from various
processes to a shared pipe. The reader gets the strings mixed
However this doesn't occur with posix message queues.
Regards.
···
El Domingo, 10 de Enero de 2010, Gary Wright escribió:
On Jan 9, 2010, at 12:54 PM, Iñaki Baz Castillo wrote:
> Posix message queues are just 20-40% slower than pipes in my benchmarks
> (but pipes are no multiprocess/thread safe).
I believe pipes can be used concurrently if reads and writes are less than
or equal to PIPE_BUF bytes. Is the size limitation the problem you were
hinting at or something else?
That's something different than you proposed initially, isn't it? This
approach (increasing the number of readers if the pipe fills too fast)
is better because it regulates read performance according to load.
A little refined (in that I skipped the buffering), but it's still the same core: check the pipe, and sin off new threads as needed.
IMHO this approach (local buffering if the pipe cannot be written to) is
not really helping because the pipe *is* a buffer already. In other
words, the same effect will happen - only later. The only argument in
favor of additional buffering I can see is less lock contention: if
every writer process has multiple threads that want to write to the
buffer, they could instead write to a Queue internally and a single
reader could read from that local queue and write to the global queue.
That would reduce the number of writers that compete for locks on the
global queue. Whether that is performant or not would need to be tested.
This might be a difference in interpretation: I see the pipe in this instance as a simple inter-process communication solution, not per se a buffer.
Otherwise: You are right.
Also in that performance would've to be tested, and the constraints have to be known (Iñaki already mentioned that getting all data is less important to him, so buffering wouldn't be strictly necessary, either).
Nevertheless I would start with a simple solution, monitor its
performance and change the implementation if it does not scale well
enough. Often simple solutions work surprisingly well...
Indeed. And it's easier to iterate from something simple, than to iterate from something complex, too.
Definitively I have no idea of how to know the status of a FIFO (not a IO pipe
but a FIFO file). The only it occurs when it's full (because no reader is
getting the data) is that the writer #flush operation gets blocked.
I've found no way to determine how "full" is a FIFO file.
···
El Jueves, 7 de Enero de 2010, Robert Klemme escribió:
> Yeah, I gathered that from your other posts. The general point, though,
> still applies: check the pipe's size, and if it grows too large, spin
> off a new reading thread.
That's something different than you proposed initially, isn't it? This
approach (increasing the number of readers if the pipe fills too fast)
is better because it regulates read performance according to load.
I just tried this on Mac OS X using threads and using forks to write to
a shared pipe. Each write was 256 bytes done using syswrite. All the
reads on the other end were: sysread(256). I tried using forked writers
and simple Ruby threads. Each writer wrote 10000 messages and I didn't
get any mixed up data in any of my tests. I also tried using plain
old write (vs. syswrite) and still didn't get any mixed up data.
The behavior you described just doesn't match my experience or
understanding of pipes.
Gary Wright
···
On Jan 10, 2010, at 11:17 AM, Iñaki Baz Castillo wrote:
I tested it (in ruby) by sending strings of 10 bytes size from various
processes to a shared pipe. The reader gets the strings mixed
However this doesn't occur with posix message queues.
FIFO are pipes, they just have a name on the filesystem.
In any case, use IO#write_nonblock. Any writes you do will raise
Errno::EAGAIN if your FIFO/pipe is full.
See the pipe(7) manpage on a Linux machine, it provides a great overview
of pipe semantics for blocking/non-blocking operations.
···
Iñaki Baz Castillo <ibc@aliax.net> wrote:
El Jueves, 7 de Enero de 2010, Robert Klemme escribió:
> > Yeah, I gathered that from your other posts. The general point, though,
> > still applies: check the pipe's size, and if it grows too large, spin
> > off a new reading thread.
>
> That's something different than you proposed initially, isn't it? This
> approach (increasing the number of readers if the pipe fills too fast)
> is better because it regulates read performance according to load.
Definitively I have no idea of how to know the status of a FIFO (not a IO pipe
but a FIFO file). The only it occurs when it's full (because no reader is
getting the data) is that the writer #flush operation gets blocked.
I've found no way to determine how "full" is a FIFO file.
I got the mixed strings using a name pipe (created with "mkfifo" command) and
shared by two different Ruby programs (a writer and a reader).
However I don't know if I've modified something since that test but the fact
is that I don't get the mixed strings...
···
El Lunes, 11 de Enero de 2010, Gary Wright escribió:
On Jan 10, 2010, at 11:17 AM, Iñaki Baz Castillo wrote:
> I tested it (in ruby) by sending strings of 10 bytes size from various
> processes to a shared pipe. The reader gets the strings mixed
> However this doesn't occur with posix message queues.
I just tried this on Mac OS X using threads and using forks to write to
a shared pipe. Each write was 256 bytes done using syswrite. All the
reads on the other end were: sysread(256). I tried using forked writers
and simple Ruby threads. Each writer wrote 10000 messages and I didn't
get any mixed up data in any of my tests. I also tried using plain
old write (vs. syswrite) and still didn't get any mixed up data.
The behavior you described just doesn't match my experience or
understanding of pipes.
El Jueves, 7 de Enero de 2010, Eric Wong escribió:
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Jueves, 7 de Enero de 2010, Robert Klemme escribió:
> > > Yeah, I gathered that from your other posts. The general point,
> > > though, still applies: check the pipe's size, and if it grows too
> > > large, spin off a new reading thread.
> >
> > That's something different than you proposed initially, isn't it? This
> > approach (increasing the number of readers if the pipe fills too fast)
> > is better because it regulates read performance according to load.
>
> Definitively I have no idea of how to know the status of a FIFO (not a IO
> pipe but a FIFO file). The only it occurs when it's full (because no
> reader is getting the data) is that the writer #flush operation gets
> blocked. I've found no way to determine how "full" is a FIFO file.
FIFO are pipes, they just have a name on the filesystem.
In any case, use IO#write_nonblock. Any writes you do will raise
Errno::EAGAIN if your FIFO/pipe is full.
See the pipe(7) manpage on a Linux machine, it provides a great overview
of pipe semantics for blocking/non-blocking operations.
Yes, I know, I just told it because I tested with named piped.
However it's true that I don't get the issue occuring when writting in a named
pipe from different processes....
···
El Lunes, 11 de Enero de 2010, Gary Wright escribió:
On Jan 10, 2010, at 8:00 PM, Iñaki Baz Castillo wrote:
> I got the mixed strings using a name pipe (created with "mkfifo" command)
> and shared by two different Ruby programs (a writer and a reader).
A named pipe and a pipe should behave the same, the only difference being
that one has a name in the filesystem.
