Hi
I noticed that there a quite a few ruby c-extensions that use blocking read,
write and select calls without properly wrapping them with some
rb_thread_select loop.
This is annoying, because I leads to all ruby threads being blocked until
the read, write or select finishes.
I first noticed that when using the postgres module. This module actually
provides non-blocking functions too (using the non-blocking api postgres
provides). But e.g. the mysql module for ruby seems not to provide any
non-blocking methods for querying.
Of course this is especially bad for database interfaces, since queries
potentially take a very long time.
Of course one solutions would be to fix those extensions (if it’s possible -
I don’t know if there is a non-blocking api to mysql). But I though, that
maybe ruby could replace the read() and write() functions of the libc with
something that respects other ruby threads (by using a rb_thread_select
loop) before loading c-extensions.
If this is possible (I don’t see why it shouldn’t be, but I tend to oversee
subtle problems sometimes… 
), it would solve the blocking problem once
and for all. Or, if there might by some problems in special cases, the
function replacement could only be done when the extension is loaded via
require_fixup() instead of require(), or something like that.
If this hasn’t already been done, or someone has arguments why this cannot
work, I might try it out if I find some spare time…
greetings, Florian Pflug
I cannot distinguish if you are talking about mysql only problem ?
or a general problem with extensions ?
if this is mysql only, then why not propose it to the maintainer of
the mysql ruby extension ?
···
On Sun, 18 May 2003 12:55:51 +0900, Florian G. Pflug wrote:
Of course one solutions would be to fix those extensions (if it’s possible -
I don’t know if there is a non-blocking api to mysql). But I though, that
maybe ruby could replace the read() and write() functions of the libc with
something that respects other ruby threads (by using a rb_thread_select
loop) before loading c-extensions.
–
Simon Strandgaard
A similar problem can be seen with the built in Regexp module; find a
regexp which hangs the engine, and just try to make it time out.
timeout.rb doesn’t work, because; you guessed it - it uses threads,
which Regexp.match blocks.
The closest I can get is a quick C wrapper to alarm(2) and:
begin
Alarm.set(2)
trap(‘SIGALRM’) { Process.kill(“SIGINT”, 0) }
/#{str}/im.match(title)
rescue Interrupt
raise Damnit_I_Want_To_Use_Timeout_rb
end
Which makes me somewhat nervous – if I’m going to be letting users
specify their own regexp’s, I don’t want to use a hack to make sure they
can’t make it hang by entering something evil.
In the mean time, does anyone have any hints for filtering a user
specified regexp so it’s still mostly functional, but doesn’t provide
opertunities for hanging the interpreter or so? Maybe just removing
backtracking?
···
I noticed that there a quite a few ruby c-extensions that use blocking
read, write and select calls without properly wrapping them with some
rb_thread_select loop.
This is annoying, because I leads to all ruby threads being blocked
until the read, write or select finishes.
–
Thomas ‘Freaky’ Hurst - freaky@aagh.net - http://www.aagh.net/
Of course one solutions would be to fix those extensions (if it’s possible -
I don’t know if there is a non-blocking api to mysql). But I though, that
maybe ruby could replace the read() and write() functions of the libc with
something that respects other ruby threads (by using a rb_thread_select
loop) before loading c-extensions.
I cannot distinguish if you are talking about mysql only problem ?
or a general problem with extensions ?
I notices this problem while working with the ruby dbi. The postgres driver
for the dbi uses the “block-all-others” function of the ruby-postgres
extensions written in C, altough the postgres driver has “correct” functions
too. Then I started checking other modules, and found the mysql and oracle
modules to have the same problem.
if this is mysql only, then why not propose it to the maintainer of
the mysql ruby extension ?
I will do that, but:
.) Maybe mysql has no non-blocking api. Then the maintainer can’t fix this
either.
.) Since two out of four db modules I checked has this problem, I believe
this might affect other c-extensions that do read()s and write()s too…
Sometimes, one might just need a quick workaround for this, because getting
a maitainer to fix it can take some time.
Of course you are correct that the Right Thing™ is to fix the extensions.
But giving a user a chance to do a workaround, at least until the problem is
fixed, is not a bad thing IMHO.
That’s why I thought that a require_fixup function would be a nice thing to
have…
greetings, Florian Pflug
···
On Sun, May 18, 2003 at 10:50:13PM +0900, Simon Strandgaard wrote:
.) Maybe mysql has no non-blocking api. Then the maintainer can’t fix this
either.
Perhaps he can, TCPSocket class for example is non-blocking.
I understand this has something to do with rb_thread_select(), However
I’ve searched in
the archives and found no much details about it’s behaviour(intern.h
wasn’t very helpful
resource as well)
I might be willing to attempt patching the mysql extension myself, If
someone can pinpoint
to a source of information about rb_thread_select().
···
–
Idan
.) Maybe mysql has no non-blocking api. Then the maintainer can’t fix this
either.
Perhaps he can, TCPSocket class for example is non-blocking.
Yes, but there is a non-blocking api to file reads and writs in unix.
I understand this has something to do with rb_thread_select(), However
I’ve searched in
the archives and found no much details about it’s behaviour(intern.h
wasn’t very helpful
resource as well)
rb_thread_select offers the same functionality as unix select(), but lets
other threads run instead of blocking until one of the fd’s you select() on
becomes readable or writeable.
Using rb_thread_select in a ruby c-extesion has the same effect as using
normal select() in an non-ruby posix-threaded app.
I might be willing to attempt patching the mysql extension myself, If
someone can pinpoint
to a source of information about rb_thread_select().
Well, look at some extesions that use it (the async_* calls in the postgres
extension are a good example), and compare this to the documentation of the
select() syscall. That should give you a clear picture of whats going on.
···
On Mon, May 19, 2003 at 02:56:56AM +0900, Idan Sofer wrote: