Is there an event driven framework for ruby? Something similar to POE
in perl or twisted in python?
Chris
Actually, yes. I'm working on the Ruby/Event library. It's a C extension
for libevent. I built a small framework on top of this called Myriad
which is similar to twisted and POE. I used Myriad to write the SCGI
client/server which can host Ruby on Rails applications.
The framework is still pretty rough, but you can see the entire code for
the SCGI server at:
Also the performance so far is pretty damn good. I'll be posting some
metrics soon for it.
You can download the last version from:
I'm going to release a new version this week which fixes some memory
issues and has better documentation.
Feedback is welcome.
Zed
···
Is there an event driven framework for ruby? Something similar to POE
in perl or twisted in python?Chris
In article <1f060c4c05080909445a69e8f9@mail.gmail.com>,
snacktime <snacktime@gmail.com> writes:
Is there an event driven framework for ruby? Something similar to POE
in perl or twisted in python?
Why do you need an event driven framework?
Ruby has its own event driven mechanism to implement threads.
So, the thread mechanism should provide I/O multiplexing behaviour
similar to other event driven frameworks.
If you want event driven programming *style*, it doesn't help you,
though.
···
--
Tanaka Akira
Does this do all it's socket work in C, to bypass Ruby's Thread blocking issues?
James Edward Gray II
···
On Aug 9, 2005, at 12:23 PM, zedshaw@zedshaw.com wrote:
Actually, yes. I'm working on the Ruby/Event library. It's a C extension
for libevent. I built a small framework on top of this called Myriad
which is similar to twisted and POE. I used Myriad to write the SCGI
client/server which can host Ruby on Rails applications.
Hey Zed-
I have been using and abusing you scgi server with lightppd 1.3.15 for the last couple of weeks. It works great so far and performs very well. The few problems that I have had are as follows:
With lighttpd in production mode it fails with an ENOENT error when caching is turned on. It seems that it is having problems writing the cache files. If I restart the same exact app with webrick or plain lighttpd/fcgi the caching works fine and my permission setup is OK.
Also I have tried to set scgi up with apachge 1.3.33. Everything seems to work except the round trip back to apache. I mean I can request different URL's and I can see them get rendered in the develpment and production logs. But in the browser I just get an internal server error for any and all pages.
I really like how easy it was to set up and manage though. Its great to not have to restart the webserver to restart my app. I'd love to be able to use this in production but the production caching issues are the main hold up right now. If you want any stack traces or log files form some of these errors let me know and I am more than happy to be a tester and troubleshooter. I would rerally like to see this project get up and running.
Thanks for your hard work Zed-
-Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
509-577-7732
ezra@yakima-herald.com
···
On Aug 9, 2005, at 10:23 AM, zedshaw@zedshaw.com wrote:
Actually, yes. I'm working on the Ruby/Event library. It's a C extension
for libevent. I built a small framework on top of this called Myriad
which is similar to twisted and POE. I used Myriad to write the SCGI
client/server which can host Ruby on Rails applications.The framework is still pretty rough, but you can see the entire code for
the SCGI server at:Also the performance so far is pretty damn good. I'll be posting some
metrics soon for it.You can download the last version from:
I'm going to release a new version this week which fixes some memory
issues and has better documentation.Feedback is welcome.
Zed
Is there an event driven framework for ruby? Something similar to POE
in perl or twisted in python?Chris
James Edward Gray II <james@grayproductions.net> writes:
Actually, yes. I'm working on the Ruby/Event library. It's a C
extension
for libevent. I built a small framework on top of this called Myriad
which is similar to twisted and POE. I used Myriad to write the SCGI
client/server which can host Ruby on Rails applications.Does this do all it's socket work in C, to bypass Ruby's Thread
blocking issues?
What blocking issue?
In any case, ruby itself uses bsd's select which, in most
implementations, degrades linearly wrt number of sockets to be
monitored.
libevent allows you to take advantage of OS-specific event monitoring
interface that has better degradation characteristics than select()
under high load.
YS.
···
On Aug 9, 2005, at 12:23 PM, zedshaw@zedshaw.com wrote:
Depending on the libevent API used, all of the actual socket
reading/writing is either done with Ruby's Socket API, or it's done by
libevent. The only additional bit of "magic" I do is to turn off
blocking. Incidentally, none of this works with $stdout, $stdin, $stderr
and probably doesn't work with files. Most likely that's because Ruby is
messing with things it shouldn't be messing with <ehem>.
This all works because libevent only deals with the OS level event API and
the direct socket file descriptor (Socket.fileno). So, with non-blocking
sockets combined with libevent, we only access the Ruby Sockets API when
there's actually something to read.
Additionally, Myriad is layered on top of the libevent bufferevent
structure, so libevent actually does *all* of the IO and we just have to
read chunks of memory data. This gets around a ton of performance
bottle-necks and interference from Ruby Sockets.
I'll hopefully have a more detailed explanation when I release later this
week.
Zed A. Shaw
···
On Aug 9, 2005, at 12:23 PM, zedshaw@zedshaw.com wrote:
Actually, yes. I'm working on the Ruby/Event library. It's a C
extension
for libevent. I built a small framework on top of this called Myriad
which is similar to twisted and POE. I used Myriad to write the SCGI
client/server which can host Ruby on Rails applications.Does this do all it's socket work in C, to bypass Ruby's Thread
blocking issues?James Edward Gray II
Hey Ezra, this stuff is actually still in my queue of things to get done.
The ENOENT is bizarre, but if you have traces for it then send them on.
The Apache problem is mostly because I haven't pulled up the courage yet
to don the S&M suit and fight the Apache config file monster. 
Keep bugging me though.
···
Hey Zed-
I have been using and abusing you scgi server with lightppd
1.3.15 for the last couple of weeks. It works great so far and
performs very well. The few problems that I have had are as follows:
With lighttpd in production mode it fails with an ENOENT error when
caching is turned on. It seems that it is having problems writing the
cache files. If I restart the same exact app with webrick or plain
lighttpd/fcgi the caching works fine and my permission setup is OK.
Also I have tried to set scgi up with apachge 1.3.33. Everything
seems to work except the round trip back to apache. I mean I can
request different URL's and I can see them get rendered in the
develpment and production logs. But in the browser I just get an
internal server error for any and all pages.I really like how easy it was to set up and manage though. Its
great to not have to restart the webserver to restart my app. I'd
love to be able to use this in production but the production caching
issues are the main hold up right now. If you want any stack traces
or log files form some of these errors let me know and I am more than
happy to be a tester and troubleshooter. I would rerally like to see
this project get up and running.Thanks for your hard work Zed-
-Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
509-577-7732
ezra@yakima-herald.com
Writing a large chunk of data to a file, for example. Ruby will block until the write completes.
James Edward Gray II
···
On Aug 9, 2005, at 12:57 PM, Yohanes Santoso wrote:
James Edward Gray II <james@grayproductions.net> writes:
Does this do all it's socket work in C, to bypass Ruby's Thread
blocking issues?What blocking issue?
This project interests me a great deal! Thanks for taking the time to explain these details. I'll watch for the promised documentation and dig a little deeper at that time...
James Edward Gray II
···
On Aug 9, 2005, at 4:55 PM, zedshaw@zedshaw.com wrote:
Additionally, Myriad is layered on top of the libevent bufferevent
structure, so libevent actually does *all* of the IO and we just have to
read chunks of memory data. This gets around a ton of performance
bottle-necks and interference from Ruby Sockets.I'll hopefully have a more detailed explanation when I release later this
week.
James Edward Gray II <james@grayproductions.net> writes:
···
On Aug 9, 2005, at 12:57 PM, Yohanes Santoso wrote:
James Edward Gray II <james@grayproductions.net> writes:
Does this do all it's socket work in C, to bypass Ruby's Thread
blocking issues?What blocking issue?
Writing a large chunk of data to a file, for example. Ruby will
block until the write completes.
Disk I/O will always block the process executing it. only way to avoid
that is to have another process do the disk I/O (at least on linux,
*bsd; there used to have a unix that does not have this limitation,
but i can't remember the name). Using native thread also does not help
since the whole process is blocked.
YS.
Non-blocking IO is another solution.
James Edward Gray II
···
On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:
Disk I/O will always block the process executing it. only way to avoid
that is to have another process do the disk I/O (at least on linux,
*bsd; there used to have a unix that does not have this limitation,
but i can't remember the name). Using native thread also does not help
since the whole process is blocked.
Actually, the entire point of "native" threading is to allow blocking
actions to be performed in one thread while another runs -- hence their
usual applications in user interfaces which must stay responsive while
some blocking operation is happening, or in network servers where each
client is assigned a thread, so that socket reads and writes don't stop
the entire process.
It's true that Linux and *BSD did not always have the best support for
native threading, but at this point all the major distributions and
variants seem to have pretty much figured it out. However, since the
threading model was inconsistent from system to system, many developers
simply used the MIT pthreads implementation, which was a purely
user-space implementation, with the same limitations as Ruby's internal
threading model.
-Lennon
In article <8764uerk3f.fsf@dessyku.is-a-geek.org>,
Yohanes Santoso <ysantoso-rubytalk@dessyku.is-a-geek.org> writes:
Disk I/O will always block the process executing it. only way to avoid
that is to have another process do the disk I/O (at least on linux,
*bsd; there used to have a unix that does not have this limitation,
but i can't remember the name). Using native thread also does not help
since the whole process is blocked.
I think native (kernel level) thread doesn't block other threads.
% uname -a
Linux nute 2.6.8-2-686 #1 Thu May 19 17:53:30 JST 2005 i686 GNU/Linux
% cat t.c
#include <pthread.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define TMPFILE "/tmp/xx"
void *f1(void *arg)
{
struct stat s;
int i, ret;
for (i = 0; i < 10; i++) {
ret = stat(TMPFILE, &s);
if (ret == -1)
printf("%d: stat fail\n", i);
else
printf("%d: %x\n", i, s.st_size);
usleep(100000);
}
}
void *f2(void *arg)
{
#define SIZ 0x10000000
char *buf = malloc(SIZ);
int fd;
int ret;
if (buf == NULL) { perror("malloc"); return NULL; }
fd = open(TMPFILE, O_WRONLY|O_CREAT|O_TRUNC, 0666);
if (fd == -1) { perror("open"); return NULL; }
printf("begin write\n");
ret = write(fd, buf, SIZ);
printf("end write %x\n", ret);
return NULL;
}
int main()
{
pthread_t th1, th2;
unlink(TMPFILE);
pthread_create(&th1, NULL, f1, NULL);
pthread_create(&th2, NULL, f2, NULL);
pthread_join(th1, NULL);
pthread_join(th2, NULL);
return 0;
}
% gcc t.c -lpthread
% ./a.out
0: stat fail
begin write
1: 2143000
2: 41a3000
3: 61d9000
4: 8264000
5: a0e8000
6: c08e000
7: e0d3000
end write 10000000
8: 10000000
9: 10000000
···
--
Tanaka Akira
James Edward Gray II <james@grayproductions.net> writes:
···
On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:
Disk I/O will always block the process executing it. only way to avoid
that is to have another process do the disk I/O (at least on linux,
*bsd; there used to have a unix that does not have this limitation,
but i can't remember the name). Using native thread also does not help
since the whole process is blocked.Non-blocking IO is another solution.
James Edward Gray II
I know no POSIX system that honours O_NONBLOCK for disk I/O. That
includes linux, *bsd.
OTOH, AsyncIO is supposed to be a solution to this problem. But except
for win32, AsyncIO support on posix systems is spotty.
YS.
In article <84D5D729-3674-4084-A092-2359932E2FF4@grayproductions.net>,
James Edward Gray II <james@grayproductions.net> writes:
Non-blocking IO is another solution.
O_NONBLOCK is not always applicable.
* read from socket: works well by default. (O_NONBLOCK is not required.)
(UDP packet with wrong checksum on Linux 2.6 is an exception.)
* write to socket: O_NONBLOCK is required.
* read from pipe, console (and devices): works well by default on Unix.
may not work on other platforms.
* write to pipe, console (and devices): O_NONBLOCK is required on Unix.
may not work on other platforms.
* read from file: AIO or native thread is required. Ruby doesn't support yet.
* write to file: AIO or native thread is required. Ruby doesn't support yet.
* connect: works well by default. (Ruby use O_NONBLOCK internally.)
* accept: works well by default. (Old BSD platform require O_NONBLOCK?)
Note that Ruby 1.8.2 or former may lost data if O_NONBLOCK is set.
It is fixed by Ruby 1.9. Ruby 1.8.3 can avoid the problem by
IO#sync=true which is default for sockets.
Also note that some methods behave differently when O_NONBLOCK is
set. So O_NONBLOCK should be used carefully. I think O_NONBLOCK can
be usable more easily if Ruby provides blocking methods and
nonblocking methods. I proposed a method for nonblocking connect on
ruby-dev but matz doesn't accept because its name is not good enough,
though.
···
--
Tanaka Akira
Hello Yohanes,
James Edward Gray II <james@grayproductions.net> writes:
Disk I/O will always block the process executing it. only way to avoid
that is to have another process do the disk I/O (at least on linux,
*bsd; there used to have a unix that does not have this limitation,
but i can't remember the name). Using native thread also does not help
since the whole process is blocked.Non-blocking IO is another solution.
James Edward Gray II
I know no POSIX system that honours O_NONBLOCK for disk I/O. That
includes linux, *bsd.
OTOH, AsyncIO is supposed to be a solution to this problem. But except
for win32, AsyncIO support on posix systems is spotty.
It should work on linux even if the linux implementation is done
really bad by starting one thread for each async operation (using a
thread pool to speed this a little bit up) - at least this was how it
was implemented a few years ago. But the API is there, stable and
an offical POSIX conform standard.
···
On Aug 9, 2005, at 2:22 PM, Yohanes Santoso wrote:
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's
Tanaka Akira wrote:
In article <84D5D729-3674-4084-A092-2359932E2FF4@grayproductions.net>,
James Edward Gray II <james@grayproductions.net> writes:Non-blocking IO is another solution.
O_NONBLOCK is not always applicable.
* read from socket: works well by default. (O_NONBLOCK is not required.)
(UDP packet with wrong checksum on Linux 2.6 is an exception.)
Can you say a bit more about what happens differently on Linux 2.6 when
a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?
(I'm working with wireless UDP on linux 2.6, having migrated from QNX,
so it sounds important to me...)
···
--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
In article <42F97309.3040901@path.berkeley.edu>,
Joel VanderWerf <vjoel@path.berkeley.edu> writes:
* read from socket: works well by default. (O_NONBLOCK is not required.)
(UDP packet with wrong checksum on Linux 2.6 is an exception.)Can you say a bit more about what happens differently on Linux 2.6 when
a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?(I'm working with wireless UDP on linux 2.6, having migrated from QNX,
so it sounds important to me...)
Linux 2.6's select notify readability by a UDP packet with a bad checksum.
But read discards it and waits next packet with a correct checksum if
O_NONBLOCK is clear. If the next packet is not available, the read
hangs. Since O_NONBLOCK avoids the hang, it is possible to care the
problem in application level.
Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.html
Debian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yes
select(2):
Under Linux, select may report a socket file descriptor as "ready for
reading", while nevertheless a subsequent read blocks. This could for
example happen when data has arrived but upon examination has wrong
checksum and is discarded. There may be other circumstances. Thus it
may be safer to use O_NONBLOCK on sockets that should not block.
[ruby-talk:148436] Nonblocking Sockets
···
--
Tanaka Akira
Tanaka Akira wrote:
In article <42F97309.3040901@path.berkeley.edu>,
Joel VanderWerf <vjoel@path.berkeley.edu> writes:* read from socket: works well by default. (O_NONBLOCK is not required.)
(UDP packet with wrong checksum on Linux 2.6 is an exception.)Can you say a bit more about what happens differently on Linux 2.6 when
a UDP packet has a bad checksum? Is it a ruby behavior or a linux behavior?(I'm working with wireless UDP on linux 2.6, having migrated from QNX,
so it sounds important to me...)Linux 2.6's select notify readability by a UDP packet with a bad checksum.
But read discards it and waits next packet with a correct checksum if
O_NONBLOCK is clear. If the next packet is not available, the read
hangs. Since O_NONBLOCK avoids the hang, it is possible to care the
problem in application level.Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.htmlDebian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yesselect(2):
Under Linux, select may report a socket file descriptor as "ready for
reading", while nevertheless a subsequent read blocks. This could for
example happen when data has arrived but upon examination has wrong
checksum and is discarded. There may be other circumstances. Thus it
may be safer to use O_NONBLOCK on sockets that should not block.[ruby-talk:148436] Nonblocking Sockets
That post (148436) suggests that the native thread running the
interpreter is not blocked, and the only danger of blocking is in the
*ruby* thread reading the socket. Is that right? A bad UDP packet won't
stop other ruby threads?
I tried to hang ruby with
hping2 --udp -p 9999 -b
but the socket did not block.
···
--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407