Worker pool - what to use?

ruby-talk
1

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker "frameworks" come with
their own queue... however a few concerns:

- does putting msgs from one into another queue make sense?
- how reliable is resque? I cannot lose mails
- how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this...
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue.. which is not really needed in this
scenario as I confirm messages on the OpenMQ..

I would appreciate any suggestions!
Thanks, Christoph

···

--
Posted via http://www.ruby-forum.com/.

2

<disclaimer>I don't know stomp and I have only rudimentary knowledge
of OpenMQ,</disclaimer>

It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:

Cheers

robert

···

On Mon, Jan 24, 2011 at 8:16 PM, Christoph B. <ch.blank@gmail.com> wrote:

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker "frameworks" come with
their own queue... however a few concerns:

- does putting msgs from one into another queue make sense?
- how reliable is resque? I cannot lose mails
- how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this...
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue.. which is not really needed in this
scenario as I confirm messages on the OpenMQ..

I would appreciate any suggestions!

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

3

Hey, thanks a lot for the input!

I will give those examples a try, looks pretty easy.
I also thought about just creating a thread pool.

One more thing:
How would you wrap this into a daemon, e.g. to have an executable with
start/stop etc commands?

I remember there was something like this, saw an example of a self
contained sinatra app, but I don't remember it anymore...

I would need that for monitoring with monit

Thanks for your help,
Christoph

···

--
Posted via http://www.ruby-forum.com/.

4

Here is an example that will spawn 7 worker process and connect to a beanstalkd job queue. Replace beanstalkd with an OpenMQ connection if you want. However beanstalkd is very simple, reliable, requeues jobs if they fail, etc.

http://kr.github.com/beanstalkd/

Blessings,
TwP

···

On Jan 25, 2011, at 3:11 AM, Robert Klemme wrote:

On Mon, Jan 24, 2011 at 8:16 PM, Christoph B. <ch.blank@gmail.com> wrote:

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker "frameworks" come with
their own queue... however a few concerns:

- does putting msgs from one into another queue make sense?
- how reliable is resque? I cannot lose mails
- how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this...
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue.. which is not really needed in this
scenario as I confirm messages on the OpenMQ..

I would appreciate any suggestions!

<disclaimer>I don't know stomp and I have only rudimentary knowledge
of OpenMQ,</disclaimer>

It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:

Sample work distribution across processes · GitHub