[ANN] rq-2.3.2

ruby-talk
1

NAME
rq v2.3.2

SYNOPSIS
rq (queue | export RQ_Q=q) mode [mode_args]* [options]*

URIS

http://raa.ruby-lang.org/project/rq/
http://codeforpeople.com/lib/ruby/rq/rq-2.3.2/README
http://www.linuxjournal.com/article/7922

DESCRIPTION
ruby queue (rq) is a zero-admin zero-configuration tool used to create
instant linux clusters. rq requires only a central nfs filesystem in order
to manage a simple sqlite database as a distributed priority work queue.
this simple design allows researchers to install and configure, in only a
few minutes and without root privileges, a robust linux cluster capable of
distributing processes to many nodes - bringing dozens of powerful cpus to
their knees with a single blow. clearly this software should be kept out of
the hands of free radicals, seti enthusiasts, and one mr. j safran.

the central concept of rq is that n nodes work in isolation to pull jobs
from an centrally mounted nfs priority work queue in a synchronized fashion.
the nodes have absolutely no knowledge of each other and all communication
is done via the queue meaning that, so long as the queue is available via
nfs and a single node is running jobs from it, the system will continue to
process jobs. there is no centralized process whatsoever - all nodes work
to take jobs from the queue and run them as fast as possible. this creates
a system which load balances automatically and is robust in face of node
failures.

although the rq system is simple in it’s design it features powerful
functionality such as priority management, predicate and sql query , compact
streaming command-line processing, programmable api, hot-backup, and
input/capture of the stdin/stdout/stderr io streams of remote jobs. to date
rq has had no reported runtime failures and is in operation at dozens of
research centers around the world.

enjoy.

-a

···

–
some people, sweet and attractive, and strong and healthy, happen to die
young. they are masters in disguise teaching us about impermanence.

  • h.h. the 14th dali lama
2

I couldn't find the changes list. What is new in this version?

Guillaume.

···

Le 28 avr. 06, à 17:36, Ara.T.Howard a écrit :

NAME
  rq v2.3.2

SYNOPSIS
  rq (queue | export RQ_Q=q) mode [mode_args]* [options]*

URIS

  http://raa.ruby-lang.org/project/rq/
  http://codeforpeople.com/lib/ruby/rq/rq-2.3.2/README
  Linux Clustering with Ruby Queue: Small Is Beautiful | Linux Journal

DESCRIPTION
  ruby queue (rq) is a zero-admin zero-configuration tool used to create
  instant linux clusters. rq requires only a central nfs filesystem in order
  to manage a simple sqlite database as a distributed priority work queue.
  this simple design allows researchers to install and configure, in only a
  few minutes and without root privileges, a robust linux cluster capable of
  distributing processes to many nodes - bringing dozens of powerful cpus to
  their knees with a single blow. clearly this software should be kept out of
  the hands of free radicals, seti enthusiasts, and one mr. j safran.

  the central concept of rq is that n nodes work in isolation to pull jobs
  from an centrally mounted nfs priority work queue in a synchronized fashion.
  the nodes have absolutely no knowledge of each other and all communication
  is done via the queue meaning that, so long as the queue is available via
  nfs and a single node is running jobs from it, the system will continue to
  process jobs. there is no centralized process whatsoever - all nodes work
  to take jobs from the queue and run them as fast as possible. this creates
  a system which load balances automatically and is robust in face of node
  failures.

  although the rq system is simple in it's design it features powerful
  functionality such as priority management, predicate and sql query , compact
  streaming command-line processing, programmable api, hot-backup, and
  input/capture of the stdin/stdout/stderr io streams of remote jobs. to date
  rq has had no reported runtime failures and is in operation at dozens of
  research centers around the world.

enjoy.

-a
--
some people, sweet and attractive, and strong and healthy, happen to die
young. they are masters in disguise teaching us about impermanence.
- h.h. the 14th dali lama

3

sorry. here you go

   http://www.codeforpeople.com/lib/ruby/rq/rq-2.3.2/HISTORY

···

On Sat, 29 Apr 2006, Guillaume Marcais wrote:

I couldn't find the changes list. What is new in this version?

Guillaume.

   ---
   2.3.2:
     - added 'rq q io' command
     - streamlined 'rq q status' output considerably
     - fixed few small bugs in util.rb
     - patched yaml inline behaviour for syck in 1.8.4. backward compatible with
       older ruby versions - at least through 1.8.2.
     - changed naming scheme for rotation and added automatic attempt to compress
       the rotation
     - made shutdown via ctrl-c when running from commandline (as opposed to
       daemon) quieter
     - fixed a few bad exit codes, for example creating a queue exited with
       $?==1, it is now correctly 0
     - added support for job stdin to be provided and stdout/stderr captured.
       for instance

         rq queue submit cat --stdin=input_for_cat.txt

         cat input_for_cat.txt | rq queue submit cat --stdin=-

         rq queue submit cat --stdin=- < input_for_cat.txt

       stdout and stderr are stored in the queue directory in a file named by
       jid.
     - changed behaviour of reading jobs from stdin: ***IMPORTANT*** : it's now
       **REQUIRED** to provide a '-' as the argument for any command that should
       read from stdin. for instance one would use

         rq queue submit - < joblist
       or
         jobgenerator.rb | rq queue submit -

       vs the old

         rq queue submit < joblist
       or
         jobgenerator.rb | rq queue submit

       the distiction was added to make running submissions from scripts/cron
       (where STDIN.tty? == false) easier and providing the job stdinput on stdin
       possible.
     - added a restart mode to restart a feeder : rq q restart
     - modes shutdown, stop, and restart now return a non-zero exit code if they
       fail to signal the feeder, zero otherwise
     - fixed bug with restart mode whereby re-exec failed due to falsely thinking
       pid was locked (it is - but only by the current process!)
     - added default bin dir underneath q dir. this dir is added to the path
       before excuting commands to it's an easy place to put things where they'll
       be found

the biggest change is the ability to provide jobs stdin at submit time and to
capture their stdout/stderr. for instance you can now do this

   rq q submit cat --stdin=/etc/passwd

or

   find | grep *geo.tif | rq q submit geolocate.rb --stdin=-

to submit jobs that take stdin. the stdin is squirled away in the q dir.
later, when the job is run, the stdout and stderr are captured in the same
way. so, in the above example, if the submitted job had jid=42 you'd have

   q/stdin/42 # the stdin provided

before the job was run and

   q/stdin/42
   q/stdout/42
   q/stderr/42

once the job was set running. this is extrememly cool since you can do

   tail -F q/stderr/42

to monitor a process on a remote node.

cheers.

-a
--
some people, sweet and attractive, and strong and healthy, happen to die
young. they are masters in disguise teaching us about impermanence.
- h.h. the 14th dali lama