Data queuing

ruby-talk
1

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc
2.0.0)

Moreover what is the best solution (simplicity, performance) for a queue
feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe,
network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

2

I'm not sure what you mean. I looked at ruby-doc.org and I see the Queue
class in the core library for 2.1, 2.2, and 2.3.

···

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc
2.0.0)

Moreover what is the best solution (simplicity, performance) for a queue
feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe,
network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--

- Raj

3

Hi
I see it changed from stdlib to core,
but nothing is found with search engine:

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

Thanks
Berg

···

Am 07.09.2016 08:32 schrieb "Raj Sahae" <rajsahae@gmail.com>:

I'm not sure what you mean. I looked at ruby-doc.org and I see the Queue
class in the core library for 2.1, 2.2, and 2.3.

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc
2.0.0)

Moreover what is the best solution (simplicity, performance) for a queue
feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe,
network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--

- Raj

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

4

jruby sounds like the easiest way forward here

···

On Tue, Sep 6, 2016 at 11:41 PM, A Berger <aberger7890@gmail.com> wrote:

Hi
I see it changed from stdlib to core,
but nothing is found with search engine: Bing
ruby+queue

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

Thanks
Berg
Am 07.09.2016 08:32 schrieb "Raj Sahae" <rajsahae@gmail.com>:

I'm not sure what you mean. I looked at ruby-doc.org and I see the Queue
class in the core library for 2.1, 2.2, and 2.3.

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc
2.0.0)

Moreover what is the best solution (simplicity, performance) for a queue
feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe,
network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--

- Raj

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

5

Just use Rubinuios Or Jruby , they support really system threads without the GIL

http://www.aqua-mail.com

···

Sent with AquaMail for Android

On September 7, 2016 9:41:43 AM A Berger <aberger7890@gmail.com> wrote:

Hi
I see it changed from stdlib to core,
but nothing is found with search engine: https://www.bing.com/search?q=ruby+queue

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

Thanks
Berg

Am 07.09.2016 08:32 schrieb "Raj Sahae" <rajsahae@gmail.com<mailto:rajsahae@gmail.com>>:

I'm not sure what you mean. I looked at ruby-doc.org<http://ruby-doc.org> and I see the Queue class in the core library for 2.1, 2.2, and 2.3.

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com<mailto:aberger7890@gmail.com>> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc 2.0.0)

Moreover what is the best solution (simplicity, performance) for a queue feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe, network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org<mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>
--

- Raj

6

I see it changed from stdlib to core,
but nothing is found with search engine:
ruby queue - Search

$ ruby --version
ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-cygwin]
$ ruby -e 'p Queue'
Thread::Queue

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

This work is under way AFAIK but it takes time. You can search for
"ruby gil" to find out a bit more. Or use JRuby as Martin suggested.

Cheers

robert

···

On Wed, Sep 7, 2016 at 8:41 AM, A Berger <aberger7890@gmail.com> wrote:

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

7

Ok - I still wanted to avoid using java...
But thank you, maybe I should cancel that precondition.

Nevertheless are there solutions for "many-processes" - data-sharing out
there?
(Thinking a database is overkill, besides for mostly needed as queue its
not well suited for that job, but possible if at once fetching+deleting
seen rows)
Shared mem has locking, but it's not "nice" (to implement/use)...
Maybe this is the fastest possibility.

Does anybody have experience using network or pipes (or ready-to-use
queuing-gems with locking/parallel-access awareness)?

thx Berg

···

Am 07.09.2016 08:49 schrieb "Martin DeMello" <martindemello@gmail.com>:

jruby sounds like the easiest way forward here

Concurrency in jruby · jruby/jruby Wiki · GitHub

On Tue, Sep 6, 2016 at 11:41 PM, A Berger <aberger7890@gmail.com> wrote:

Hi
I see it changed from stdlib to core,
but nothing is found with search engine: Bing
ruby+queue

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

Thanks
Berg
Am 07.09.2016 08:32 schrieb "Raj Sahae" <rajsahae@gmail.com>:

I'm not sure what you mean. I looked at ruby-doc.org and I see the
Queue class in the core library for 2.1, 2.2, and 2.3.

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in Rubydoc
2.0.0)

Moreover what is the best solution (simplicity, performance) for a
queue feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a pipe,
network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--

- Raj

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

8

Hard to say more without knowing what problem you're trying to solve, but
see if resque does what you want

martin

···

On Wed, Sep 7, 2016 at 12:32 AM, A Berger <aberger7890@gmail.com> wrote:

Ok - I still wanted to avoid using java...
But thank you, maybe I should cancel that precondition.

Nevertheless are there solutions for "many-processes" - data-sharing out
there?
(Thinking a database is overkill, besides for mostly needed as queue its
not well suited for that job, but possible if at once fetching+deleting
seen rows)
Shared mem has locking, but it's not "nice" (to implement/use)...
Maybe this is the fastest possibility.

Does anybody have experience using network or pipes (or ready-to-use
queuing-gems with locking/parallel-access awareness)?

thx Berg
Am 07.09.2016 08:49 schrieb "Martin DeMello" <martindemello@gmail.com>:

jruby sounds like the easiest way forward here

Concurrency in jruby · jruby/jruby Wiki · GitHub

On Tue, Sep 6, 2016 at 11:41 PM, A Berger <aberger7890@gmail.com> wrote:

Hi
I see it changed from stdlib to core,
but nothing is found with search engine: https://www.bing.com/search?q=
ruby+queue

I want to use all cpu-cores, so processes look better than threads,
so I cant use this queue.
Better solution:
Perhaps someone can change threads to utilize all cores?

Thanks
Berg
Am 07.09.2016 08:32 schrieb "Raj Sahae" <rajsahae@gmail.com>:

I'm not sure what you mean. I looked at ruby-doc.org and I see the
Queue class in the core library for 2.1, 2.2, and 2.3.

On Tue, Sep 6, 2016, 11:23 PM A Berger <aberger7890@gmail.com> wrote:

Hi
It looks like Class queue disappeared in Ruby 2.1 (last seen in
Rubydoc 2.0.0)

Moreover what is the best solution (simplicity, performance) for a
queue feeding different Ruby-processes?
Having 1) fifo and 2) distibutor functionality.
Data should stay in Ram (e.g. Array), but for the communication a
pipe, network/socket or shared mem is possible.
Something fast&proved out there, or do I have to do it by hand?
Whats commonly used (best already with integrated locking)?

Thank you
Berg

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--

- Raj

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

9

The easy way out is to use Queue together with DRb and use processes
instead of threads for concurrent work.

Cheers

robert

···

On Wed, Sep 7, 2016 at 9:32 AM, A Berger <aberger7890@gmail.com> wrote:

Ok - I still wanted to avoid using java...
But thank you, maybe I should cancel that precondition.

Nevertheless are there solutions for "many-processes" - data-sharing out
there?
(Thinking a database is overkill, besides for mostly needed as queue its not
well suited for that job, but possible if at once fetching+deleting seen
rows)

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

10

> Nevertheless are there solutions for "many-processes" - data-sharing out
> there?
> (Thinking a database is overkill, besides for mostly needed as queue its
not
> well suited for that job, but possible if at once fetching+deleting seen
> rows)

The easy way out is to use Queue together with DRb and use processes
instead of threads for concurrent work.

Always remembering that DRb is as insecure as an insecure thing.

The more secure, but definitely more painful, direction to go: message passing to/from a STOMP Server, or something like RabbitMQ, or Redis' with its publish/subscribe feature.

The nice thing about DRb is that it "magically" transmits objects and method calls from server to client. AFAIK there isn't anything that will give you that with those other messaging solutions. (Hmmm ... adding that to my "mad science" list...)

Click here to view Company Information and Confidentiality Notice.<http://www.jameshall.co.uk/index.php/small-print/email-disclaimer&gt;

11

The magic is called "Marshalling". :wink: This also immediately gives you
the limitation of objects that you can transfer (e.g. File).

Kind regards

robert

···

On Wed, Sep 7, 2016 at 10:16 AM, Andy Jones <Andy.Jones@jameshall.co.uk> wrote:

The nice thing about DRb is that it "magically" transmits objects and method calls from server to client. AFAIK there isn't anything that will give you that with those other messaging solutions. (Hmmm ... adding that to my "mad science" list...)

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

12

Hi

Thanks for the different infos!

I'm searching for sg more small&fast, for msging like Twitter [inside one
server:) ] (never used Twitter, so I don't know what it is today...).

Small data-pieces are exchanged, mostly between each two of many pids (n/2
x 1:1 , bidir) and a server getting the results (n:1, unidir)
Seems one big memory-area (array) would be best: using the same memory is
faster than copying / packing / sending via network, and *looks* easier.
Also the whole state can easily being saved.
Are there no ready-used solutions for that yet?

Thank you
Berg

···

Am 07.09.2016 10:16 schrieb "Andy Jones" <Andy.Jones@jameshall.co.uk>:

> > Nevertheless are there solutions for "many-processes" - data-sharing
out
> > there?
> > (Thinking a database is overkill, besides for mostly needed as queue
its
> not
> > well suited for that job, but possible if at once fetching+deleting
seen
> > rows)
>
> The easy way out is to use Queue together with DRb and use processes
> instead of threads for concurrent work.

Always remembering that DRb is as insecure as an insecure thing.

The more secure, but definitely more painful, direction to go: message
passing to/from a STOMP Server, or something like RabbitMQ, or Redis' with
its publish/subscribe feature.

The nice thing about DRb is that it "magically" transmits objects and
method calls from server to client. AFAIK there isn't anything that will
give you that with those other messaging solutions. (Hmmm ... adding that
to my "mad science" list...)

Click here to view Company Information and Confidentiality Notice.<
http://www.jameshall.co.uk/index.php/small-print/email-disclaimer&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

13

The magic is called "Marshalling". :wink: This also immediately gives you
the limitation of objects that you can transfer (e.g. File).

To be fair to the authors of DRb, there is *slightly* more going on than just Marshal::load( Marshal::dump )...

Click here to view Company Information and Confidentiality Notice.<http://www.jameshall.co.uk/index.php/small-print/email-disclaimer&gt;

14

...
At the BEGINNING all processes are fed with data, so the main-queue is
unidirectional, but direction is switchable :wink:
(Which makes no difference if you see the SAME queue as 2 unidir queues ,
but only one active; whereas the client-queues are really bidir: my idea
was to put an receiver-adress into every packet, but 2 queues each seem to
be more efficient)
- There are many msg-streams, so I want to avoid duplicating the data.

Thanks
Berg

15

For the "magic" transfer of objects I think that is really all there
is to it. You basically have ordinary objects which are sent via
mashalling and other objects for which only a stub is transferred that
then invokes methods on the remote original. That would have to be
done for the Queue instance in this scenario btw.

Kind regards

robert

···

On Wed, Sep 7, 2016 at 11:56 AM, Andy Jones <Andy.Jones@jameshall.co.uk> wrote:

The magic is called "Marshalling". :wink: This also immediately gives you
the limitation of objects that you can transfer (e.g. File).

To be fair to the authors of DRb, there is *slightly* more going on than just Marshal::load( Marshal::dump )...

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

16

Hi
...and lastly the (local running) service for remotely accessing the local
objects (same the other way round), if ever needed.

- would all be much easier than using SharedMemory :((

kind regards
berg

···

Am 07.09.2016 13:15 schrieb "Robert Klemme" <shortcutter@googlemail.com>:

On Wed, Sep 7, 2016 at 11:56 AM, Andy Jones <Andy.Jones@jameshall.co.uk> > wrote:
>> The magic is called "Marshalling". :wink: This also immediately gives you
>> the limitation of objects that you can transfer (e.g. File).
>
> To be fair to the authors of DRb, there is *slightly* more going on than
just Marshal::load( Marshal::dump )...

For the "magic" transfer of objects I think that is really all there
is to it. You basically have ordinary objects which are sent via
mashalling and other objects for which only a stub is transferred that
then invokes methods on the remote original. That would have to be
done for the Queue instance in this scenario btw.

Kind regards

robert

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can
- without end}
http://blog.rubybestpractices.com/

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;