Drb over pipes

ruby-talk
1

Hi

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

Unfortunately using drb's tcp protocol directly isn't really an option in
our environment.

Does anybody have a good idea?

Thanks!

Cheers,
Reto Schüttel

2

Unfortunately using drb's tcp protocol directly isn't really an option in
our environment.

I would suggest using ssh-tunneling.

That way you have the security of secure shell
and the comfort of tcp sockets.

···

--- Ursprüngliche Nachricht ---
Von: Reto Schuettel <reto-ruby-talk@schuettel.ch>
An: ruby-talk@ruby-lang.org (ruby-talk ML)
Betreff: drb over pipes
Datum: Thu, 23 Mar 2006 18:42:49 +0900

Hi

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

Unfortunately using drb's tcp protocol directly isn't really an option in
our environment.

Does anybody have a good idea?

Thanks!

Cheers,
Reto Schüttel

3

Hi

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

With drb, the client needs to talk to the agent and vice-versa, so this
command line will never work (the client will talk to the agent, but not the
other way around).

Unfortunately using drb's tcp protocol directly isn't really an option in
our environment.

Does anybody have a good idea?

As suggested, tunelling is probably the best/simpler choice.

···

On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto Schuettel wrote:

Thanks!

Cheers,
Reto Schüttel

--
Yoann Guillot

4

I implemented a quick hack to do this. Hopefully the code below will help.
It's a bit of a mess, but allows you to use a DRb scheme like
'drbfd://0,1' which means uses file descriptors 0 and 1 (stdin/stdout) for
communication.

module DRb
  class DRbFDSocket
    def self.parse_uri(uri)
      if /^drbfd:\/\/(\d+),(\d+)(?:\?(.*))?$/ =~ uri
        [$1.to_i,$2.to_i, $3]
      else
        raise DRbBadScheme,uri unless uri =~ /^drbfd:/
        raise DRbBadURI, "can't parse uri: " + uri
      end
    end
  
    def self.uri_option(uri,config)
      infd,outfd, option = parse_uri(uri)
      return ["drbfd://#{infd},#{outfd}", option]
    end
  
    def self.open(uri,config)
      opts = parse_uri(uri)
      self.new(uri,opts,config)
    end
  
    def self.open_server(uri,config)
      opts = parse_uri(uri)
      self.new(uri,opts,config)
    end
  
    def initialize(uri,fds,config)
      @uri = uri
      @in_fp = IO.new(fds[0],"r")
      @out_fp = IO.new(fds[1],"w")
      @out_fp.sync = true
      
      @msg = DRbMessage.new(config)
    end
  
    attr_reader :uri
  
    def send_request(ref, msg_id, arg, b)
      begin
        result = @msg.send_request(@out_fp, ref, msg_id, arg, b)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end
  
    def recv_request
      begin
        result = @msg.recv_request(@in_fp)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end
  
    def send_reply(succ, result)
      begin
        result = @msg.send_reply(@out_fp, succ, result)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end
  
    def recv_reply
      begin
        @msg.recv_reply(@in_fp)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
    end

    def alive?
      true
    end
  
    def close
      @in_fp.close
      @out_fp.close
    end
    
    def accept
      if @accepted then
        sleep 60 while true
      else
        @accepted = true
        self
      end
    end
     
  end
  DRbProtocol.add_protocol(DRbFDSocket)
end

···

On Thu, 23 Mar 2006 18:42:49 +0900, Reto Schuettel wrote:

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

5

some
links:

http://www.rzg.mpg.de/networking/tunnelling.html

···

--- Ursprüngliche Nachricht ---
Von: "Peter Ertl" <pertl@gmx.org>
An: ruby-talk@ruby-lang.org (ruby-talk ML)
Betreff: Re: drb over pipes
Datum: Thu, 23 Mar 2006 18:46:54 +0900

> Unfortunately using drb's tcp protocol directly isn't really an option
in
> our environment.

I would suggest using ssh-tunneling.

That way you have the security of secure shell
and the comfort of tcp sockets.

> --- Ursprüngliche Nachricht ---
> Von: Reto Schuettel <reto-ruby-talk@schuettel.ch>
> An: ruby-talk@ruby-lang.org (ruby-talk ML)
> Betreff: drb over pipes
> Datum: Thu, 23 Mar 2006 18:42:49 +0900
>
> Hi
>
> I'd like to establish a drb connection over ssh, preferably over the
> stdin/stdout channels which are provided by ssh itself. Is there a
> drb protocol which uses pipes? If not, is it easy to implement?
>
> I've seen that the tcpprotocol uses two connections, this could be
> problem, as pipes/stdin/stdout only provide one stream.
>
> At the end of the day I'd like to do something like that:
> # ruby client.rb | ssh host 'ruby agent.rb'
>
> Unfortunately using drb's tcp protocol directly isn't really an option
in
> our environment.
>
> Does anybody have a good idea?
>
> Thanks!
>
> Cheers,
> Reto Schüttel
>

6

Hi

> At the end of the day I'd like to do something like that:
> # ruby client.rb | ssh host 'ruby agent.rb'
>

With drb, the client needs to talk to the agent and vice-versa, so this
command line will never work (the client will talk to the agent, but not the
other way around).

A pipe provides two streams (in & out), and tcp does the same. The
question is now if drb really needs two connections (-> 4 streams)?

> Unfortunately using drb's tcp protocol directly isn't really an option in
> our environment.
>
> Does anybody have a good idea?

As suggested, tunelling is probably the best/simpler choice.

Tunneling may be a solution, although I'd prefer the pipe. Otherwise
there would be a daemon, somewhere listening on localhost:port.

Regards,
Reto Schüttel

···

On Thu, Mar 23, 2006 at 07:01:35PM +0900, Yoann Guillot wrote:

On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto Schuettel wrote:

7

Jonathan Paisley wrote:

···

On Thu, 23 Mar 2006 18:42:49 +0900, Reto Schuettel wrote:

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

I implemented a quick hack to do this. Hopefully the code below will help

The code didn't make it into the newsgroup. I can't detect something strange about your original message - it's all there on my email account. This is strange...

Cheers

  robert

8

Hi Jonathan

I implemented a quick hack to do this. Hopefully the code below will help.
It's a bit of a mess, but allows you to use a DRb scheme like
'drbfd://0,1' which means uses file descriptors 0 and 1 (stdin/stdout) for
communication.

module DRb

That's great! exactly what I needed.

For the record: My statement in the first mail wasn't completely
correct, the pipe I used in my example is only a 'one-way' thing. But
ssh actually transports three streams/pipes: stdin, stdout and stderr, so this
isn't really a problem.

There's one small problem with your fdsocket, it seems like the drb
server/thread doesn't always terminate itself when the streams get
closed. But I will try to investigate this by myself tomorrow :).

I've appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then execs
the ssh client. The same technique could also be used to communicate
with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
that's really nice, great work Jonathan!

require 'drb'
require 'fdsocket'

ctprd, ctpwr = IO.pipe
ptcrd, ptcwr = IO.pipe

pid = fork

unless pid
  # child
  ctprd.close
  ptcwr.close

  $stdin.reopen( ptcrd )
  $stdout.reopen( ctpwr )

  exec("ssh", "hostname", "./agent.rb")

  exit # child shounld't reach this point
end

# parent
ctpwr.close
ptcrd.close

fd_read = ctprd.fileno
fd_write = ptcwr.fileno

# The URI to connect to
SERVER_URI="drbfd://#{fd_read},#{fd_write}"
DRb.start_service

obj = DRbObject.new_with_uri(SERVER_URI)

[...]

Regards,
Reto Schuettel

···

On Thu, Mar 23, 2006 at 12:34:31PM +0000, Jonathan Paisley wrote:

9

the 'daemon' is just ssh which you already have.
there will be no additional executables when you use tunneling,
just secure shell on port 22 (or whatever). tunneling is configured
in the client and can be allowed/forbidden on the server side.

···

--- Ursprüngliche Nachricht ---
Von: Reto Schuettel <reto-ruby-talk@schuettel.ch>
An: ruby-talk@ruby-lang.org (ruby-talk ML)
Betreff: Re: drb over pipes
Datum: Thu, 23 Mar 2006 19:24:03 +0900

Hi

On Thu, Mar 23, 2006 at 07:01:35PM +0900, Yoann Guillot wrote:
> On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto Schuettel wrote:
> > At the end of the day I'd like to do something like that:
> > # ruby client.rb | ssh host 'ruby agent.rb'
> >
>
> With drb, the client needs to talk to the agent and vice-versa, so this
> command line will never work (the client will talk to the agent, but not
the
> other way around).

A pipe provides two streams (in & out), and tcp does the same. The
question is now if drb really needs two connections (-> 4 streams)?

> > Unfortunately using drb's tcp protocol directly isn't really an option
in
> > our environment.
> >
> > Does anybody have a good idea?
>
> As suggested, tunelling is probably the best/simpler choice.

Tunneling may be a solution, although I'd prefer the pipe. Otherwise
there would be a daemon, somewhere listening on localhost:port.

Regards,
Reto Schüttel

10

Very strange. I sent the message to the mailing list, where it has shown
up correctly in the archives [1] and been forwarded correctly as your
email account attests. I guess it must be something broken in the gateway.

Jonathan

[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/185525

···

On Thu, 23 Mar 2006 21:58:52 +0900, Robert Klemme wrote:

Jonathan Paisley wrote:

I implemented a quick hack to do this. Hopefully the code below will help

The code didn't make it into the newsgroup. I can't detect something
strange about your original message - it's all there on my email
account. This is strange...

11

There's one small problem with your fdsocket, it seems like the drb
server/thread doesn't always terminate itself when the streams get
closed. But I will try to investigate this by myself tomorrow :).

I encountered that too; hence all those exception handlers and calls to
exit. I worked around it by ensuring that my code made periodic calls
across the DRb connection --- this would trigger an exception. Otherwise,
I ended up with loads of server processes doing nothing.

I'd be interested to hear if you've figured out how to get notification of
the socket closing.

I've appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then execs
the ssh client. The same technique could also be used to communicate
with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
that's really nice, great work Jonathan!

I should have included an example with my code --- although it turns out
now that mine is practically the same as yours (I too am using ssh
transport). Sorry for omitting the extra code...

Jonathan

···

On Fri, 24 Mar 2006 03:30:00 +0900, Reto Schuettel wrote:

12

i think this will fail with calls that use 'yield' or that return
'DrbUndumped' objects because they will attempt to open a connection __from__
'hostname' to 'localhost'.

this doccument details this in the ssh section

   http://www.rubygarden.org/ruby?DrbTutorial

fyi.

-a

···

On Fri, 24 Mar 2006, Reto Schuettel wrote:

I've appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then execs
the ssh client. The same technique could also be used to communicate with a
child (instead of a unix/tcp socket or a simple plain pipe). IMHO that's
really nice, great work Jonathan!

require 'drb'
require 'fdsocket'

ctprd, ctpwr = IO.pipe
ptcrd, ptcwr = IO.pipe

pid = fork

unless pid
# child
ctprd.close
ptcwr.close

$stdin.reopen( ptcrd )
$stdout.reopen( ctpwr )

exec("ssh", "hostname", "./agent.rb")

exit # child shounld't reach this point
end

# parent
ctpwr.close
ptcrd.close

fd_read = ctprd.fileno
fd_write = ptcwr.fileno

# The URI to connect to
SERVER_URI="drbfd://#{fd_read},#{fd_write}"
DRb.start_service

obj = DRbObject.new_with_uri(SERVER_URI)

[...]

--
share your knowledge. it's a way to achieve immortality.
- h.h. the 14th dali lama

13

Hi Peter

the 'daemon' is just ssh which you already have.
there will be no additional executables when you use tunneling,
just secure shell on port 22 (or whatever). tunneling is configured
in the client and can be allowed/forbidden on the server side.

Well.. yeah, there's no need for an additional daemon on the outside,
but a daemon at the beginning of the session would be needed (although
it could be shut down shortly later). Eg

$ ssh -L 4444:localhost:4444 'drb.rb drb://localhost:4444'

This would work quite well, but the drb port would be accessible by
local processes.

Reto Schuettel

···

On Thu, Mar 23, 2006 at 07:49:56PM +0900, Peter Ertl wrote: