IO and non-strings

ruby-talk
1

How can I write non-string data using the IO class?
In particular, I'd like to write an integer across a socket connection.
All the methods I see in IO for writing either only write strings or
only write the to_s version of an object, which is of course a string.

···

--
R. Mark Volkmann
Partner, Object Computing, Inc.

2

see String#unpack and Array#pack.

you probably want something like

   socket.write [42].pack("N")

cheers.

-a

···

On Wed, 1 Feb 2006, Mark Volkmann wrote:

How can I write non-string data using the IO class?
In particular, I'd like to write an integer across a socket connection.
All the methods I see in IO for writing either only write strings or
only write the to_s version of an object, which is of course a string.

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama

3

Thanks! I wasn't aware of pack/unpack. I still can't get this simple
example to work though. I'm wondering if I need to use puts and gets.
If I use write, I'm not sure if the other end will know how many bytes
to read. Can you spot anything I'm doing wrong? The client sends
integers to the server. The server computes the average and returns
it. Currently the server only gets two of the three ints and the
client says "'gets': Invalid argument".

···

On 1/31/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:

On Wed, 1 Feb 2006, Mark Volkmann wrote:

> How can I write non-string data using the IO class?
> In particular, I'd like to write an integer across a socket connection.
> All the methods I see in IO for writing either only write strings or
> only write the to_s version of an object, which is of course a string.

see String#unpack and Array#pack.

you probably want something like

   socket.write [42].pack("N")

-----------
client.rb
-----------

require 'socket'

# Create socket.
host, port = 'localhost', 1919
socket = TCPSocket.new(host, port)

# Send request message.
binary_string = [19, 10, 8].pack('N*')
socket.puts binary_string

# Get response message.
binary_string = socket.gets
avg = binary_string.unpack('g') # float
puts "avg = #{avg}"

socket.close

------------
server.rb
------------

require 'socket'

# Create server.
host, port = 'localhost', 1919
server = TCPServer.new(host, port)

# Process connection requests.
loop do
  session = server.accept
  break if server == nil

  # Process session requests.
  binary_string = session.gets
  values = binary_string.unpack('N*')
  sum = 0
  values.each { |value| sum += value }
  puts "sum = #{sum}"
  puts "size = #{values.size}"

  avg = sum.to_f / values.size

  binary_string = [avg].pack('g') # float
  session.puts(binary_string)
  session.close
end

--
R. Mark Volkmann
Partner, Object Computing, Inc.

4

Mark Volkmann wrote:
...

require 'socket'

# Create socket.
host, port = 'localhost', 1919
socket = TCPSocket.new(host, port)

# Send request message.
binary_string = [19, 10, 8].pack('N*')
socket.puts binary_string

You want to avoid using gets/puts with binary strings. Those functions
are for text, and will detect/insert line ends. In fact, 10 is the same
as a \n character, so that's probably where it's choking.

···

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

5

What do you recommend I use instead of gets and puts?
If I use write, will I need to also call flush?
Will I have to first send a number to the server that indicates the
number of bytes I'm going to send next so it will know how many to
read?

···

On 2/1/06, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:

Mark Volkmann wrote:
...
> require 'socket'
>
> # Create socket.
> host, port = 'localhost', 1919
> socket = TCPSocket.new(host, port)
>
> # Send request message.
> binary_string = [19, 10, 8].pack('N*')
> socket.puts binary_string

You want to avoid using gets/puts with binary strings. Those functions
are for text, and will detect/insert line ends. In fact, 10 is the same
as a \n character, so that's probably where it's choking.

--
R. Mark Volkmann
Partner, Object Computing, Inc.

6

Mark Volkmann wrote:

···

On 2/1/06, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:

Mark Volkmann wrote:
...

require 'socket'

# Create socket.
host, port = 'localhost', 1919
socket = TCPSocket.new(host, port)

# Send request message.
binary_string = [19, 10, 8].pack('N*')
socket.puts binary_string

You want to avoid using gets/puts with binary strings. Those functions
are for text, and will detect/insert line ends. In fact, 10 is the same
as a \n character, so that's probably where it's choking.

What do you recommend I use instead of gets and puts?
If I use write, will I need to also call flush?
Will I have to first send a number to the server that indicates the
number of bytes I'm going to send next so it will know how many to
read?

--
R. Mark Volkmann
Partner, Object Computing, Inc.

I am in the habit of using send/recv, which are unbuffered, and also
sending a length field before the data. If you're interested, I've got a
couple of classes (subclasses of TCPSocket and TCPServer) that
encapsulate this and are pretty well tested. (Also a C version of the
same thing.)

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

7

Mark Volkmann wrote:
...

require 'socket'

# Create socket.
host, port = 'localhost', 1919
socket = TCPSocket.new(host, port)

# Send request message.
binary_string = [19, 10, 8].pack('N*')
socket.puts binary_string

You want to avoid using gets/puts with binary strings. Those functions
are for text, and will detect/insert line ends. In fact, 10 is the same
as a \n character, so that's probably where it's choking.

What do you recommend I use instead of gets and puts?
If I use write, will I need to also call flush?

yes. yes.

Will I have to first send a number to the server that indicates the number
of bytes I'm going to send next so it will know how many to read?

that certainly makes it easier in some cases. be careful about the length of
the lenght though; for instance, if you decide to send something like
'length:data' to the client then

   42:foobar # length length == 2
   4242:foobar # length length == 4

might want the client to read chars, based on return value of select, into a
re-sizeable buf until ':' is seen and then convert to int to do a read(n).

of course, if this were ruby to ruby you would be using drb right?

cheers.

-a

···

On Thu, 2 Feb 2006, Mark Volkmann wrote:

On 2/1/06, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama

8

I'd like to put a plug in for the book: Advanced Programming in the Unix
Environment (APUE) by Rich Stevens. (Disclaimer: I worked with and co-authored
a book with Rich). If you are doing lots of network programming then you
want to read Unix Network Programming also.

Many of the IO and networking questions that pop up on this list are
not really Ruby questions but questions about the Unix IO model, socket
programming and/or Unix OS concepts (or Posix if you prefer). If you are
doing lots of programming of that type it will really help to understand
the underlying OS abstractions. Understanding the Posix OS model will
also help you write more portable code that works correctly on Linux, Mac OS X,
Windows, and so on.

Gary Wright

···

On Feb 1, 2006, at 11:11 AM, Mark Volkmann wrote:

You want to avoid using gets/puts with binary strings. Those functions
are for text, and will detect/insert line ends. In fact, 10 is the same
as a \n character, so that's probably where it's choking.

What do you recommend I use instead of gets and puts?
If I use write, will I need to also call flush?
Will I have to first send a number to the server that indicates the
number of bytes I'm going to send next so it will know how many to
read?

9

please share! i just wrote that, for like the 10th time, last month :wink:

-a

···

On Thu, 2 Feb 2006, Joel VanderWerf wrote:

I am in the habit of using send/recv, which are unbuffered, and also
sending a length field before the data. If you're interested, I've got a
couple of classes (subclasses of TCPSocket and TCPServer) that
encapsulate this and are pretty well tested. (Also a C version of the
same thing.)

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama

10

If you want a general mechanism for doing this type of work between Ruby and Other Languages, use NetStrings [1]. If it's Ruby to Ruby, drb is a great choice. (Oh, there is also a Ruby NetStrings class [2] but I can't vouch for its correctness.)

cr

[1] http://cr.yp.to/proto/netstrings.txt
[2] http://raa.ruby-lang.org/project/djb-netstrings/

···

On Feb 1, 2006, at 10:31 AM, ara.t.howard@noaa.gov wrote:

On Thu, 2 Feb 2006, Mark Volkmann wrote:

On 2/1/06, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:
[snip]
Will I have to first send a number to the server that indicates the number
of bytes I'm going to send next so it will know how many to read?

that certainly makes it easier in some cases. be careful about the length of
the lenght though; for instance, if you decide to send something like
'length:data' to the client then

  42:foobar # length length == 2
  4242:foobar # length length == 4

might want the client to read chars, based on return value of select, into a
re-sizeable buf until ':' is seen and then convert to int to do a read(n).

11

thanks. i actually read that and used something very close to netstrings for
my acgi package - but i'd forgotten where i read about it. now i remember!
:wink:

-a

···

On Thu, 2 Feb 2006 cremes.devlist@mac.com wrote:

On Feb 1, 2006, at 10:31 AM, ara.t.howard@noaa.gov wrote:

On Thu, 2 Feb 2006, Mark Volkmann wrote:

On 2/1/06, Joel VanderWerf <vjoel@path.berkeley.edu> wrote:
[snip]
Will I have to first send a number to the server that indicates the number
of bytes I'm going to send next so it will know how many to read?

that certainly makes it easier in some cases. be careful about the length of
the lenght though; for instance, if you decide to send something like
'length:data' to the client then

  42:foobar # length length == 2
  4242:foobar # length length == 4

might want the client to read chars, based on return value of select, into a
re-sizeable buf until ':' is seen and then convert to int to do a read(n).

If you want a general mechanism for doing this type of work between Ruby and Other Languages, use NetStrings [1]. If it's Ruby to Ruby, drb is a great choice. (Oh, there is also a Ruby NetStrings class [2] but I can't vouch for its correctness.)

cr

[1] http://cr.yp.to/proto/netstrings.txt
[2] http://raa.ruby-lang.org/project/djb-netstrings/

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama

12

MTCP -- Message TCP -- Wrapper around TCPSocket and TCPServer that
provides a message (datagram) abstraction implemented by data stream
with added length fields.

Here's the ruby version, attached. There's a tarball at
  http://redshift.sourceforge.net/mtcp/
with the full deal: c version, tests for both, etc. The c version works
with both blocking and non-blocking sockets. The c lib code works on
windows, but the test doesn't (it forks). It's fine on linux and qnx.
The ruby version works on all platforms I've tried (windows and linux).

mtcp.rb (2.42 KB)

···

ara.t.howard@noaa.gov wrote:

On Thu, 2 Feb 2006, Joel VanderWerf wrote:

I am in the habit of using send/recv, which are unbuffered, and also
sending a length field before the data. If you're interested, I've got a
couple of classes (subclasses of TCPSocket and TCPServer) that
encapsulate this and are pretty well tested. (Also a C version of the
same thing.)

please share! i just wrote that, for like the 10th time, last month :wink:

-a

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

13

sweet! thanks joel.

-a

mtcp.rb (2.42 KB)

···

On Thu, 2 Feb 2006, Joel VanderWerf wrote:

ara.t.howard@noaa.gov wrote:

On Thu, 2 Feb 2006, Joel VanderWerf wrote:

I am in the habit of using send/recv, which are unbuffered, and also
sending a length field before the data. If you're interested, I've got a
couple of classes (subclasses of TCPSocket and TCPServer) that
encapsulate this and are pretty well tested. (Also a C version of the
same thing.)

please share! i just wrote that, for like the 10th time, last month :wink:

-a

MTCP -- Message TCP -- Wrapper around TCPSocket and TCPServer that
provides a message (datagram) abstraction implemented by data stream
with added length fields.

Here's the ruby version, attached. There's a tarball at
http://redshift.sourceforge.net/mtcp/
with the full deal: c version, tests for both, etc. The c version works
with both blocking and non-blocking sockets. The c lib code works on
windows, but the test doesn't (it forks). It's fine on linux and qnx.
The ruby version works on all platforms I've tried (windows and linux).

--
     vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

--
happiness is not something ready-made. it comes from your own actions.
- h.h. the 14th dali lama