TCPserver how to signal EOT

ruby-talk
1

Hi I am writing some networking code and am having a heck of a time
figuring out how to break on EOT. I was sending an ASCII control
character, but this doesn't work when I send binary data because
sometimes it is seen as the EOT character so breaks early. Here is one
thing I have tried:

server = TCPServer.open(address, port)

loop do
  Thread.start(server.accept) do |client|
    message = ""
      loop do
        message << client.recvfrom( 4 )[0]
        break if client.eof? == true
      end
     yield(message, client)
  end
end

if I replace the break if client.eof? == true with simply puts
client.eof? it will return false if it is not the end of what is being
sent, otherwise it just hangs there indefinitely. This is only one of
many ways I have tried, client.recvfrom does not seem to return 0 or -1
when no more data is to be sent, it also does not seem to return an
empty string. Pretty much the only way I have gotten this to work is by
looping client.recvfrom and then breaking if the character at -1 in what
it gets is the ASCII control character for EOT. Unfortunately I can not
use that because when I send random binary data there is a chance it
will end on EOT without it being a control character.

Surely there is some easy way to do this, but none of the Ruby
documentation mentions anything about how to and most of the examples do
not even have recv in a loop for some reason.

···

--
Posted via http://www.ruby-forum.com/.

2

Hi I am writing some networking code and am having a heck of a time
figuring out how to break on EOT.

This is only one of
many ways I have tried, client.recvfrom does not seem to return 0 or -1
when no more data is to be sent, it also does not seem to return an
empty string.

Then it probably blocks.

Pretty much the only way I have gotten this to work is by
looping client.recvfrom and then breaking if the character at -1 in what
it gets is the ASCII control character for EOT. Unfortunately I can not
use that because when I send random binary data there is a chance it
will end on EOT without it being a control character.

I have seen recvfrom mainly used with UDP sockets and I am not sure
whether it blocks. Apparently you want blocking behavior since you
create a thread per connection.

Surely there is some easy way to do this, but none of the Ruby
documentation mentions anything about how to and most of the examples do
not even have recv in a loop for some reason.

Yes, just use Ruby's ways to use custom line delimiters. Here's one
working way:

DELIMITER = "\x04".freeze

loop do
  Thread.new(server.accept) do |client|
    printf "Client %p START\n", client

    client.each_line DELIMITER do |msg|
      printf "msg len = %4d msg: %p bytes: %p\n", msg.length, msg,
msg.unpack('C*').map {|i| "%02x" % i}
      msg.chomp! DELIMITER
      printf "msg len = %4d msg: %p bytes: %p\n", msg.length, msg,
msg.unpack('C*').map {|i| "%02x" % i}
    end

    printf "Client %p STOP\n", client
  end
end

Kind regards

robert

···

On Sun, Aug 19, 2012 at 1:03 PM, tammy roberts <lists@ruby-forum.com> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

3

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was.

class Networking

  def initialize
    @settings = Settings.new
  end

  def listen(server)
    loop do
      Thread.start(server.accept) do |client|
        data_received = []
        total_received = 0
        incoming_data_size = client.recvfrom( 4 )[0].unpack('N*')[0]

        loop do
          section_size = client.recvfrom( 4 )[0].unpack('N*')[0]
          message_section = ""

          loop do
            message_section << client.recvfrom( section_size )[0]
            break if message_section.bytesize == section_size
          end

          data_received << message_section
          total_received += 4 + message_section.bytesize
          break if total_received == incoming_data_size
        end

    yield(data_received, client)
      end
    end
  end

  def send_data(socket, message)
    forward_message = [message.bytesize].pack('N*') + message
    socket.send(forward_message, 0)
    yield(socket)
  end

  def get_reply(socket)
    incoming_data_size = socket.recvfrom( 4 )[0].unpack('N*')[0]
    section_size = socket.recvfrom( 4 )[0].unpack('N*')[0]
    reply = ""

    loop do
      reply << socket.recvfrom( section_size )[0]
      break if reply.bytesize == incoming_data_size - 4
    end

    yield(reply, socket)
  end
end

···

--
Posted via http://www.ruby-forum.com/.

4

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was.

This is a completely different approach: you do not use a termination
byte any more but transmit the message length beforehand.

class Networking

  def initialize
    @settings = Settings.new
  end

  def listen(server)
    loop do
      Thread.start(server.accept) do |client|
        data_received =
        total_received = 0
        incoming_data_size = client.recvfrom( 4 )[0].unpack('N*')[0]

        loop do
          section_size = client.recvfrom( 4 )[0].unpack('N*')[0]
          message_section = ""

          loop do
            message_section << client.recvfrom( section_size )[0]
            break if message_section.bytesize == section_size
          end

          data_received << message_section
          total_received += 4 + message_section.bytesize
          break if total_received == incoming_data_size
        end

    yield(data_received, client)
      end
    end
  end

  def send_data(socket, message)
    forward_message = [message.bytesize].pack('N*') + message
    socket.send(forward_message, 0)
    yield(socket)
  end

Why do you yield the socket?

  def get_reply(socket)
    incoming_data_size = socket.recvfrom( 4 )[0].unpack('N*')[0]
    section_size = socket.recvfrom( 4 )[0].unpack('N*')[0]
    reply = ""

    loop do
      reply << socket.recvfrom( section_size )[0]
      break if reply.bytesize == incoming_data_size - 4
    end

    yield(reply, socket)
  end
end

If you use #send then I'd also use #read for reading instead of
#recvfrom. I'd only use that method if I needed the extra options.
Method #read will also block until as many bytes have arrived so you
do not take care of fragment messages yourself.

I'd probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

# could be more sophisticated
Message = Struct.new :bytes

Connection = Struct.new :socket do
  def send(msg)
    b = msg.bytes
    socket.write [b.bytesize].pack 'N'
    socket.write b
    self
  end

  def close
    socket.close
  end

  def each_msg
      until socket.eof?
        size = socket.read(4).unpack('N').first
        msg = socket.read(size)
        yield Message.new(msg)
      end
    end

    self
  end
end

loop do
  Thread.new(server.accept) do |client|
    conn = Connection.new client

    conn.each_msg do |msg|
      yield msg, conn
    end
  end
end

Kind regards

robert

···

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 <lists@ruby-forum.com> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

5

Robert Klemme wrote in post #1072878:

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was.

This is a completely different approach: you do not use a termination
byte any more but transmit the message length beforehand.

        data_received =
          end

  def send_data(socket, message)
    forward_message = [message.bytesize].pack('N*') + message
    socket.send(forward_message, 0)
    yield(socket)
  end

Why do you yield the socket?

So that it can be passed to the get_reply method if the client expects a
reply from the server, or closed by the calling method if it does not.

    yield(reply, socket)
  end
end

If you use #send then I'd also use #read for reading instead of
#recvfrom. I'd only use that method if I needed the extra options.
Method #read will also block until as many bytes have arrived so you
do not take care of fragment messages yourself.

So am I correct in thinking that if I use read I do not need to put it
in a loop because it will always wait until that many bytes have
arrived?

I'd probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

The listen method currently yields to the caller an array with the parts
of the data that has been transmitted to it, and the calling method then
knows how to process this array depending on what it is (the message
type).

# could be more sophisticated
Message = Struct.new :bytes

Connection = Struct.new :socket do
  def send(msg)
    b = msg.bytes
    socket.write [b.bytesize].pack 'N'
    socket.write b
    self
  end

  def close
    socket.close
  end

  def each_msg
      until socket.eof?
        size = socket.read(4).unpack('N').first
        msg = socket.read(size)
        yield Message.new(msg)
      end
    end

    self
  end
end

loop do
  Thread.new(server.accept) do |client|
    conn = Connection.new client

    conn.each_msg do |msg|
      yield msg, conn
    end
  end
end

Kind regards

robert

Thanks for your advice I will be seriously considering the way you
presented this :).

···

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 <lists@ruby-forum.com> > wrote:

--
Posted via http://www.ruby-forum.com/\.

6

Robert Klemme wrote in post #1072878:

This is a completely different approach: you do not use a termination
byte any more but transmit the message length beforehand.

        data_received =
          end

  def send_data(socket, message)
    forward_message = [message.bytesize].pack('N*') + message
    socket.send(forward_message, 0)
    yield(socket)
  end

Why do you yield the socket?

So that it can be passed to the get_reply method if the client expects a
reply from the server, or closed by the calling method if it does not.

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

    yield(reply, socket)
  end
end

If you use #send then I'd also use #read for reading instead of
#recvfrom. I'd only use that method if I needed the extra options.
Method #read will also block until as many bytes have arrived so you
do not take care of fragment messages yourself.

So am I correct in thinking that if I use read I do not need to put it
in a loop because it will always wait until that many bytes have
arrived?

Yes, #read will block. But I confused method pairs: it should be
#read and #write and not #read and #send.

I'd probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

The listen method currently yields to the caller an array with the parts
of the data that has been transmitted to it, and the calling method then
knows how to process this array depending on what it is (the message
type).

No. Method #listen does not yield. It's the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Thanks for your advice I will be seriously considering the way you
presented this :).

You're welcome!

Kind regards

robert

···

On Tue, Aug 21, 2012 at 4:17 AM, tammy roberts2 <lists@ruby-forum.com> wrote:

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 <lists@ruby-forum.com> >> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

7

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

Please correct me if I am wrong, but shouldn't there be no problem as
the yield is inside the thread itself? It seems like the listen method
can not pass the socket to the caller until it is done reading from it.

No. Method #listen does not yield. It's the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Ah good observation. That makes it seem like the first issue you point
out will not come into play though, because the reader thread in listen
is what yields the socket so it seems it will not yield it until it is
done reading from it. Am I maybe misunderstanding something here?

I am going to switch my code to read and write as per your suggestion. I
notice that these methods do not seem to be documented in the Ruby
socket documentation, could you maybe point me towards some
documentation regarding them please? I will post my networking code once
more when it is finished up, again thanks for the tips and advice.

···

--
Posted via http://www.ruby-forum.com/\.

8

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

Please correct me if I am wrong, but shouldn't there be no problem as
the yield is inside the thread itself? It seems like the listen method
can not pass the socket to the caller until it is done reading from it.

Right you are. But this is a quite inefficient approach IMHO: you
must read all announced messages before you can process them. Also,
what happens if there are messages after the initially announced n
messages? Processing will usually be easier and more efficient if you
yield each message as soon as it has arrived. But then again, I do
know nothing about your use case and your requirements.

No. Method #listen does not yield. It's the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Ah good observation. That makes it seem like the first issue you point
out will not come into play though, because the reader thread in listen
is what yields the socket so it seems it will not yield it until it is
done reading from it. Am I maybe misunderstanding something here?

No, apparently not. But see above.

I am going to switch my code to read and write as per your suggestion. I
notice that these methods do not seem to be documented in the Ruby
socket documentation, could you maybe point me towards some
documentation regarding them please? I will post my networking code once
more when it is finished up, again thanks for the tips and advice.

Which methods are not documented? If you refer to #read and #write,
please observe this:

$ irb19 -r socket
Ruby version 1.9.3
irb(main):001:0> TCPSocket.instance_method(:read)
=> #<UnboundMethod: TCPSocket(IO)#read>
irb(main):002:0> TCPSocket.instance_method(:write)
=> #<UnboundMethod: TCPSocket(IO)#write>

In other words: these methods are inherited from class IO.

Kind regards

robert

···

On Tue, Aug 21, 2012 at 3:17 PM, tammy roberts2 <lists@ruby-forum.com> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/