Custom Protocol

ruby-talk
1

I understand that I've asked a similar question that of custom packets.
But this is different. This is __protocol__.

Has anyone constructed a custom protocol, such as UDT, in Ruby?
What would one use to construct a custom protocol?

Right now, I'm using code from metasploit 3.0 (written in Ruby :-), and
the CStruct2 library included. Some sample stuff I have...

UDT_CTRL_PKT = Rex::Struct2::CStructTemplate.new(
  [ 'uint1v', 'PktType', 0 ],
  [ 'uint8v', 'Type', 0 ],
  [ 'uint16v', 'ExtType', 0 ],
  [ 'uint1', 'X', 0 ],
  [ 'uint30v', 'SubSeqNum', 0 ],
  [ 'uint32v', 'TimeStamp', 0 ]
)

UDT_DATA_PKT = Rex::Struct2::CStructTemplate.new(
  [ 'uint1v', 'PktType', 0 ],
  [ 'uint30v', 'SeqNum', 0 ],
  [ 'uint2v', 'FF', 0 ],
  [ 'uint1v', 'O', 0 ],
  [ 'uint28v', 'MsgNum', 0 ],
  [ 'uint32v', 'TimeStamp', 0 ]
)

What I'm intending to do - what you're **supposed** to do - with the
CStruct2 lib is to create a binary string out of it and write that to a raw socket.

What are some techniques someone would use to accomplish the goal of a
new protocol?

Thanks,
Ari Brown

2

Typically, protocols have state. So I'd start with as clear an
expression of the protocol state machine as you can generate, and then
see about twisting it into a DSL that can be implemented in Ruby.

cjs

···

On 2007-11-29 07:38 +0900 (Thu), thefed wrote:

What are some techniques someone would use to accomplish the goal of a
new protocol?

--
Curt Sampson <cjs@starling-software.com> +81 90 7737 2974
Starling Software <office-admin@starling-software.com>
Mobile sites and software consulting: http://www.starling-software.com

3

I took a very quick look at UDT and I think it's quite interesting. The
basic approach has appeared before (people were discussing the
"long-fat-pipe" problem in TCP even before the WWW came along), but I like
the optimizations in this one.

If I read their material correctly, UDT is based completely on UDP so
there's no need for you to create another protocol at the IP level
(analogous to TCP, UDP, ICMP, etc). That means you don't need raw sockets,
which makes your task considerably easier. You don't need to construct the
IP header either.

Instead, implementing UDT is analogous to rebuilding TCP inside of sets of
related UDP packets. Is there any particular reason you want to do this in
Ruby rather than just wrapping a C or Java implementation?

···

On Nov 28, 2007 5:38 PM, thefed <fedzor@gmail.com> wrote:

I understand that I've asked a similar question that of custom packets.
But this is different. This is __protocol__.

Has anyone constructed a custom protocol, such as UDT, in Ruby?
What would one use to construct a custom protocol?

4

What do you mean by state machine?

I'm looking to implement UDT, which requires custom packet headers and all.
Is there some suggested way to do that?

BTW, creating a DSL should be pretty much standard :slight_smile:

Thanks,
-Ari

···

On Nov 28, 2007, at 6:55 PM, Curt Sampson wrote:

On 2007-11-29 07:38 +0900 (Thu), thefed wrote:

What are some techniques someone would use to accomplish the goal of a
new protocol?

Typically, protocols have state. So I'd start with as clear an
expression of the protocol state machine as you can generate, and then
see about twisting it into a DSL that can be implemented in Ruby.

5

The reason I'd like to do it in Ruby is first for my comprehension of the code. Second, the only implementation so far is in C++, and the bindings there are pretty messy (I think). UDT + Ruby DSL = a lot nicer, IMHO.

As you might've seen, there is a lot of information, such as the sub-sequence number and a variety of other things that are important and pertain to the packets. If I didn't use raw sockets, wouldn't that information be held in the payload?

Thanks,
Ari

···

On Nov 29, 2007, at 8:37 AM, Francis Cianfrocca wrote:

I took a very quick look at UDT and I think it's quite interesting. The
basic approach has appeared before (people were discussing the
"long-fat-pipe" problem in TCP even before the WWW came along), but I like
the optimizations in this one.

If I read their material correctly, UDT is based completely on UDP so
there's no need for you to create another protocol at the IP level
(analogous to TCP, UDP, ICMP, etc). That means you don't need raw sockets,
which makes your task considerably easier. You don't need to construct the
IP header either.

Instead, implementing UDT is analogous to rebuilding TCP inside of sets of
related UDP packets. Is there any particular reason you want to do this in
Ruby rather than just wrapping a C or Java implementation?

6

For a typical protocol, you might have states such as:

    - Idle
    - Sent start message, waiting for response
    - Connection running but idle
    - Message sent, waiting for response,

And you'd move between various states based on the packets you send.

I will send you (off-list) an example of a state machine described in a
DSL embedded in Ruby.

cjs

···

On 2007-11-29 12:18 +0900 (Thu), thefed wrote:

On Nov 28, 2007, at 6:55 PM, Curt Sampson wrote:

>Typically, protocols have state. So I'd start with as clear an
>expression of the protocol state machine as you can generate, and then
>see about twisting it into a DSL that can be implemented in Ruby.

What do you mean by state machine?

--
Curt Sampson <cjs@starling-software.com> +81 90 7737 2974
Starling Software <office-admin@starling-software.com>
Mobile sites and software consulting: http://www.starling-software.com

7

I'm planning to look the implementation over and give it a try. Maybe even
wrap it as a Ruby extension. Based on my limited reading of the spec, all of
the data needed to relate packets to streams goes into the UDP packets. I'm
still seeing no need for you to use raw sockets.

···

On Nov 29, 2007 4:18 PM, thefed <fedzor@gmail.com> wrote:

On Nov 29, 2007, at 8:37 AM, Francis Cianfrocca wrote:

> I took a very quick look at UDT and I think it's quite interesting.
> The
> basic approach has appeared before (people were discussing the
> "long-fat-pipe" problem in TCP even before the WWW came along), but
> I like
> the optimizations in this one.
>
> If I read their material correctly, UDT is based completely on UDP so
> there's no need for you to create another protocol at the IP level
> (analogous to TCP, UDP, ICMP, etc). That means you don't need raw
> sockets,
> which makes your task considerably easier. You don't need to
> construct the
> IP header either.
>
> Instead, implementing UDT is analogous to rebuilding TCP inside of
> sets of
> related UDP packets. Is there any particular reason you want to do
> this in
> Ruby rather than just wrapping a C or Java implementation?

The reason I'd like to do it in Ruby is first for my comprehension of
the code. Second, the only implementation so far is in C++, and the
bindings there are pretty messy (I think). UDT + Ruby DSL = a lot
nicer, IMHO.

As you might've seen, there is a lot of information, such as the sub-
sequence number and a variety of other things that are important and
pertain to the packets. If I didn't use raw sockets, wouldn't that
information be held in the payload?

8

I would like to see that example on-list, or at least be part of the off-list
conversation, if possible :-).

Thanks,

Jesus.

···

On Nov 29, 2007 4:45 AM, Curt Sampson <cjs@cynic.net> wrote:

On 2007-11-29 12:18 +0900 (Thu), thefed wrote:

> On Nov 28, 2007, at 6:55 PM, Curt Sampson wrote:
>
> >Typically, protocols have state. So I'd start with as clear an
> >expression of the protocol state machine as you can generate, and then
> >see about twisting it into a DSL that can be implemented in Ruby.
>
> What do you mean by state machine?

Finite-state machine - Wikipedia

For a typical protocol, you might have states such as:

    - Idle
    - Sent start message, waiting for response
    - Connection running but idle
    - Message sent, waiting for response,

And you'd move between various states based on the packets you send.

I will send you (off-list) an example of a state machine described in a
DSL embedded in Ruby.

9

Thanks! Thats a nice protocol you have!

For a typical protocol, you might have states such as:

    - Idle
    - Sent start message, waiting for response
    - Connection running but idle
    - Message sent, waiting for response,

And you'd move between various states based on the packets you send.

Ah, you mean a high level state. In UDT, since it is used to make large data transfer faster, I still need to have lower-level control.

My current bottleneck is that of the packets. Is there a recommended way to go about constructing them?

BTW, the website: http://udt.sourceforge.net/

Thanks,
-Ari Brown

···

On Nov 28, 2007, at 10:45 PM, Curt Sampson wrote:

10

Where would you recommend, then, including the excess information? The Payload?

Also, is there a way to get C++ libraries loaded into C?

Thanks again,
ari

···

On Nov 29, 2007, at 8:37 PM, Francis Cianfrocca wrote:

I'm planning to look the implementation over and give it a try. Maybe even
wrap it as a Ruby extension. Based on my limited reading of the spec, all of
the data needed to relate packets to streams goes into the UDP packets. I'm
still seeing no need for you to use raw sockets.

11

Where would you recommend, then, including the excess information? The Payload?

Also, is there a way to get C++ libraries loaded into C?

Thanks again,
ari

···

On Nov 29, 2007, at 8:37 PM, Francis Cianfrocca wrote:

I'm planning to look the implementation over and give it a try. Maybe even
wrap it as a Ruby extension. Based on my limited reading of the spec, all of
the data needed to relate packets to streams goes into the UDP packets. I'm
still seeing no need for you to use raw sockets.

12

> I'm planning to look the implementation over and give it a try.
> Maybe even
> wrap it as a Ruby extension. Based on my limited reading of the
> spec, all of
> the data needed to relate packets to streams goes into the UDP
> packets. I'm
> still seeing no need for you to use raw sockets.

Where would you recommend, then, including the excess information?
The Payload?

According to the spec, the header of a UDT data packet is 16 bytes long.
That's going to the first 16 bytes of the UDP packet's data. You don't need
to do anything with the "UDP header" that IP looks at. This implementation
is completely in userland and not in IP-land, which is a positive thing.
(They could have defined another IP-level protocol instead, which would have
been a pain.)

However, I'm considerably less impressed with this protocol after reading
the spec, and I decided not to spend any time on it. There is no
straightforward way to do end-to-end encryption. They've redone from scratch
all of the painstaking work of congestion avoidance, etc. that has been
fine-tuned in TCP for decades. And who knows what kind of security
vulnerabilities they've introduced. For one thing, packet numbers are
sequential, a problem that TCP got rid of years ago.

Also, is there a way to get C++ libraries loaded into C?

Look at the C wrapper in Ruby/EventMachine, which includes an extension
written in C++.

···

On Nov 29, 2007 9:52 PM, thefed <fedzor@gmail.com> wrote:

On Nov 29, 2007, at 8:37 PM, Francis Cianfrocca wrote:

Thanks again,
ari

13

According to the spec, the header of a UDT data packet is 16 bytes long.
That's going to the first 16 bytes of the UDP packet's data. You don't need
to do anything with the "UDP header" that IP looks at. This implementation
is completely in userland and not in IP-land, which is a positive thing.
(They could have defined another IP-level protocol instead, which would have
been a pain.)

That's good news. Does this mean their alternate information is contained in the UDP Payload?
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

However, I'm considerably less impressed with this protocol after reading
the spec, and I decided not to spend any time on it. There is no
straightforward way to do end-to-end encryption. They've redone from scratch
all of the painstaking work of congestion avoidance, etc. that has been
fine-tuned in TCP for decades. And who knows what kind of security
vulnerabilities they've introduced. For one thing, packet numbers are
sequential, a problem that TCP got rid of years ago.

I saw the sequential number issue and the cryptography problem, but I figured two things.
a) These people aren't **that** stupid, so they probably know someway to protect against injection.
b) I was going to try to use some fast one-way encryption algorithm, and use public keys and a central server, if I bundled this with some other software.
b 1) Or i was going to wrap something secure and simple around it, and figure out of to get the symmetric key across the net.

Thanks for your help anyways,
Ari

···

On Nov 30, 2007, at 12:09 AM, Francis Cianfrocca wrote:

14

Strictly out of curiosity, what's your need for this protocol? Are you
running up against the long-fat-pipe limitation in TCP?

···

On Nov 30, 2007 7:43 AM, thefed <fedzor@gmail.com> wrote:

On Nov 30, 2007, at 12:09 AM, Francis Cianfrocca wrote:

> According to the spec, the header of a UDT data packet is 16 bytes
> long.
> That's going to the first 16 bytes of the UDP packet's data. You
> don't need
> to do anything with the "UDP header" that IP looks at. This
> implementation
> is completely in userland and not in IP-land, which is a positive
> thing.
> (They could have defined another IP-level protocol instead, which
> would have
> been a pain.)

That's good news. Does this mean their alternate information is
contained in the UDP Payload?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> However, I'm considerably less impressed with this protocol after
> reading
> the spec, and I decided not to spend any time on it. There is no
> straightforward way to do end-to-end encryption. They've redone
> from scratch
> all of the painstaking work of congestion avoidance, etc. that has
> been
> fine-tuned in TCP for decades. And who knows what kind of security
> vulnerabilities they've introduced. For one thing, packet numbers are
> sequential, a problem that TCP got rid of years ago.

I saw the sequential number issue and the cryptography problem, but I
figured two things.
a) These people aren't **that** stupid, so they probably know someway
to protect against injection.
b) I was going to try to use some fast one-way encryption algorithm,
and use public keys and a central server, if I bundled this with some
other software.
b 1) Or i was going to wrap something secure and simple around it,
and figure out of to get the symmetric key across the net.

15

I don't actually need this for anything. Just something that I wanted to do. I'm not running up against anything, I just wanted to implement something that would make data transfer faster, since connectivity and hardware are at a low for me.

Is there some alternate to fast data transfer that you could suggest?

Thanks again,
Ari

···

On Nov 30, 2007, at 10:35 AM, Francis Cianfrocca wrote:

Strictly out of curiosity, what's your need for this protocol? Are you
running up against the long-fat-pipe limitation in TCP?

16

If you're bandwidth-constrained, then I would doubt that replacing TCP will
help you. Especially with a pure-Ruby implementation. Still, if it weren't
for the unknown-unknown issues, I'd think this was a pretty interesting
protocol. (By that, I mean that we have no idea what bizarre edge-conditions
this protocol will create in the network, that have already been thoroughly
shaken out of TCP.) Thanks for bringing it to our attention.

···

On Nov 30, 2007 3:49 PM, thefed <fedzor@gmail.com> wrote:

On Nov 30, 2007, at 10:35 AM, Francis Cianfrocca wrote:

> Strictly out of curiosity, what's your need for this protocol? Are you
> running up against the long-fat-pipe limitation in TCP?

I don't actually need this for anything. Just something that I wanted
to do. I'm not running up against anything, I just wanted to
implement something that would make data transfer faster, since
connectivity and hardware are at a low for me.

Is there some alternate to fast data transfer that you could suggest?