Extending/embedding Ruby

ruby-talk
1

Hi! I would like to extend nfdump in Ruby, which will be probably
challenge for me, because I have never write any extension.
What is core thing is, that nfdump reads binary file with huge amount of
data. This reading and all other job I want to do nfdump which is in C
and dont have library files, so I cant use ffi or something like that.
Nfdump reads file header, makes a structure represents a netflow file
and than it reads n flow records which is represented by struct in which
is another struct and variables.
I need to get this structure to Ruby.
My first question is, if its possible to get struct in which is other
struct etc. to Ruby.
Second is about approach. I have some problems with embedding Ruby
Interpreter in C- example from Programming in Ruby fails on SegFault. So
do you thing it is possible to write this as Ruby extension?
In the case of Ruby extension I didnt see in any example how to get to
Ruby struct which is inicialized in C. I cant inicialize it in Ruby
because its a lot of data and I dont know how many flows are in file. I
cant give structs to array because of memory problem and also I need to
work with record as soon as its read from the file by C.
Your suggestion are highly welcomed. Thanks in advance.

···

--
Posted via http://www.ruby-forum.com/.

2

Hi Matěj P,

I might be able to offer a few starting points. I'll just pre-warn you that I'm not in any way an expert on this stuff!

> Hi! I would like to extend nfdump in Ruby, which will be probably
> challenge for me, because I have never write any extension.
> What is core thing is, that nfdump reads binary file with huge amount of
> data. This reading and all other job I want to do nfdump which is in C
> and dont have library files, so I cant use ffi or something like that.
> Nfdump reads file header, makes a structure represents a netflow file
> and than it reads n flow records which is represented by struct in which
> is another struct and variables.
> I need to get this structure to Ruby.
> My first question is, if its possible to get struct in which is other
> struct etc. to Ruby.

You can wrap up opaque data (eg. C structures) using Data_Wrap_Struct, and it back using Data_Get_Struct. You can then define a number of calls (using rb_define_method) to extract relevant values from your structures.

> Second is about approach. I have some problems with embedding Ruby
> Interpreter in C- example from Programming in Ruby fails on SegFault.

I've personally found "Programming Ruby - The Pragmatic Programmers' Guide", in the chapter "Extending Ruby" to be the best source of information on embedding Ruby. It's not easy- in fact, I've embedded Lua and Python and they were far easier to get working, but once I'd got it going, I found Ruby to be more rewarding (since I vastly prefer it over Lua and Python).

As for the seg fault, you might need to give a bit more info for someone to help you. :slight_smile:

> So
> do you thing it is possible to write this as Ruby extension?
> In the case of Ruby extension I didnt see in any example how to get to
> Ruby struct which is inicialized in C. I cant inicialize it in Ruby
> because its a lot of data and I dont know how many flows are in file.

> I
> cant give structs to array because of memory problem and also I need to
> work with record as soon as its read from the file by C.
> Your suggestion are highly welcomed. Thanks in advance.

Data_Wrap_Struct will probably solve your problem. You can create your object in C, and then wrap it up in a Ruby object.

I hope this helps.

Garth

···

On 19/01/12 08:53, Matěj P. wrote:

3

Thank you Garth for your answer.
I figured out the problem with segfault. It was caused by shared
parametr in make file.
I will try to embeded Ruby in C.
So your opinion is that Data_Wrap_Struct should work also for nested
structs? Do you have any experience with it?

···

--
Posted via http://www.ruby-forum.com/.

4

I quite understand, what you wrote. When I wrap structure I have to
specify a Ruby class. If I understand well this Ruby class will wrap my
C struct so it should have same signature like my struct? How is it
inicialized?

···

--
Posted via http://www.ruby-forum.com/.

5

I tried to use API to make a struct and its working like a charm... But
its not a way I can (want to) you in the real, where I have predefined
structures in C, so I would like to wrap them.

···

--
Posted via http://www.ruby-forum.com/.

6

As you can see from my link on gist, I have that program for
experiments. And I own book you have mentioned.
The good point which I didnt realize is that probably I can wrap my
struct to Ruby class which I already have and I dont have to make
another one.

···

--
Posted via http://www.ruby-forum.com/.

7

Hi Matěj P,

Thank you Garth for your answer.
I figured out the problem with segfault. It was caused by shared
parametr in make file.
I will try to embeded Ruby in C.
So your opinion is that Data_Wrap_Struct should work also for nested
structs? Do you have any experience with it?

Data_Wrap_Struct essentially bundles up any C pointer into a Ruby object. Thus anything that you can create a reference or pointer to, you can use it for. Ruby on its own can't do anything with it, but you can supply methods that act upon it on the C side. Like any pointer though, if you need to clean up the resource it points to, you have to make sure that *something* is performing the required cleanup. So yes, it's perfectly fine for nested structures, if I understand you correctly.

For my own project, I have Data_Wrap_Struct in use under an abstraction layer that I've written. The call that triggers Data_Wrap_Struct in my code is used *extensively*, so in answer to your question, yes, I have direct experience with it. I use it to manage a huge number of different C++ objects, including appropriate cleanup, etc. Some of these objects are highly complex- from the point of view of the Ruby code though, it's just a pointer to an opaque object. I have layered my own type management system on top of that, but you may not need to do this in your case. Once I'd gotten everything working correctly, it has been quite stable.

Garth

···

On 19/01/12 19:55, Matěj P. wrote:

8

Hi Matěj P,

The Ruby class you specify (as a Ruby "VALUE") defines the class of the object that is created. This object then carries around the pointer you supplied it, which can be extracted at a later time. For me, the Ruby class comes from the alloc call for the object, but you can create a class with "rb_define_class" as well.

At this point I would definitely recommend writing a small, detached program, and experimenting with it there, because honestly, some of this stuff is *really* hard, and you can learn a lot from this experimentation (this is what I did initially, reluctantly, and it helped enormously). The book I mentioned before is useful, and has lots of good examples, but failing that, Google searches with terms such as "Ruby embed" or "Ruby extend" might give you something close. As a last resort, the two ruby.h headers are good- search for the text of what you are after and sometimes you turn up useful functions. :slight_smile:

Garth

···

On 20/01/12 03:39, Matěj P. wrote:

I quite understand, what you wrote. When I wrap structure I have to
specify a Ruby class. If I understand well this Ruby class will wrap my
C struct so it should have same signature like my struct? How is it
inicialized?