Wrapping a C struct[] constant

I have some C code with a struct definition

struct flag_str {
  unsigned int val;
  const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
  { FIEMAP_EXTENT_LAST, "last" },
  { FIEMAP_EXTENT_UNKNOWN, "unkown" },
  { FIEMAP_EXTENT_DELALLOC, "delalloc" },
  { FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
  { FIEMAP_EXTENT_SECONDARY, "secondary" },
  { FIEMAP_EXTENT_NET, "net" },
  { FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
  { FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
  { FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
  { FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
  { FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
  { FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
  { FIEMAP_EXTENT_MERGED, "merged" },
  { 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

martin

Martin DeMello wrote:

I have some C code with a struct definition

struct flag_str {
  unsigned int val;
  const char *str;
};

and an inline array

struct flag_str extent_flags = {
  { FIEMAP_EXTENT_LAST, "last" },
  { FIEMAP_EXTENT_UNKNOWN, "unkown" },
  { FIEMAP_EXTENT_DELALLOC, "delalloc" },
  { FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
  { FIEMAP_EXTENT_SECONDARY, "secondary" },
  { FIEMAP_EXTENT_NET, "net" },
  { FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
  { FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
  { FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
  { FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
  { FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
  { FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
  { FIEMAP_EXTENT_MERGED, "merged" },
  { 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

How/where are you *using* it?

FFI is generally a good solution, although it
would probably be best to actually define the
Array on the Ruby side and pass it in as needed.

Eero

···

--
Magic is insufficiently advanced technology.
--
Posted via http://www.ruby-forum.com/\.

Even using FFI, you'll need to define the wrapping between C constant to
Ruby constant, and using Ruby's API, you'll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn't a faster way to do it.

Jason

···

On Fri, Jun 12, 2009 at 7:42 AM, Eero Saynatkari <ruby-ml@kittensoft.org>wrote:

Martin DeMello wrote:
> I have some C code with a struct definition
>
> struct flag_str {
> unsigned int val;
> const char *str;
> };
>
> and an inline array
>
> struct flag_str extent_flags = {
> { FIEMAP_EXTENT_LAST, "last" },
> { FIEMAP_EXTENT_UNKNOWN, "unkown" },
> { FIEMAP_EXTENT_DELALLOC, "delalloc" },
> { FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
> { FIEMAP_EXTENT_SECONDARY, "secondary" },
> { FIEMAP_EXTENT_NET, "net" },
> { FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
> { FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
> { FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
> { FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
> { FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
> { FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
> { FIEMAP_EXTENT_MERGED, "merged" },
> { 0, NULL },
> };
>
> What's the simplest way to expose that array as a ruby constant (other
> than just copy/pasting the data, of course :))?

How/where are you *using* it?

FFI is generally a good solution, although it
would probably be best to actually define the
Array on the Ruby side and pass it in as needed.

Eero
--
Magic is insufficiently advanced technology.
--
Posted via http://www.ruby-forum.com/\.

Using it (or want to use it!) from the ruby side to generate a hash of
constant => name and use that to unpack and display flag settings.

m.

···

On Fri, Jun 12, 2009 at 5:12 PM, Eero Saynatkari<ruby-ml@kittensoft.org> wrote:

Martin DeMello wrote:

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags = {
{ FIEMAP_EXTENT_LAST, "last" },
{ FIEMAP_EXTENT_UNKNOWN, "unkown" },
{ FIEMAP_EXTENT_DELALLOC, "delalloc" },
{ FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
{ FIEMAP_EXTENT_SECONDARY, "secondary" },
{ FIEMAP_EXTENT_NET, "net" },
{ FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
{ FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
{ FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
{ FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
{ FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
{ FIEMAP_EXTENT_MERGED, "merged" },
{ 0, NULL },
};

What's the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

How/where are you *using* it?

I was afraid of that :frowning: The docs are a bit on the sketchy side too. I
guess some sort of code generation would be the way to go then (swig
seems like overkill). Will give cgenerator a look.

martin

···

On Fri, Jun 12, 2009 at 7:01 PM, Jason Roelofs<jameskilton@gmail.com> wrote:

Even using FFI, you'll need to define the wrapping between C constant to
Ruby constant, and using Ruby's API, you'll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn't a faster way to do it.

Martin DeMello wrote:

Using it (or want to use it!) from the ruby side to generate a hash of
constant => name and use that to unpack and display flag settings.

If all you want is this data structure in ruby, you could parse the source (gccxml?), and then you have a pure ruby library that defines this hash. Define the parse->xml>rb part as a rake task if you need to keep up to date with the source files.

···

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

Martin DeMello wrote:

Even using FFI, you'll need to define the wrapping between C constant to
Ruby constant, and using Ruby's API, you'll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn't a faster way to do it.

I was afraid of that :frowning: The docs are a bit on the sketchy side too. I
guess some sort of code generation would be the way to go then (swig
seems like overkill). Will give cgenerator a look.

martin

Cgenerator will work, and might be a good idea if the source (the list of values and strings) changes from time to time, and you would rather have your program regenerate the ruby extension (and data structure) automatically.

Here's how it works, assuming your source files are flag.c and flag.h:

$ ls
flag.c flag.h flag.rb
$ cat flag.h
struct flag_str {
         unsigned int val;
         const char *str;
};

extern struct flag_str extent_flags;

typedef enum {
     FIEMAP_EXTENT_LAST = 1,
     FIEMAP_EXTENT_UNKNOWN,
     FIEMAP_EXTENT_DELALLOC,
     FIEMAP_EXTENT_NO_BYPASS,
     FIEMAP_EXTENT_SECONDARY,
     FIEMAP_EXTENT_NET,
     FIEMAP_EXTENT_DATA_COMPRESSED,
     FIEMAP_EXTENT_DATA_ENCRYPTED,
     FIEMAP_EXTENT_NOT_ALIGNED,
     FIEMAP_EXTENT_DATA_INLINE,
     FIEMAP_EXTENT_DATA_TAIL,
     FIEMAP_EXTENT_UNWRITTEN,
     FIEMAP_EXTENT_MERGED
} FIEMAP_EXTENT;

$ cat flag.c
#include "flag.h"

#ifndef NULL
#define NULL 0
#endif

struct flag_str extent_flags = {
         { FIEMAP_EXTENT_LAST, "last" },
         { FIEMAP_EXTENT_UNKNOWN, "unkown" },
         { FIEMAP_EXTENT_DELALLOC, "delalloc" },
         { FIEMAP_EXTENT_NO_BYPASS, "no_bypass" },
         { FIEMAP_EXTENT_SECONDARY, "secondary" },
         { FIEMAP_EXTENT_NET, "net" },
         { FIEMAP_EXTENT_DATA_COMPRESSED, "data_compressed" },
         { FIEMAP_EXTENT_DATA_ENCRYPTED, "data_encrypted" },
         { FIEMAP_EXTENT_NOT_ALIGNED, "not_aligned" },
         { FIEMAP_EXTENT_DATA_INLINE, "data_inline" },
         { FIEMAP_EXTENT_DATA_TAIL, "data_tail" },
         { FIEMAP_EXTENT_UNWRITTEN, "unwritten" },
         { FIEMAP_EXTENT_MERGED, "merged" },
         { 0, NULL },
};

$ cat flag.rb
#!/usr/bin/env ruby

require 'cgen/cgen'
require 'fileutils'

module Flag; end

# Generate the extension source code.
lib = CGenerator::Library.new "flag_lib"
lib.include "flag.h"

lib.define_c_singleton_method(Flag, :extent_flags).instance_eval {
   # no args
   body %{\
     VALUE a;
     struct flag_str *pfs;

     a = rb_ary_new();
     for (pfs = extent_flags; pfs->val; pfs++) {
       rb_ary_push(a,
         rb_ary_new3(2,
           INT2NUM(pfs->val),
           rb_str_new2(pfs->str)
       ));
     }
   }
   returns "a"
}

# Normally, this isn't needed, but we need to set up the external files
# as symlinks in this dir before calling commit.
FileUtils.mkdir_p("flag_lib")

# Put links to sources where they will be found.
Dir.chdir "flag_lib" do
   %w{ flag.h flag.c }.each do |file|
     FileUtils.ln_s("../#{file}", file) rescue nil
   end
end

# Write and build
lib.commit

# Use the library
Flag::EXTENT_FLAGS = Hash[*Flag.extent_flags.flatten]
p Flag::EXTENT_FLAGS

$ ruby flag.rb
{5=>"secondary", 11=>"data_tail", 6=>"net", 12=>"unwritten", 1=>"last", 7=>"data_compressed", 13=>"merged", 2=>"unkown", 8=>"data_encrypted", 3=>"delalloc", 9=>"not_aligned", 4=>"no_bypass", 10=>"data_inline"}

···

On Fri, Jun 12, 2009 at 7:01 PM, Jason Roelofs<jameskilton@gmail.com> wrote:

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

Thanks, that looks great. I have to say, I'm a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

martin

···

On Sat, Jun 13, 2009 at 2:52 AM, Joel VanderWerf<vjoel@path.berkeley.edu> wrote:

Cgenerator will work, and might be a good idea if the source (the list of
values and strings) changes from time to time, and you would rather have
your program regenerate the ruby extension (and data structure)
automatically.

Here's how it works, assuming your source files are flag.c and flag.h:

Could you give an example of how this would work? I'm having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would map
to said C struct.

Jason

···

On Mon, Jun 15, 2009 at 5:13 AM, Martin DeMello <martindemello@gmail.com>wrote:

On Sat, Jun 13, 2009 at 2:52 AM, Joel VanderWerf<vjoel@path.berkeley.edu> > wrote:
>
> Cgenerator will work, and might be a good idea if the source (the list of
> values and strings) changes from time to time, and you would rather have
> your program regenerate the ruby extension (and data structure)
> automatically.
>
> Here's how it works, assuming your source files are flag.c and flag.h:

Thanks, that looks great. I have to say, I'm a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

martin

Martin DeMello wrote:

another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

Would this be like swig in reverse? Input a ruby class with type annotations, output C header, accessors, etc? I'd like to do that for the special case of BitStruct classes (fixed-length, packed fields, usually numeric or character, stored in a ruby String using pack/unpack), so that once you define a network protocol or other binary format in ruby, you can share your code with C programmers (and via swig with other languages). That seems feasible, and I did actually do this with a predecessor of BitStruct, using cgen. But how would you handle a more general case, like a class with a hash or array attribute?

CShadow is similar, but instead of inheriting from String like BitStruct, its instances are T_DATA (ruby objects with a blob that can only be accessed from C). Unlike a BitStruct, this blob is not intended for sending outside of the ruby+C world. For example, a "self" pointer is automatically added at the beginning of the blob.

Including CShadow in your class lets you define the structure of the T_DATA object in ruby, in terms of accessors of various types. When your code runs, it uses cgenerator to build an extension that takes care of all the boiler plate functions: mark, free, accessors with type checking/conversion, alloc, init, inspect, marshal, and yaml. Plus it handles inheritance and sets up introspection methods. [See example below.]

CShadow also gives you tools to distribute source code in separate .c and .h files, so that when you make a minor change in the definitions embedded in ruby, you minimize how much needs to be recompiled. This works well enough that even if the generated code is a few Mb, you can let cgenerator do its full generate/make cycle every time you run your program, without noticing much delay.

You can add C functions to a CShadow class inline in ruby source just as with cgenerator. But CShadow only supports a few kinds of attributes "out of the box": ruby object references, native scalar data of various sizes, and native pointers to char, double, array of double, etc. You can add new attr types with a little work, but it's far from automatic.

These are just two special case solutions. I'm not seeing what a general solution would look like. What kinds of classes/attributes are there in your use cases?

···

--------------------------------
require 'cgen/cshadow'

class Parent
   include CShadow

   shadow_attr_accessor :ruby_str => String # type-checked VALUE type
   shadow_attr_accessor :c_int => "int c_int"
end

class Child < Parent
   shadow_attr_accessor :c_str => "char *c_str"
   shadow_attr_accessor :obj => Object # VALUE type
end

Parent.commit
   # we're done adding attrs and methods, so make.

x = Child.new
x.ruby_str = "foo"
x.c_str = "bar"
x.obj = [1,2,3]
x.c_int = 3

p x

CShadow.allow_yaml
y x

__END__

Output:

#<Child:0xb79716cc ruby_str="foo", c_int=3, c_str="bar", obj=[1, 2, 3]>
--- !path.berkeley.edu,2006/cshadow:Child
ruby_str: foo
c_int: 3
c_str: bar
obj:
- 1
- 2
- 3

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

Martin DeMello wrote:

another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator's
CShadow is all about?). Are these not common use cases?

Would this be like swig in reverse? Input a ruby class with type
annotations, output C header, accessors, etc?

I actually meant like a simplified swig - you have a C struct
(arbitrary nesting of structs and arrays, actually), and you have a
ruby class that you define a typemap for and then have it
serialise/deserialise. Basically you know that you can mirror C
structures in ruby with a bit of gruntwork, all you need is a tool to
do the gruntwork for you

I'd like to do that for the
special case of BitStruct classes (fixed-length, packed fields, usually
numeric or character, stored in a ruby String using pack/unpack), so that
once you define a network protocol or other binary format in ruby, you can
share your code with C programmers (and via swig with other languages). That
seems feasible, and I did actually do this with a predecessor of BitStruct,
using cgen. But how would you handle a more general case, like a class with
a hash or array attribute?

Network protocols were one of the use cases I was thinking of, except
that you usually see them defined in C. I wanted the ability to use
the structs from ruby, without going through C the way
data_wrap_struct makes you do. Not asking for magic, just the ability
to make the common case trivial.

Including CShadow in your class lets you define the structure of the T_DATA
object in ruby, in terms of accessors of various types. When your code runs,
it uses cgenerator to build an extension that takes care of all the boiler
plate functions: mark, free, accessors with type checking/conversion, alloc,
init, inspect, marshal, and yaml. Plus it handles inheritance and sets up
introspection methods. [See example below.]

Okay, that's most of what I was looking for :slight_smile: If it can convert an
array of structs into a ruby array, that's *all* of what I'm looking
for. Need to look through the CGenerator docs some more - great piece
of work!

martin

···

On Mon, Jun 15, 2009 at 11:00 PM, Joel VanderWerf<vjoel@path.berkeley.edu> wrote:

Yes, I was simply looking for something like

struct foo {
  int bar;
  char *baz;
}

cFoo = rb_define_class('cFoo')
rb_map_struct(cFoo, foo)
rb_accessor(cFoo, 'int', 'bar', bar)
rb_accessor(cFoo, 'str', 'bar', bar)

VALUE rb_function() {
  foo* a = some_function_call()
  return rb_deserialize_data(foo)
}

and from ruby

foo = rb_function
foo.bar
foo.baz

martin

···

On Mon, Jun 15, 2009 at 7:13 PM, Jason Roelofs<jameskilton@gmail.com> wrote:

Could you give an example of how this would work? I'm having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would map
to said C struct.

You are attacking your problem from the wrong end. The problem is that C
doesn't have reflection so there is no way the rb_map_struct concept would
ever work. C won't tell you what the members are so you can't map it to
anything.

However, working in the other direction - you could use the C preprocessor
to build both your structs and the appropriate C code to define your ruby
class you want for each struct. It's not pretty but it does work - have a
look here for a nice defintion of what you are looking for

John

···

On Mon, Jun 15, 2009 at 11:11 AM, Martin DeMello <martindemello@gmail.com>wrote:

On Mon, Jun 15, 2009 at 7:13 PM, Jason Roelofs<jameskilton@gmail.com> > wrote:
>
> Could you give an example of how this would work? I'm having a hard time
> picturing how such a thing would be possible to have done automagically.
> There has to be something somewhere that defines how a Ruby class would
map
> to said C struct.

Yes, I was simply looking for something like

struct foo {
int bar;
char *baz;
}

cFoo = rb_define_class('cFoo')
rb_map_struct(cFoo, foo)
rb_accessor(cFoo, 'int', 'bar', bar)
rb_accessor(cFoo, 'str', 'bar', bar)

VALUE rb_function() {
foo* a = some_function_call()
return rb_deserialize_data(foo)
}

and from ruby

foo = rb_function
foo.bar
foo.baz

martin

Martin DeMello wrote:

If it can convert an array of structs into a ruby array, that's *all* of what I'm looking for.

How is the length stored? A separate field? Or is it fixed length?

···

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

In true C style, you need to pass in a pointer in which the length is
returned :slight_smile: I'm not thinking of anything magical; I'd expect to pass
in the length as a parameter.

martin

···

On Tue, Jun 16, 2009 at 1:55 AM, Joel VanderWerf<vjoel@path.berkeley.edu> wrote:

Martin DeMello wrote:

If it can convert an array of structs into a ruby array, that's *all* of
what I'm looking for.

How is the length stored? A separate field? Or is it fixed length?