Problem wrapping the C++ STL

ruby-talk
#1

Hi all,

Is it possible to wrap the C++ STL stuff? Say, for example, vector?

#include "ruby.h"
#include <vector>
using namespace std;

struct cvector{
    vector<VALUE> array(0);
};

void Init_cvector(){
    VALUE cVector = rb_define_class("CVector", rb_cObject);
}

When I try to compile this (using g++ on OS X) I get:

/usr/local/lib/ruby/1.8/i686-darwin8.9.1/intern.h:207: error: use of enum 'rb_thread_status' without previous declaration
/usr/local/lib/ruby/1.8/i686-darwin8.9.1/intern.h:207: error: invalid type in declaration before ';' token
cvector.c:6: error: expected identifier before numeric constant
cvector.c:6: error: expected ',' or '...' before numeric constant
cvector.c: In function 'void Init_cvector()':
cvector.c:10: warning: unused variable 'cVector'
make: *** [cvector.o] Error 1

Thanks,

Dan

#2

Is it possible to wrap the C++ STL stuff? Say, for example, vector?

Yes. I wrapped set for instance

/usr/local/lib/ruby/1.8/i686-darwin8.9.1/intern.h:207: error: use of
enum 'rb_thread_status' without previous declaration

This is not a C++ bug. This is a bug in 1.8.6-p36, for which I know no
workaround. rb_thread_status is defined in node.h but ruby.h inclu

cvector.c:6: error: expected identifier before numeric constant
cvector.c:6: error: expected ',' or '...' before numeric constant

You're trying to initialize a struct field. You need to do
struct cvector {
  vector<VALUE> array;
};
an resize the array if needed.

Note that you can wrap std::vector directly (no need to go through an
intermediate cvector structure)

Sylvain

#3

Have you tried swig? Gonzalo Garramuño has done some work to make the
ruby<->STL interface better (faster and more STL functionality). I've
also done similar python<->STL modifications for swig. I'd love to
see all of this completed with full STL functionality (containers,
iterators, algorithms) and the big-O performance preserved. C++ STL
is quite rich w/ a lot of thought put into big-O performance (great
when dealing with large datasets).

···

On 6/18/07, Daniel Berger <djberg96@gmail.com> wrote:

Hi all,

Is it possible to wrap the C++ STL stuff? Say, for example, vector?

#include "ruby.h"
#include <vector>
using namespace std;

struct cvector{
    vector<VALUE> array(0);
};

void Init_cvector(){
    VALUE cVector = rb_define_class("CVector", rb_cObject);
}

When I try to compile this (using g++ on OS X) I get:

/usr/local/lib/ruby/1.8/i686-darwin8.9.1/intern.h:207: error: use of
enum 'rb_thread_status' without previous declaration
/usr/local/lib/ruby/1.8/i686-darwin8.9.1/intern.h:207: error: invalid
type in declaration before ';' token
cvector.c:6: error: expected identifier before numeric constant
cvector.c:6: error: expected ',' or '...' before numeric constant
cvector.c: In function 'void Init_cvector()':
cvector.c:10: warning: unused variable 'cVector'
make: *** [cvector.o] Error 1

Thanks,

Dan

#4

Hi,

At Mon, 18 Jun 2007 23:15:26 +0900,
Daniel Berger wrote in [ruby-talk:256045]:

Is it possible to wrap the C++ STL stuff? Say, for example, vector?

In general, C++ isn't supported in 1.8.

···

--
Nobu Nakada

#5

Absolutely. But I suggest you download the latest swig from svn, as
it contains a lot of enhancements for this.

This is how you do it in swig.

---- mystl.i ----

%module mystl
%include std_vector.i

// GC_VALUE is a special class that knows how to mark itself.
%template(ValueVector) std::vector< GC_VALUE >;

----- end mystl.i ----

On a console:

$ swig -ruby -c++ mystl.i # creates mystl_wrap.cxx

Compile:

# using extconf.rb (recommended) or...

···

On Jun 18, 11:15 am, Daniel Berger <djber...@gmail.com> wrote:

Is it possible to wrap the C++ STL stuff? Say, for example, vector?

#
# posix
#
$ g++ -fPIC -DPIC -O2 mystl_wrap.cxx -o mystl_wrap.so -lruby -lstdc++

#
# Windows
#
$ vcvars32.bat
$ cl.exe -MD -LD -O2 mystl_wrap.cxx -o mystl_wrap.so /link msvcrt-
ruby1.8.dll

------ usage -----
require 'mystl'
include Mystl

a = ValueVector.new
puts a.methods # you'll get iterators, most array methods, etc.

-------------------

#6

g++ -Wall -I. -I/opt/lib/ruby/1.8/i686-darwin8.9.1
-I/opt/lib/ruby/1.8/i686-darwin8.9.1 -I. -fno-common -g -O2 -pipe
-fno-common -c cvector.c
cvector.c: In function 'void Init_cvector()':
cvector.c:10: error: invalid conversion from 'VALUE (*)(int, VALUE*,
VALUE)' to 'VALUE (*)(...)'
cvector.c:10: error: initializing argument 3 of 'void
rb_define_method(VALUE, const char*, VALUE (*)(...), int)'
make: *** [cvector.o] Error 1

C++ is more pissy than C about type convertions. You have to use the
RUBY_METHOD_FUNC macro to convert cvector_init into the right type:

  rb_define_method(cVector, "initialize",
      RUBY_METHOD_FUNC(cvector_init), -1);

> Note that you can wrap std::vector directly (no need to go through an
> intermediate cvector structure)

I guess I could store it as an instance variable within the constructor
and refer back to it that way. I don't think it's faster, though. Or,
did you have something else in mind?

The idea is *not* to use a cvector structure. Since you'll have to provide
an alloc/free method pair anyway, allocate std::vector in them. Check
value_set_alloc/value_set_free in
http://www.laas.fr/~sjoyeux/darcs/utilrb/ext/value_set.cc

Sylvain

#7

> g++ -Wall -I. -I/opt/lib/ruby/1.8/i686-darwin8.9.1
> -I/opt/lib/ruby/1.8/i686-darwin8.9.1 -I. -fno-common -g -O2 -pipe
> -fno-common -c cvector.c
> cvector.c: In function 'void Init_cvector()':
> cvector.c:10: error: invalid conversion from 'VALUE (*)(int, VALUE*,
> VALUE)' to 'VALUE (*)(...)'
> cvector.c:10: error: initializing argument 3 of 'void
> rb_define_method(VALUE, const char*, VALUE (*)(...), int)'
> make: *** [cvector.o] Error 1

C++ is more pissy than C about type convertions. You have to use the
RUBY_METHOD_FUNC macro to convert cvector_init into the right type:

  rb_define_method(cVector, "initialize",
      RUBY_METHOD_FUNC(cvector_init), -1);

Ah, thanks.

> > Note that you can wrap std::vector directly (no need to go through an
> > intermediate cvector structure)

> I guess I could store it as an instance variable within the constructor
> and refer back to it that way. I don't think it's faster, though. Or,
> did you have something else in mind?

The idea is *not* to use a cvector structure. Since you'll have to provide
an alloc/free method pair anyway, allocate std::vector in them. Check
value_set_alloc/value_set_free inhttp://www.laas.fr/~sjoyeux/darcs/utilrb/ext/value_set.cc

Interesting, thank you. For kicks, I tried to compile your source code
on my Solaris 10 box (after installing boost). It built (with some
warnings), but I can't get it to load.

Here's the extconf.rb file I used:

require 'mkmf'
dir_config('set2')

case RUBY_PLATFORM
   when /sunos|solaris/
      CONFIG['CC'] = 'CC'
   when /mswin/i
      CONFIG["COMPILE_C"].sub!(/-Tc/, '-Tp')
   else
      CONFIG['CC'] = 'g++ -Wall'
end

create_makefile('set2')

Here was the result of the build step:

djberge-/export/home/djberge/workspace/set/ext-635>ruby extconf.rb --
with-set2-include=/opt/csw/include
creating Makefile

djberge-/export/home/djberge/workspace/set/ext-636>make
CC -I. -I/usr/local/lib/ruby/1.8/sparc-solaris2.10 -I/usr/local/lib/
ruby/1.8/sparc-solaris2.10 -I. -I/opt/csw/include -KPIC -dalign -fns -
xbuiltin=%all -xlibmil -xtarget=ultra2e -xO5 -xipo -c set.c
"set.c", line 349: Warning (Anachronism): Formal argument 3 of type
extern "C" unsigned long(*)(...) in call to rb_iterate(extern "C"
unsigned long(*)(unsigned long), unsigned long, extern "C" unsigned
long(*)(...), unsigned long) is being passed unsigned long(*)(...).
"set.c", line 369: Warning (Anachronism): Formal argument 2 of type
extern "C" unsigned long(*)(unsigned long) in call to
rb_define_alloc_func(unsigned long, extern "C" unsigned long(*)
(unsigned long)) is being passed unsigned long(*)(unsigned long).
2 Warning(s) detected.
ld -G -o set2.so set.o -L'.' -L'/usr/local/lib' -R'/usr/local/lib' -
L. -lrt -lpthread -ldl -lcrypt -lm -lc

Ok, a couple warnings. I proceed to try to "require 'set2'" and I get
this:

djberge-/export/home/djberge/workspace/set/ext-637>ruby test.rb
/export/home/djberge/workspace/set/ext/set2.so: ld.so.1: ruby: fatal:
relocation error: file /export/home/djberge/workspace/set/ext/set2.so:
symbol __1cDstdJbad_allocG__vtbl_: referenced symbol not found - /
export/home/djberge/workspace/set/ext/set2.so (LoadError)
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/
custom_require.rb:27:in `require'
        from test.rb:2

This was with Ruby 1.8.6-p38 (today's svn checkout of the 1.8.6
branch).

Any ideas?

Dan

···

On Jun 19, 1:07 am, Sylvain Joyeux <sylvain.joy...@polytechnique.org> wrote:

#8

Incidentally, I did want to throw in a note to be very, very careful about exceptions when wrapping C++ code as a Ruby extension.

It's necessary to catch and wrap/unwrap exceptions at all C++/C boundaries, since otherwise, Ruby cleanup (ensure, rescue, etc...) will be skipped when C++ exceptions unwind the stack (potentially trashing the interpreter), and C++ cleanup (destructors for objects in automatic storage, catch, etc...) will get skipped when Ruby exceptions unwind the stack.

Also, on some architectures, a C++ exception tearing down a C stack frame is by itself enough to cause badness.

boost.python takes care of all this for writing Python extensions in C++, but sadly there's not a maintained Ruby equivalent.

-mental

#9

Maybe we should just rewrite the interpreter in C++ then. Think of all
the STL and Boost stuff we could then integrate "for free". I can only
guess what other advantages this might bring in the long run.

I know Matz has said in the past that the C++ object model would get
in the way, but I'm convinced it could be done (by smarter minds than
I).

Dan

···

On Jun 19, 11:13 am, MenTaLguY <men...@rydia.net> wrote:

Incidentally, I did want to throw in a note to be very, very careful about exceptions when wrapping C++ code as a Ruby extension.

It's necessary to catch and wrap/unwrap exceptions at all C++/C boundaries, since otherwise, Ruby cleanup (ensure, rescue, etc...) will be skipped when C++ exceptions unwind the stack (potentially trashing the interpreter), and C++ cleanup (destructors for objects in automatic storage, catch, etc...) will get skipped when Ruby exceptions unwind the stack.

Also, on some architectures, a C++ exception tearing down a C stack frame is by itself enough to cause badness.

boost.python takes care of all this for writing Python extensions in C++, but sadly there's not a maintained Ruby equivalent.

-mental

#10

It should be nearly sufficient to replace the setjmp/longjump bits
with C++ exceptions and build the interpreter with a C++ compiler.

However, that just exchanges one problem for another since the bulk of Ruby
extensions are written in C, and embedding Ruby becomes even more
difficult.

I think we'd all be much better off if someone wrote a Ruby equivalent to
boost.python. There is a partially-completed one out there, but it's
unmaintained I'm not able to find it again at the moment.

-mental

···

On Wed, 20 Jun 2007 02:37:39 +0900, Daniel Berger <djberg96@gmail.com> wrote:

Maybe we should just rewrite the interpreter in C++ then. Think of all
the STL and Boost stuff we could then integrate "for free". I can only
guess what other advantages this might bring in the long run.

#11

I'd make that trade. The potential benefits far exceed the pain of
reworking extensions IMHO, especially when you consider that *most*
people using Ruby don't do embedding or extending.

And this is coming from a guy who has written quite a few extensions.

Regards,

Dan

···

On Jun 19, 11:47 am, MenTaLguY <men...@rydia.net> wrote:

On Wed, 20 Jun 2007 02:37:39 +0900, Daniel Berger <djber...@gmail.com> wrote:
> Maybe we should just rewrite the interpreter in C++ then. Think of all
> the STL and Boost stuff we could then integrate "for free". I can only
> guess what other advantages this might bring in the long run.

It should be nearly sufficient to replace the setjmp/longjump bits
with C++ exceptions and build the interpreter with a C++ compiler.

However, that just exchanges one problem for another since the bulk of Ruby
extensions are written in C, and embedding Ruby becomes even more
difficult.