[ANN] FFI 0.1.1 (Foreign Function Interface) for Ruby 1.8.6/7 and 1.9

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7 and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic libraries, binding functions within them, and calling those functions from Ruby code. Here's a quick sample of binding and calling the getpid C library function:

require 'ffi'

module GetPid
   extend FFI::Library

   attach_function :getpid, [], :uint
end

puts GetPid.getpid

Here's another, calling qsort and passing a Ruby block as a C callback:

require 'ffi'

module LibC
   extend FFI::Library
   callback :qsort_cmp, [ :pointer, :pointer ], :int
   attach_function :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
   i1 = p1.get_int32(0)
   i2 = p2.get_int32(0)
   i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"

I posted a blog entry with a longer description of the library, additional examples, and links to some other documentation and posts. Docs are a little slim at this point, so feel free to experiment and update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into the general cosmos.

Finally, there's no need to write a C extension to call C libraries, and the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby 1.1.4+, and Rubinius (though Rubinius has no callback support yet).

Don't be an extension stooge! Use FFI!

- Charlie

Awesome. I've used DL to link up some custom libs to a Ruby service,
will give FFI a go and see how it compares :slight_smile:

Are things like structs likely to be supported in future, ala Python
ctypes?

···

* Charles Oliver Nutter (charles.nutter@sun.com) wrote:

The JRuby team is proud to announce the release of FFI for Ruby
1.8.6/7 and 1.9!

Finally, there's no need to write a C extension to call C libraries,
and the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby
1.1.4+, and Rubinius (though Rubinius has no callback support yet).

--
Thomas 'Freaky' Hurst
    http://hur.st/

Charles Oliver Nutter wrote:

I posted a blog entry with a longer description of the library, additional examples, and links to some other documentation and posts.

And then I completely forgot to include the blog post URL...

- Charlie

Interesting. On what kind of architectures is the binding part working ? I'm
using dyncall to do the actual interfacing work (http://www.dyncall.org/\) in a
DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
What are you using on your side ?

Sylvain

···

On Sat, Nov 01, 2008 at 08:16:39AM +0900, Charles Oliver Nutter wrote:

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:

Based on all of the examples I've seen, there's something that's still
not clear to me. How does ruby-ffi know which library to load to find the
specified function?

--Ken

···

On Fri, 31 Oct 2008 18:16:39 -0500, Charles Oliver Nutter wrote:

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:

require 'ffi'

module GetPid
   extend FFI::Library

   attach_function :getpid, , :uint
end

puts GetPid.getpid

Here's another, calling qsort and passing a Ruby block as a C callback:

require 'ffi'

module LibC
   extend FFI::Library
   callback :qsort_cmp, [ :pointer, :pointer ], :int attach_function
   :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
   i1 = p1.get_int32(0)
   i2 = p2.get_int32(0)
   i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"

I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.
Docs are a little slim at this point, so feel free to experiment and
update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into
the general cosmos.

Finally, there's no need to write a C extension to call C libraries, and
the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby 1.1.4+, and
Rubinius (though Rubinius has no callback support yet).

Don't be an extension stooge! Use FFI!

- Charlie

--
Chanoch (Ken) Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/

On windows, I can't get it to work for jruby or ruby

C:\jruby-1.1.5>jruby -v
jruby 1.1.5 (ruby 1.8.6 patchlevel 114) (2008-11-03 rev 7996) [x86-java]

C:\jruby-1.1.5>cd samples\ffi

C:\jruby-1.1.5\samples\ffi>dir
Volume in drive C has no label.
Volume Serial Number is F4EA-F50A

Directory of C:\jruby-1.1.5\samples\ffi

11/03/2008 05:04 PM <DIR> .
11/03/2008 05:04 PM <DIR> ..
11/03/2008 04:42 PM 301 ffi.rb
11/03/2008 04:42 PM 367 gettimeofday.rb
11/03/2008 04:42 PM 2,340 pty.rb
11/03/2008 04:42 PM 476 qsort.rb
11/03/2008 04:42 PM 2,078 win32api.rb
               5 File(s) 5,562 bytes
               2 Dir(s) 18,840,158,208 bytes free

C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
        from ffi.rb:4
        from ffi.rb:1:in `require'
        from win32api.rb:1

C:\jruby-1.1.5\samples\ffi>

For MRI, it tries to compile

C:\jruby-1.1.5\samples\ffi>gem install ffi
Building native extensions. This could take a while...
ERROR: Error installing ffi:
        ERROR: Failed to build gem native extension.

c:/ruby/bin/ruby.exe extconf.rb install ffi
creating Makefile

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.

Gem files will remain installed in c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1
for inspection.
Results logged to c:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.1/ext/gem_make.out

C:\jruby-1.1.5\samples\ffi>

Charles Oliver Nutter wrote:

I posted a blog entry with a longer description of the library, additional examples, and links to some other documentation and posts. Docs are a little slim at this point, so feel free to experiment and update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into the general cosmos.

FYI, I've moved the content of the page above to the Ruby FFI wiki here:

http://kenai.com/projects/ruby-ffi/pages/Home

Please make updates and additions to that copy. I'll modify the JRuby version to link to Ruby FFI's wiki.

- Charlie

Very cool!

I don't know how I missed this before, somehow slipped past my mental
filter (and now everything you were talking about at RubyConf starts to
make a lot more sense... :).

One question: a complaint I've had with DL is that it requires that I
specify the dynamic library that contains symbols I'm interested in. Is
this necessary with FFI? Can FFI be used to call functions in the Ruby
standard library when Ruby is not build with --enable-shared?

If so, I'd be interested in using this for Ludicrous to avoid having to
hard-code all the function pointers to the Ruby API functions I want to
call. That would mean, though, I would need FFI to be able to give me a
real function pointer (so I can pass the pointer to libjit.. I can
imagine this being useful for other cases as well).

Paul

Thomas Hurst wrote:

The JRuby team is proud to announce the release of FFI for Ruby
1.8.6/7 and 1.9!

Finally, there's no need to write a C extension to call C libraries,
and the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby
1.1.4+, and Rubinius (though Rubinius has no callback support yet).

Awesome. I've used DL to link up some custom libs to a Ruby service,
will give FFI a go and see how it compares :slight_smile:

Are things like structs likely to be supported in future, ala Python
ctypes?

Actually structs are already supported! See the blog post, and I believe there's some examples shipped with the gem. There needs to be more docs, certainly, and hopefully they'll get some TLC soon.

Also, I forgot to call out Evan Phoenix for coming up with the API and initial design, and he or someone else on Rubinus wrote up the templating/header-file-processing stuff as well. And of course a huge thanks to Wayne Meissner for implementing FFI not just once (for JRuby) but twice (for C Ruby). His work will mean a huge leap forward in cross-impl portability.

- Charlie

···

* Charles Oliver Nutter (charles.nutter@sun.com) wrote:

Looks cool, great work.

Two questions:

- Are variadic functions supported?
- Do you have any idea or measurements of the overhead of calling through FFI as opposed to using a compiled extension?

···

On 1 nov. 08, at 11:19, Charles Oliver Nutter wrote:

And then I completely forgot to include the blog post URL...

--
Luc Heinrich - luc@honk-honk.com

Sylvain Joyeux wrote:

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7 and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic libraries, binding functions within them, and calling those functions from Ruby code. Here's a quick sample of binding and calling the getpid C library function:

Interesting. On what kind of architectures is the binding part working ? I'm
using dyncall to do the actual interfacing work (http://www.dyncall.org/\) in a
DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
What are you using on your side ?

Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not certain about libffi specifically. but JNA claims to support OSX (ppc, x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris (x86, amd64, sparc, sparcv9) and Windows (x86, amd64).

I'm not sure if linux-ppc support is not provided because it's not supported or because nobody has a linux-ppc machine to build on. The latter has been the case for several entries on the list; I myself was the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time, before which there was no shipped support.

There's certainly one way to find out...gem install ffi. Report back here or on ruby-ffi mailing lists what you learn :slight_smile:

http://kenai.com/projects/ruby-ffi

- Charlie

···

On Sat, Nov 01, 2008 at 08:16:39AM +0900, Charles Oliver Nutter wrote:

Use ffi_lib in the extended module, e.g.

require 'rubygems'
require 'ffi'

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', , :int
attach_function 'endwin', , :int
attach_function 'getch', , :int
attach_function 'initscr', , :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', , :int
end

NCursesFFI.initscr
NCursesFFI.printw("Hello again")
NCursesFFI.refresh
NCursesFFI.getch
NCursesFFI.endwin

Regards,
Sean

···

On Tue, Nov 4, 2008 at 2:38 PM, Ken Bloom <kbloom@gmail.com> wrote:

Based on all of the examples I've seen, there's something that's still
not clear to me. How does ruby-ffi know which library to load to find the
specified function?

--Ken

Dominic Sisneros wrote:

C:\jruby-1.1.5\samples\ffi>jruby win32api.rb
null:1:in `const_missing': uninitialized constant POSIX::FFI (NameError)
        from ffi.rb:4
        from ffi.rb:1:in `require'
        from win32api.rb:1

That sample could be behind the times...play with it a bit and see what you can see. None of the core JRuby devs use Windows on a regular basis...I think this was just a quick experiment created early in FFI dev cycle.

FWIW, I believe someone (Dan Berger, probably) is working on a full FFI win32api impl. He could use some help.

For MRI, it tries to compile

C:\jruby-1.1.5\samples\ffi>gem install ffi
Building native extensions. This could take a while...
ERROR: Error installing ffi:
        ERROR: Failed to build gem native extension.

c:/ruby/bin/ruby.exe extconf.rb install ffi
creating Makefile

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.

We need to get a win32 binary FFI gem released. I think one or two people are working on that. Jump on Ruby FFI mailing lists if you are interested in helping.

- Charlie

I get the following when I try to compile on windows - any
suggestions? Not sure what srcdir should be or where I should set it.

···

========================================================================

C:\ffi>gem install ffi-0.1.0.gem
Building native extensions. This could take a while...
ERROR: Error installing ffi-0.1.0.gem:
        ERROR: Failed to build gem native extension.

C:/ruby/bin/ruby.exe extconf.rb install ffi-0.1.0.gem
creating Makefile

nmake

Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
Copyright (C) Microsoft Corporation. All rights reserved.

makefile(182) : fatal error U1052: file '$(srcdir)/ffi.mk' not found
Stop.

Gem files will remain installed in C:/ruby/lib/ruby/gems/1.8/gems/
ffi-0.1.0 for inspection.
Results logged to C:/ruby/lib/ruby/gems/1.8/gems/ffi-0.1.0/ext/
gem_make.out

Your blog is preaty known but for clarity:

:slight_smile:

···

On Sat, Nov 1, 2008 at 6:40 AM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:

Actually structs are already supported! See the blog post, and I believe
there's some examples shipped with the gem.

--
Radosław Bułat

http://radarek.jogger.pl - mój blog

Luc Heinrich wrote:

And then I completely forgot to include the blog post URL...

Looks cool, great work.

Two questions:

- Are variadic functions supported?
- Do you have any idea or measurements of the overhead of calling through FFI as opposed to using a compiled extension?

Wayne answers the latter, sorta, on his followup blog post:

He doesn't have specific numbers for performance at the moment, but the short story is that FFI introduces a bit of overhead; ultimately I believe that the overhead gets lost in the flow of a Ruby application, especially when you're tossing units of work across like SQL queries or arrays. Wayne probably can fill in more details on what the actual overhead is like.

And I'd also expect that any small amount of overhead is vastly outweighed by the ability to write an FFI-based library once and use it across implementations.

- Charlie

···

On 1 nov. 08, at 11:19, Charles Oliver Nutter wrote:

Ubuntu 8.10 provides libffi on ppc. The gem builds, but a simple ruby
program hangs (rather uses 100% CPU).

I can investigate further (e.g building my own libffi), but that may
not happen immediately.

-jh

···

On Mon, 3 Nov 2008 05:01:11 -0500, Charles Oliver Nutter wrote:

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on. The
latter has been the case for several entries on the list; I myself was
the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time,
before which there was no shipped support.

There's certainly one way to find out...gem install ffi. Report back
here or on ruby-ffi mailing lists what you learn :slight_smile:

Interesting. On what kind of architectures is the binding part working ? I'm
using dyncall to do the actual interfacing work (http://www.dyncall.org/\) in a
DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
What are you using on your side ?

Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not
certain about libffi specifically. but JNA claims to support OSX (ppc,
x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris
(x86, amd64, sparc, sparcv9) and Windows (x86, amd64).

I did not know about libffi ... I'll have to look if I should not replace
dyncall by libffi then. Thanks for the info.

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on.

I guess it is a bit of both. Given that libffi is supported on Linux/PPC64, I
guess having it on Linux/PPC should not be that much of a problem -- but still,
I think the calling convention can be slightly different between the two
architectures.

Sylvain

Sean O'halpin wrote:

module NCursesFFI
extend FFI::Library
ffi_lib 'ncurses'
attach_function 'clear', , :int
attach_function 'endwin', , :int
attach_function 'getch', , :int
attach_function 'initscr', , :int
attach_function 'printw', [ :string ], :int
attach_function 'refresh', , :int
end

Sean,
Going by the above, and the samples of structs on the blog, for a medium
sized library such as ncurses, there can be some effort in getting this
module ready. Will there be some kind of repository of such modules, so
we are not duplicating each others effort ?

I'd like to also add that the above sample is working on osx
10.5.5/darwin/ppc after a small modification in platform.rb above line
28.

    ARCH = case CPU.downcase
    when /i?86|x86|i86pc|powerpc/
      "i386"
    when /amd64|x86_64/
      "x86_64"

I have added "powerpc" to the first "when". btw, i had a clean gem
install.
Thanks.

···

On Tue, Nov 4, 2008 at 2:38 PM, Ken Bloom <kbloom@gmail.com> wrote:

--
Posted via http://www.ruby-forum.com/\.

Charles Oliver Nutter wrote:

Actually structs are already supported!

Sorry if you see this twice, posting difficulties.

I'm having a bash at wrapping freetds. This library
has a Context struct which contains three callback
function pointers. POLS says these should look like
this:

callback :handle_message, [ :pointer, :pointer, :pointer ], :in # TDSCONTEXT, TSSSOCKET, TDSMESSAGE
callback :handle_int, [ :pointer ], :in # void*

class Context < FFI:Struct
   layout \
     :locale, :pointer, 0, # TDSLOCALE
     :parent, :pointer, 4, # void *
     :msg_handler, :handle_message, 8, # callback(TDSCONTEXT, TDSSOCKET, TDSMESSAGE)
     :err_handler, :handle_message, 12, # callback(TDSCONTEXT, TDSSOCKET, TDSMESSAGE)
     :int_handler, :handle_int, 16 # callback(void*)
end

Am I on the right track, or is this not possible yet?

Clifford Heath.