Native gem roundup!

ruby-talk
1

I'm curious what native gems/extensions people are typically using. In general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both categories.

Wrappers can now largely be handled by FFI, and I hope more and more of them will use FFI as needed to access those libraries. But I'm concerned about extensions written for performance, since Ruby 1.9 and JRuby do almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for things FFI or "faster Ruby" could do, since it will improve the future for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or why do they exist?

- Charlie

2

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

I use mysql, mysqlplus, eventmachine, ruby debug, hitimes, win32-api,
ruby prof, mongrel, and ferret. Most wrappers to C libs [which is for
speed].
Cheers!
-=r

···

--
Posted via http://www.ruby-forum.com/\.

3

Charles Oliver Nutter writes:
> I'm curious what native gems/extensions people are typically using. In
> general it seems like most native extensions fall into two categories:
>
> * They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
> * They are written for performance reasons, to implement a particular
> algorithm in a native language or call a library for the same reasons
>
> And there's a lot of grey area, with some extensions falling in both
> categories.
>
> Wrappers can now largely be handled by FFI, and I hope more and more of
> them will use FFI as needed to access those libraries. But I'm concerned
> about extensions written for performance, since Ruby 1.9 and JRuby do
> almost as much to speed Ruby up.
>
> Ultimately, my quest is to eliminate Ruby's dependence on extensions for
> things FFI or "faster Ruby" could do, since it will improve the future
> for both the standard and alternative implementations.
>
> So, what native gems or extensions do you use? Why do you use them or
> why do they exist?
>
> - Charlie
>

For me, they would be FastCGI and ruby-postgres, two more examples of
wrapping a C library. (Technically, I currently run FastCGI as
installed --- in 'site_ruby' --- rather than a gem; there was always a
hiccup if I tried to install it as a gem, so I went with the quickest
way to get it working.)

As you may guess, this is a Rails application. FastCGI is used for
Apache deployment; ruby-postgres is for accessing the PostgreSQL
database. I know there are other options for Apache/web deployment
(e.g. Mongrel), but this install is stable and other priorities take
precedence over investigating alternative deployment configurations.
For PostgreSQL, even if I moved to ruby-pg, that is still a C library
wrapper.

Anyways, just trying to answer your query.

···

--
Coey Minear

4

Charles Oliver Nutter wrote:

I'm curious what native gems/extensions people are typically using. In
general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick,
nokogiri
* They are written for performance reasons, to implement a particular
algorithm in a native language or call a library for the same reasons

* They interact directly with the Ruby interpreter (e.g. rcov,
ruby-debug)

Here's the set on my home machine:

$ cd /usr/local/lib/ruby/gems/1.8/gems/; find . -name '*.so'
./json-1.1.3/ext/json/ext/parser/parser.so
./json-1.1.3/ext/json/ext/generator/generator.so
./json-1.1.3/ext/json/ext/parser.so
./json-1.1.3/ext/json/ext/generator.so
./fastthread-1.0.1/ext/fastthread/fastthread.so
./fastthread-1.0.1/lib/fastthread.so
./rcov-0.8.1.2.0/lib/rcovrt.so
./rcov-0.8.1.2.0/ext/rcovrt/rcovrt.so
./linecache-0.43/ext/trace_nums.so
./linecache-0.43/lib/trace_nums.so
./mongrel-1.1.5/ext/http11/http11.so
./mongrel-1.1.5/lib/http11.so
./sqlite3-ruby-1.2.4/ext/sqlite3_api/sqlite3_api.so
./sqlite3-ruby-1.2.4/lib/sqlite3_api.so
./ruby-debug-base-0.10.3/ext/win32/ruby_debug.so
./ruby-debug-base-0.10.3/ext/ruby_debug.so
./ruby-debug-base-0.10.3/lib/ruby_debug.so
./ruby-gpgme-1.0.3/lib/gpgme_n.so
./ruby-gpgme-1.0.3/gpgme_n.so

On another machine I also have termios, rubywmq, ruby-postgres,
sys-filesystem, RedCloth

···

--
Posted via http://www.ruby-forum.com/\.

5

I'm curious what native gems/extensions people are typically using. In
general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular
algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both
categories.

Wrappers can now largely be handled by FFI, and I hope more and more of
them will use FFI as needed to access those libraries. But I'm concerned
about extensions written for performance, since Ruby 1.9 and JRuby do
almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for
things FFI or "faster Ruby" could do, since it will improve the future
for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

- Charlie

I use RUDL or Rubygame (SDL wrappers) to make games;
ruby-prof for profiling of these games, because pure Ruby profiler from the standard library slows programs too much down.

Jakub

···

--
"Configure complete, now type 'make' and PRAY."

                (configure script of zsnes - www.zsnes.com)

6

or they're written to get into ruby internals... I don't quite consider that a wrapper around C API and it looks like you don't either.

···

On Feb 2, 2009, at 07:15 , Charles Oliver Nutter wrote:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

7

Charles Oliver Nutter wrote:

I'm curious what native gems/extensions people are typically using. In general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both categories.

Wrappers can now largely be handled by FFI, and I hope more and more of them will use FFI as needed to access those libraries. But I'm concerned about extensions written for performance, since Ruby 1.9 and JRuby do almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for things FFI or "faster Ruby" could do, since it will improve the future for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or why do they exist?

- Charlie

Charlie,

As someone with a social survey research background, I want to advise you that this is an extremely poor way to get your question answered. If you're not serious about getting a good answer, your request is very close to list-noise. If you are, then you need a decent sample OR all the parametric data (i.e., don't sample it - get it all).

I wonder why you don't go after the latter? Is there someway to get download counts for various gems? I realize there are hundreds, but THAT would be useful data.

Alternatively, you could use the gems themselves as your population. Draw a sample of them, as samples of serious Ruby code (and get at least 35, and preferably much more than that), then scrape from them the gems THEY use, and get a frequency distribution from that sample.

In terms of bang for buck, I'd go with the latter alternative, 'cause THAT data would actually be something from which you might reasonably infer something.

What you're doing with this list-post nonsense is akin to putting a box of surveys on the sidewalk, with a sign "please fill one out", then taking your results and thinking they actually MEAN something. Believe me, they do not.

Hope this helps the "cause".

t.

···

--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Cloyd, MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< tc@tomcloyd.com >> (email)
<< TomCloyd.com >> (website) << sleightmind.wordpress.com >> (mental health weblog)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

8

strongtyping, used by html-table. I sent the author a patch that makes it work
with 1.8.6 and 1.9.1 but haven't heard back.

···

--
Jos Backus
jos at catnook.com

9

I use Gosu for game development, which I suppose is a wrapper around the
Gosu C++ library from the same developer... I think more people use Gosu
from Ruby than from C++, though I haven't asked. Gosu as a whole is a
wrapper around OpenGL, fmod (or SDL), and Cocoa/Win32/GTK (depending on
platform).

I also used RubyCocoa to allow me to construct a native OS X application
in Ruby... I'm not sure how you'd classify that into one of the
categories... it's kind of a wrapper, but around a whole language
(Objective-C) instead of a library. Apple calls it a bridge. I'm not
sure what the utility of having it available in JRuby would be, since
most RubyCocoa apps are distributed as self-contained bundles which
don't rely on an interpreter being already present on the end user's
machine, but it'd be neat to see it.

Other than that, ruby-debug and ruby-prof, same as everyone else.

···

--
Posted via http://www.ruby-forum.com/.

10

Tom Cloyd wrote:

Charles Oliver Nutter wrote:

I'm curious what native gems/extensions people are typically using. In general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both categories.

Wrappers can now largely be handled by FFI, and I hope more and more of them will use FFI as needed to access those libraries. But I'm concerned about extensions written for performance, since Ruby 1.9 and JRuby do almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for things FFI or "faster Ruby" could do, since it will improve the future for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or why do they exist?

- Charlie

Charlie,

As someone with a social survey research background, I want to advise you that this is an extremely poor way to get your question answered. If you're not serious about getting a good answer, your request is very close to list-noise. If you are, then you need a decent sample OR all the parametric data (i.e., don't sample it - get it all).

I wonder why you don't go after the latter? Is there someway to get download counts for various gems? I realize there are hundreds, but THAT would be useful data.

Alternatively, you could use the gems themselves as your population. Draw a sample of them, as samples of serious Ruby code (and get at least 35, and preferably much more than that), then scrape from them the gems THEY use, and get a frequency distribution from that sample.

In terms of bang for buck, I'd go with the latter alternative, 'cause THAT data would actually be something from which you might reasonably infer something.

What you're doing with this list-post nonsense is akin to putting a box of surveys on the sidewalk, with a sign "please fill one out", then taking your results and thinking they actually MEAN something. Believe me, they do not.

Hope this helps the "cause".

t.

There is always this:

http://gems.rubyforge.org/stats.html

-Justin

11

Tom Cloyd wrote:

As someone with a social survey research background, I want to advise you that this is an extremely poor way to get your question answered. If you're not serious about getting a good answer, your request is very close to list-noise. If you are, then you need a decent sample OR all the parametric data (i.e., don't sample it - get it all).

I disagree. I think the people most likely to respond to a mailing-list request are the exact people I'm looking to reach. Did it occur to you that perhaps I know there are download stats for all gems that I could mine? I'm looking to gauge the importance of gems to vocal mailing-list participants directly, by posting a general message here and seeing who responds. And I've gotten a lot of great responses, both on-list and off. If I wanted an accurate count of gem downloads (no doubt inflated by older gems including dev-time libraries as runtime dependencies) I certainly could get that. But people coming out and saying "I really need X" is far more interesting.

So please, others, feel free to reply with what X you find important.

And Tom, thanks for your input. What native extensions do you find useful?

- Charlie

12

Roger Pack wrote:

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

I use mysql, mysqlplus, eventmachine, ruby debug, hitimes, win32-api, ruby prof, mongrel, and ferret. Most wrappers to C libs [which is for speed].
Cheers!

Thanks for the response!

Is that the win32 that ships with MRI or something else? I don't use MRI much, so I'm not familiar with that library.

eventmachine and mongrel..I assume we're talking about evented mongrel?

ruby_debug and ruby_prof...I see ruby_prof coming up a lot. I think we'll simply need to provide something similar out of the box for JRuby. Hmmm.

One last question....why mysql or mysqlplus gems instead of the "nice" DB wrapper libraries like Sequel, DataMapper, and so on?

- Charlie

13

Coey Minear wrote:

As you may guess, this is a Rails application. FastCGI is used for
Apache deployment; ruby-postgres is for accessing the PostgreSQL
database. I know there are other options for Apache/web deployment
(e.g. Mongrel), but this install is stable and other priorities take
precedence over investigating alternative deployment configurations.
For PostgreSQL, even if I moved to ruby-pg, that is still a C library
wrapper.

Anyways, just trying to answer your query.

Cool, thanks. Are you using ruby-postgres through ActiveRecord itself or standalone in some way

- Charlie

14

Brian Candler wrote:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular
algorithm in a native language or call a library for the same reasons

* They interact directly with the Ruby interpreter (e.g. rcov, ruby-debug)

Damn good point...you totally trumped me there. So there's three cases, with this new one perhaps being truly the most impl-specific case.

Here's the set on my home machine:

$ cd /usr/local/lib/ruby/gems/1.8/gems/; find . -name '*.so'
./json-1.1.3/ext/json/ext/parser/parser.so
./json-1.1.3/ext/json/ext/generator/generator.so
./json-1.1.3/ext/json/ext/parser.so
./json-1.1.3/ext/json/ext/generator.so

Do people generally use the native json gem for performance reasons? Does a pure-ruby version not cut it? Are there benchmarks?

./fastthread-1.0.1/ext/fastthread/fastthread.so
./fastthread-1.0.1/lib/fastthread.so

Ok, not needed under JRuby, recent Ruby 1.8.6 or 1.8.7+.

./rcov-0.8.1.2.0/lib/rcovrt.so
./rcov-0.8.1.2.0/ext/rcovrt/rcovrt.so

Tooling...blasted tooling. JVM has so much good tooling, if I could only harness it well for Ruby stuff.

./linecache-0.43/ext/trace_nums.so
./linecache-0.43/lib/trace_nums.so
./mongrel-1.1.5/ext/http11/http11.so
./mongrel-1.1.5/lib/http11.so
./sqlite3-ruby-1.2.4/ext/sqlite3_api/sqlite3_api.so
./sqlite3-ruby-1.2.4/lib/sqlite3_api.so
./ruby-debug-base-0.10.3/ext/win32/ruby_debug.so
./ruby-debug-base-0.10.3/ext/ruby_debug.so
./ruby-debug-base-0.10.3/lib/ruby_debug.so
./ruby-gpgme-1.0.3/lib/gpgme_n.so
./ruby-gpgme-1.0.3/gpgme_n.so

On another machine I also have termios, rubywmq, ruby-postgres, sys-filesystem, RedCloth

Thanks for the response. Obviously I'm interested because of JRuby, and I think most of these have equivalents already, but there are obviously gaps. I think dev-time tooling may actually be where we're weakest at the moment, which is ironic given the history of JVM/Java-based tooling.

- Charlie

15

Jakub Pavlík jn. wrote:

I'm curious what native gems/extensions people are typically using. In general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both categories.

Wrappers can now largely be handled by FFI, and I hope more and more of them will use FFI as needed to access those libraries. But I'm concerned about extensions written for performance, since Ruby 1.9 and JRuby do almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for things FFI or "faster Ruby" could do, since it will improve the future for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or why do they exist?

- Charlie

I use RUDL or Rubygame (SDL wrappers) to make games;
ruby-prof for profiling of these games, because pure Ruby profiler from the standard library slows programs too much down.

I think someone was working on SDL over FFI recently, but I don't remember who and I don't remember how far they got. I'd love to see it myself.

ruby-prof...noted. Seems like an essential tool for most folks, we'll just have to provide something.

- Charlie

16

Ryan Davis wrote:

or they're written to get into ruby internals... I don't quite consider that a wrapper around C API and it looks like you don't either.

Yeah, that's definitely a third category I totally missed. It's an unusual one, though, since it doesn't necessarily have to be "native" (it could be in Ruby in Rubinius, or Java in JRuby...)

And my failure to recognize it could be why we don't have a ruby-prof for JRuby yet. Definitely noted.

- Charlie

17

Jos Backus wrote:

strongtyping, used by html-table. I sent the author a patch that makes it work
with 1.8.6 and 1.9.1 but haven't heard back.

I was unaware of strongtyping...very glad you brought it to my attention. So many projects, so little time...but this warrants some time.

Thank you!

- Charlie

18

How would it be technically possible to do jruby cocoa If the machines weren't macs?

Blog: http://random8.zenunit.com/
Learn rails: http://sensei.zenunit.com/

···

On 05/02/2009, at 10:46 AM, Adam Gardner <adam.oddfellow@gmail.com> wrote:

I use Gosu for game development, which I suppose is a wrapper around the
Gosu C++ library from the same developer... I think more people use Gosu
from Ruby than from C++, though I haven't asked. Gosu as a whole is a
wrapper around OpenGL, fmod (or SDL), and Cocoa/Win32/GTK (depending on
platform).

I also used RubyCocoa to allow me to construct a native OS X application
in Ruby... I'm not sure how you'd classify that into one of the
categories... it's kind of a wrapper, but around a whole language
(Objective-C) instead of a library. Apple calls it a bridge. I'm not
sure what the utility of having it available in JRuby would be, since
most RubyCocoa apps are distributed as self-contained bundles which
don't rely on an interpreter being already present on the end user's
machine, but it'd be neat to see it.

Other than that, ruby-debug and ruby-prof, same as everyone else.
--
Posted via http://www.ruby-forum.com/\.

19

Justin Collins wrote:

Tom Cloyd wrote:

Charles Oliver Nutter wrote:

I'm curious what native gems/extensions people are typically using. In general it seems like most native extensions fall into two categories:

* They are wrappers around a C API/library, as in zlib, rmagick, nokogiri
* They are written for performance reasons, to implement a particular algorithm in a native language or call a library for the same reasons

And there's a lot of grey area, with some extensions falling in both categories.

Wrappers can now largely be handled by FFI, and I hope more and more of them will use FFI as needed to access those libraries. But I'm concerned about extensions written for performance, since Ruby 1.9 and JRuby do almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby's dependence on extensions for things FFI or "faster Ruby" could do, since it will improve the future for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or why do they exist?

- Charlie

Charlie,

As someone with a social survey research background, I want to advise you that this is an extremely poor way to get your question answered. If you're not serious about getting a good answer, your request is very close to list-noise. If you are, then you need a decent sample OR all the parametric data (i.e., don't sample it - get it all).

I wonder why you don't go after the latter? Is there someway to get download counts for various gems? I realize there are hundreds, but THAT would be useful data.

Alternatively, you could use the gems themselves as your population. Draw a sample of them, as samples of serious Ruby code (and get at least 35, and preferably much more than that), then scrape from them the gems THEY use, and get a frequency distribution from that sample.

In terms of bang for buck, I'd go with the latter alternative, 'cause THAT data would actually be something from which you might reasonably infer something.

What you're doing with this list-post nonsense is akin to putting a box of surveys on the sidewalk, with a sign "please fill one out", then taking your results and thinking they actually MEAN something. Believe me, they do not.

Hope this helps the "cause".

t.

There is always this:

RubyGems.org | your community gem host

-Justin

Now, THAT's outright cheating. Worse yet, you probably Googled to find this (or could have, if you didn't know about it). What is this world coming too?

t.

···

--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Cloyd, MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< tc@tomcloyd.com >> (email)
<< TomCloyd.com >> (website) << sleightmind.wordpress.com >> (mental health weblog)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

20

Yeah. just depends on the impl's API into itself. ParseTree simply isn't necessary on rubinius.

···

On Feb 3, 2009, at 01:24 , Charles Oliver Nutter wrote:

Ryan Davis wrote:

or they're written to get into ruby internals... I don't quite consider that a wrapper around C API and it looks like you don't either.

Yeah, that's definitely a third category I totally missed. It's an unusual one, though, since it doesn't necessarily have to be "native" (it could be in Ruby in Rubinius, or Java in JRuby...)