Why can't Ruby load .gem files directly?

ruby-talk
1

I've just been reading the recent threads about gems. I don't have much background on this issue, but I have a few questions/concerns about the current state of gems.

My primary question is: why do I have to "install" a gem? What about the gem design would make it fundamentally impossible for Ruby to load a .gem file directly?

Think how great it would be if "installing a gem" was equivalent to "putting the gem file in your RUBYLIB path." What could be simpler? Sure, you couldn't put checks in place to make sure I've installed all the right dependencies, but you'll figure that out at runtime, when I try to load that gem (or you could use "gem check").

My concern with the current design is that it is too inflexible. It imposes a model where you have to use the "gem" command to do anything, and you have to hope that your use case is one of the operations that "gem" supports. For example, can gems support these use cases?

- I want to try out a gem on a system where I don't have root. (or where I don't want to install the software globally yet)

- I want to have more than one local gem repository (eg. /usr/lib vs. /usr/local/lib), and I want dependencies to be fulfilled across repositories (eg. a gem in /usr/lib can have a dependency be fulfilled by a gem in /usr/local/lib)

- I want to install a gem from a gemfile I already have (ie: don't fetch it from the network)

- I want to "unpack" a gem into a directory structure that can be deployed to another machine, without having to run "gem" on the remote machine (and without updating any global index on my current machine)

- same as the previous one, but I also want to "unpack" the gem into a different directory root than where it will live on the remote machine. I may not even know what directory it will live in on the remote machine.

What I don't like is the feeling that the "gem" command is a gatekeeper between you and your system -- you have to go through it to get anything done. And it hides the details of what's going on, making the system very opaque. It was difficult for me to figure out which of the above use cases are supported, because the rubygems documentation doesn't tell you the details of what "gem" does.

If Ruby could load .gem files directly, you would immediately support any use case, because they would be as flexible as single .rb files:

- "cp foo.gem /path/to/my/RUBYLIB" could be used in place of "gem install" (you could still use "gem install" to query remote servers if you want)

- "rm foo.gem" could be used in place of "gem uninstall"

- "ls foo*.gem" could be used in place of "gem list"

I think this would make the system more flexible and transparent. I would argue that any properties of the gem format that make this impossible should be reconsidered.

Josh

2

[...] For example, can gems support these use cases?

- I want to try out a gem on a system where I don't have root. [...]

Yes.

- I want to have more than one local gem repository [...]

Yes.

- I want to install a gem from a gemfile I already have [...]

Yes.

- I want to "unpack" a gem into a directory structure that can be
deployed to another machine, without having to run "gem" on the
remote machine

Yes.

- same as the previous one, but I also want to "unpack" the gem into
a different directory root [...]

Yes.

···

On Saturday 01 October 2005 11:12 pm, Joshua Haberman wrote:

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

3

Actually, this was strongly considered in early versions of gems. It would
require a require hack that is much more intrusive than the current one
RubyGems uses now. The real stopper was that we didn't see an easy way to
support shared libraries (.so's and .dll's) on all the different platforms.

···

On Saturday 01 October 2005 11:12 pm, Joshua Haberman wrote:

If Ruby could load .gem files directly, you would immediately support
any use case, because they would be as flexible as single .rb files:

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

4

My original prototype for RubyGems did exactly this. I considered gem
files to be much like a Java jar file, and the code was loaded
directly from the gem when required. My prototype used some custom
file formats (two actually, a plain text and a binary one, just for
fun) that contained the embedded code and some metadata. I presented
this prototype at RubyConf 2001, but stopped development due to lack
of certain libraries (zip for example) as well as some other real-life
issues (like being overwhelmed, losing interest and not having time.)

When Jim and the other guys decided to bring RubyGems back to life at
RubyConf 2003, they started with some fresh code, and I'm not sure why
they chose to implement gems as they did (but I'm sure there was a
good reason for having the separate install step, like maybe
performance.)

Either way, as Jim says your use cases are supported in the latest
RubyGems, though now that is has been brought up, I'd like to play
with some of my old ideas again.

Based on recent discussions though, I wonder if RubyGems need to be
re-written for a third time (though strictly speaking my first
revision really was a prototype and didn't have even close to the
feature-set of the current system.)

Ryan

···

On 10/1/05, Joshua Haberman <joshua@reverberate.org> wrote:

Think how great it would be if "installing a gem" was equivalent to
"putting the gem file in your RUBYLIB path." What could be simpler?
Sure, you couldn't put checks in place to make sure I've installed
all the right dependencies, but you'll figure that out at runtime,
when I try to load that gem (or you could use "gem check").

5

I'm not sure what you mean by that. Could you elaborate (or point me to documentation that provides more detail about this problem)?

Do gems attempt to be platform-independent even where there are C extensions, and compile extensions at gem install time? That doesn't sound like a very good idea.

If gems that have C extensions were kept platform-specific, there shouldn't be a problem with shared libraries.

Josh

···

On Oct 1, 2005, at 9:08 PM, Jim Weirich wrote:

On Saturday 01 October 2005 11:12 pm, Joshua Haberman wrote:

If Ruby could load .gem files directly, you would immediately support
any use case, because they would be as flexible as single .rb files:

Actually, this was strongly considered in early versions of gems. It would
require a require hack that is much more intrusive than the current one
RubyGems uses now. The real stopper was that we didn't see an easy way to
support shared libraries (.so's and .dll's) on all the different platforms.

6

The shared library part makes sense. I didn't even get close to that
in my little prototype.

It seems that the interpreter itself would need to natively support
gem files and be able to load shared libraries from them directly
before you could get away from having the separate install step. It
would be interesting to have gems with C source that are compiled on
demand (given the right setup on the installation machine) and then
used directly. So for example you would start off with
binlib-1.0-src.gem and afterwards have the src gem and then
binlib-1.0.gem with the embedded shared libs. But maybe you guys on
the RubyGems team already considered all this and have several good
arguments against it.

Ryan

···

On 10/2/05, Jim Weirich <jim@weirichhouse.org> wrote:

Actually, this was strongly considered in early versions of gems. It would
require a require hack that is much more intrusive than the current one
RubyGems uses now. The real stopper was that we didn't see an easy way to
support shared libraries (.so's and .dll's) on all the different platforms.

7

Either way, as Jim says your use cases are supported in the latest
RubyGems, though now that is has been brought up, I'd like to play
with some of my old ideas again.

Please do! :slight_smile:

Say that you had two implementations of gems with identical feature sets, but one used the "gem" command to do everything and one let you use .gem files directly. I would *much* prefer using the latter.

Though I am glad that Jim says my use cases are supported, I still feel that having to interact with the system using an opaque command like 'gem' is inferior to being able to interact with my filesystem directly. It also will make me worry that in the future I'll run into an unsupported use case I hadn't thought of.

Based on recent discussions though, I wonder if RubyGems need to be
re-written for a third time (though strictly speaking my first
revision really was a prototype and didn't have even close to the
feature-set of the current system.)

Yes, I don't know the details of the current controversy (ie. why Debian is having a problem packaging gems), but standardizing on a gem format that is at odds with OS packages seems like a very bad idea.

Josh

···

On Oct 1, 2005, at 9:25 PM, Ryan Leavengood wrote:

8

- I want to try out a gem on a system where I don't have root. [...]

Yes.

How is this one done? In particular, how would I install a gem as a non-root user, and have access to both my local gems and those that are installed by root?

9

[...] For example, can gems support these use cases?

- I want to try out a gem on a system where I don't have root. [...]

Yes.

How?

- I want to have more than one local gem repository [...]

Yes.

How?

- I want to install a gem from a gemfile I already have [...]

Yes.

How?

- I want to "unpack" a gem into a directory structure that can be
deployed to another machine, without having to run "gem" on the
remote machine

Yes.

How?

- same as the previous one, but I also want to "unpack" the gem into
a different directory root [...]

Yes.

How?

(References to documentation are fine, I just haven't been able to find much information about how to to do these things).

Josh

···

On Oct 1, 2005, at 8:59 PM, Jim Weirich wrote:

On Saturday 01 October 2005 11:12 pm, Joshua Haberman wrote:

10

If you have a shared library embedded in your gem file, how do you point your
LD_LIBRARY_PATH environment variable at it so that the OS knows about it.
Not only is a hard problem to solve, it is a hard problem that must
(potentially) be solved differently on each platform.

···

On Sunday 02 October 2005 12:27 am, Joshua Haberman wrote:

I'm not sure what you mean by that. Could you elaborate (or point me
to documentation that provides more detail about this problem)?

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

11

Joshua Haberman wrote:

Say that you had two implementations of gems with identical feature sets, but one used the "gem" command to do everything and one let you use .gem files directly. I would *much* prefer using the latter.

Actually, it doesn't sound like they're mutually exclusive. Stick a little flag in the metadata.gz that says "I don't have a compile step!" and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon require-ing something from inside the gem, you'll get an error "Install meeee!!"

Yes, I don't know the details of the current controversy (ie. why Debian is having a problem packaging gems)

Neither do I. Are they trying to repackage gems as... err.. whatever the apt-get file extension is? If so, why?

Devin

12

>> [...] For example, can gems support these use cases?

[... yes to all use cases ...]

How?

Ok, I was lazy with my responces. Here's the details.

>> - I want to try out a gem on a system where I don't have root. [...]

http://docs.rubygems.org/read/chapter/10#page33 -- See the --install-dir
option.

>> - I want to have more than one local gem repository [...]

http://docs.rubygems.org/read/chapter/12 -- See the GEM_PATH and GEM_HOME
environment options. Also see http://docs.rubygems.org/read/chapter/11 for
config file variations.

>> - I want to install a gem from a gemfile I already have [...]

http://docs.rubygems.org/read/chapter/10#page33 -- see --local option.

>> - I want to "unpack" a gem into a directory structure that can be
>> deployed to another machine, without having to run "gem" on the
>> remote machine

http://docs.rubygems.org/read/chapter/10#page39 -- The "unpack" command. Copy
the resulting directory to whereever you want on any machine and (a) run the
packages native install/setup scripts or (b) arrange for the proper
directories to be included in the LOAD_PATH when ruby is run. You can
determine the "proper directories" by examining the gemspec for that gem (see
http://docs.rubygems.org/read/chapter/10#page37 on how to extract the gem
spec).

If you don't want to install the gem first (unpack works on installed gems),
you can just untar the gem file and then untar the data embedded there

Sorry, there's more manual steps in this one, but the information is there and
is public. If you don't want to do all these steps by hand, we do include a
script that does it for you (see http://docs.rubygems.org/read/book/2\).

>> - same as the previous one, but I also want to "unpack" the gem into
>> a different directory root [...]

Same answer as above, note the "whereever you want" in the answer.

···

On Sunday 02 October 2005 01:11 pm, Joshua Haberman wrote:

On Oct 1, 2005, at 8:59 PM, Jim Weirich wrote:
> On Saturday 01 October 2005 11:12 pm, Joshua Haberman wrote:

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

13

Michael Schoen wrote:

- I want to try out a gem on a system where I don't have root. [...]

Yes.

How is this one done? In particular, how would I install a gem as a non-root user, and have access to both my local gems and those that are installed by root?

<URL:http://docs.rubygems.org/read/chapter/3#page83&gt;

mathew

···

--
<URL:http://www.pobox.com/~meta/&gt;
          WE HAVE TACOS

14

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

The problem is when Gems provide guarantees to the authors that are in
violation of how Debian users are used to software working, and no way
to implement things the authors need in a way that is possible to use
with the way Debian users want. Specifically, there's a problem with
the the-software-is-in-a-single-directory assumption.

Eivind.

···

On 10/2/05, Devin Mullins <twifkak@comcast.net> wrote:

Joshua Haberman wrote:

> Say that you had two implementations of gems with identical feature
> sets, but one used the "gem" command to do everything and one let you
> use .gem files directly. I would *much* prefer using the latter.

Actually, it doesn't sound like they're mutually exclusive. Stick a
little flag in the metadata.gz that says "I don't have a compile step!"
and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon
require-ing something from inside the gem, you'll get an error "Install
meeee!!"

> Yes, I don't know the details of the current controversy (ie. why
> Debian is having a problem packaging gems)

Neither do I. Are they trying to repackage gems as... err.. whatever the
apt-get file extension is? If so, why?

--
Hazzle free packages for Ruby?
RPA is available from http://www.rubyarchive.org/

15

Yes, I can definitely see that loading a shared library packaged inside a gem is not going to work.

So why not have shared libraries live in their own files in the gem directory?

my-gem-dir/
my-gem-dir/foo-1.2.gem
my-gem-dir/foo-1.2-libfoo.so
my-gem-dir/foo-1.2-libbar.so
my-gem-dir/baz-0.3.gem

The gem could specify what shared libraries need to be loaded (or you could pattern match on file names -- eg. load "foo-1.2*.so")

Yes, that would mean that packages containing C extensions would no longer be one single gem file, but luckily we have tar to help with that.

Josh

···

On Oct 1, 2005, at 10:25 PM, Jim Weirich wrote:

On Sunday 02 October 2005 12:27 am, Joshua Haberman wrote:

I'm not sure what you mean by that. Could you elaborate (or point me
to documentation that provides more detail about this problem)?

If you have a shared library embedded in your gem file, how do you point your
LD_LIBRARY_PATH environment variable at it so that the OS knows about it.
Not only is a hard problem to solve, it is a hard problem that must
(potentially) be solved differently on each platform.

16

It sounds like a debian user problem, not a Ruby user problem. I think most
Ruby people (who probably are not running ruby on Debian) want Ruby to work
like...Ruby, whether it's on a Mac, Windows, RedHat, Debian, Slolaris,
whatever. Has a similar beef been made about Java on Debian? (Why must Java
apps be deployed as .jar, .ear, war, whatever. Can't they pick just one and
make it work with apt-get?).

···

On 10/2/05, Eivind Eklund <eeklund@gmail.com> wrote:

On 10/2/05, Devin Mullins <twifkak@comcast.net> wrote:
> Joshua Haberman wrote:
>
> > Say that you had two implementations of gems with identical feature
> > sets, but one used the "gem" command to do everything and one let you
> > use .gem files directly. I would *much* prefer using the latter.
>
> Actually, it doesn't sound like they're mutually exclusive. Stick a
> little flag in the metadata.gz that says "I don't have a compile step!"
> and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon
> require-ing something from inside the gem, you'll get an error "Install
> meeee!!"
>
> > Yes, I don't know the details of the current controversy (ie. why
> > Debian is having a problem packaging gems)
>
> Neither do I. Are they trying to repackage gems as... err.. whatever the
> apt-get file extension is? If so, why?

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

The problem is when Gems provide guarantees to the authors that are in
violation of how Debian users are used to software working, and no way
to implement things the authors need in a way that is possible to use
with the way Debian users want. Specifically, there's a problem with
the the-software-is-in-a-single-directory assumption.

Eivind.
--
Hazzle free packages for Ruby?
RPA is available from http://www.rubyarchive.org/

17

Eivind Eklund wrote:

Yes, I don't know the details of the current controversy (ie. why
Debian is having a problem packaging gems)
     

Neither do I. Are they trying to repackage gems as... err.. whatever the
apt-get file extension is? If so, why?
   

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Okay (and thanks for the reply). That's weird. If they don't paticularly care about Ruby, what are they doing installing Ruby gems (usually libraries)? If they're trying to install an application that happens to be written in Ruby, then does it not suffice for the .deb to list ruby and rubygems as dependences, and then for the install step to be `gem install the-app-dependencies` or whatever?

Devin

18

May I request that we talk about technical issues like grown-up engineers, not like guests on "Jerry Springer?"

Tell me if this is an accurate list of the problems packaging gems:

1. Gems provide more guarantees than Debian packages can

2. Gems let you install more than one version of the same software, but Debian packages don't

3. something about Debian wanting software to be in a single directory?

I've tried to bring myself up to speed on these issues by reading recent threads, but it's very difficult to separate the technical information from the arguing.

Josh

···

On Oct 2, 2005, at 9:10 AM, Eivind Eklund wrote:

On 10/2/05, Devin Mullins <twifkak@comcast.net> wrote:

Joshua Haberman wrote:

Say that you had two implementations of gems with identical feature
sets, but one used the "gem" command to do everything and one let you
use .gem files directly. I would *much* prefer using the latter.

Actually, it doesn't sound like they're mutually exclusive. Stick a
little flag in the metadata.gz that says "I don't have a compile step!"
and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon
require-ing something from inside the gem, you'll get an error "Install
meeee!!"

Yes, I don't know the details of the current controversy (ie. why
Debian is having a problem packaging gems)

Neither do I. Are they trying to repackage gems as... err.. whatever the
apt-get file extension is? If so, why?

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

19

Exactly. A .gem file is just a tar file.

Once you have to start unpacking gem files for a portion of the payload, you
might as well go all the way and unpack the entire thing.

···

On Sunday 02 October 2005 04:08 pm, Joshua Haberman wrote:

Yes, that would mean that packages containing C extensions would no
longer be one single gem file, but luckily we have tar to help with
that.

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

20

Except that my-gem-dir/foo-1.2-libfoo.so would *actually* be:
my-gem-dir/i386-linux/foo-1.2-libfoo.so or something like that.

Ruby and RubyGems solves a lot of this already.

-austin

···

On 10/2/05, Joshua Haberman <joshua@reverberate.org> wrote:

my-gem-dir/
my-gem-dir/foo-1.2.gem
my-gem-dir/foo-1.2-libfoo.so
my-gem-dir/foo-1.2-libbar.so
my-gem-dir/baz-0.3.gem

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca