Rubygems thoughts

ruby-talk
1

My participation here is sporadic at best, so forgive me if these thoughts have been brought up before.

I just discovered rubygems[1], and it looks neat and promising. I have a few reservations about it, though. I'll throw them out here in hopes that the authors and/or users and potential users see them, and answer them if they have been addressed.

First:
   require 'rubygems'
   require_gem 'progressbar'
Right there you're commiting yourself to gem dependencies, which requires rubygems to be installed anywhere you want your package to go, whether you've packaged your program as a gem or not. Therefore it makes no sense to package it as anything but a gem. That 'viral' nature is great for adoption, but not until critical mass has been reached. If not all the dependencies are available as gems, if you decide to go gems you now have a sticky mess of traditional and non-traditional. In theory, everyone jumps on the bandwagon ASAP and life with gems is simple. In practice I think we'll see many gems that say "oh, after you've installed the gem and automatically its dependencies, you also have to go grab package X from RAA or from site X." You haven't gained anything but a marginal decrease in complexity which is offset by the confusion for the user who has just been converted to how cool gems are in the first place. IMHO gem-isms belong in the spec file, and the code should be transparent. As a result packages won't be pushed into making the absolute gem/no-gem decision, which should result in cleaner gems.

But I'm having a hard time convincing even myself that I make any sense in that last paragraph so you may just think I'm blabbering. :wink: I guess it boils down to a gut feeling. If using/creating gems requires special code then you're locked into gems. If another more preferable technology comes along later, you've got to change all the code instead of just repackaging it, and it becomes very difficult to maintain a gem and another kind of package.

Point two, for gems to be really useful pretty much all the common libraries at least need to be gems. That's normal, and what I'd call critical mass for a packaging system. If gems is successful, it will grow quickly, perhaps even exponentially. Sending them by email to an individual is going to quickly become a bottleneck in that case. So there needs to be an upload system, preferably with gpg signing.

So there are my thoughts, for what they're worth. Enjoy! :slight_smile:

1. http://rubygems.rubyforge.org/

2

Hans Fugal said:

I just discovered rubygems[1], and it looks neat and promising.

Thanks!

I have a

few reservations about it, though. [...]

First:
   require 'rubygems'
   require_gem 'progressbar'
Right there you're commiting yourself to gem dependencies, which
requires rubygems to be installed anywhere you want your package to go,

A two-part response to your question...

(1) Gems can be installed with a "stub" in the standard library location.
This stub requires rubygems, then requires your package with the proper
version, providing a transparent access layer for gems. Stubs are now
installed by default (controlled by an option on the gem command line).

(2) One of the benefits of rubygems is the ability to specify some version
control over what gems get used with your library/application. If you
wish to take advantage of version control, then you necessarily become
dependent on the gems.

We are still evolving the "Best Practices" for gems, but I suspect that a
lot of projects will collect their "require_gems" in one area for better
version control. (I don't want to say I want version ">= 2.1" in more
than one place!) So I suspect require_gem with version specs to become
centralized and normal requires to continue to appear in whatever file
needs them.

Once the version control is centralized, it is easy to make it optional.
For example, just bury the require_gems in a rescue block and if it fails,
continue normally.

Point two, for gems to be really useful pretty much all the common
libraries at least need to be gems. That's normal, and what I'd call
critical mass for a packaging system.

We are working on it! I'll say it again, if someone needs help packaging
their library as a gem, feel free to drop me a line at
jim@weirichhouse.org. Or post on the rubygems developement list (on
RubyForge).

If gems is successful, it will
grow quickly, perhaps even exponentially. Sending them by email to an
individual is going to quickly become a bottleneck in that case. So
there needs to be an upload system, preferably with gpg signing.

For now, if you create a project on RubyForge and upload a GEM file, the
gem will automatically be served by the rubyforge gemserver. Use gem -Rl
to get a list of gems.

So there are my thoughts, for what they're worth. Enjoy! :slight_smile:

Thanks for the feedback.

···

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

3

First:
  require 'rubygems'
  require_gem 'progressbar'
Right there you're commiting yourself to gem dependencies, which
requires rubygems to be installed anywhere you want your package to go,
whether you've packaged your program as a gem or not. Therefore it makes
no sense to package it as anything but a gem. That 'viral' nature is
great for adoption, but not until critical mass has been reached. If not
all the dependencies are available as gems, if you decide to go gems you
now have a sticky mess of traditional and non-traditional.>

RubyGems no longer requires you to use that scheme; it can install
'library stubs' so
  require 'progressbar'
should work.

Point two, for gems to be really useful pretty much all the common
libraries at least need to be gems. That's normal, and what I'd call
critical mass for a packaging system. If gems is successful, it will
grow quickly, perhaps even exponentially. Sending them by email to an
individual is going to quickly become a bottleneck in that case. So

gem files released through Rubyforge are automatically copied to the
'repository', so direct submission by email is no longer the recommended
way to publish a gem.

there needs to be an upload system, preferably with gpg signing.

There's unfortunately no security mechanism in RubyGems atm.; this is
somewhat difficult due to RubyGems "distributed" nature, where the
packaging work is pushed down to upstream developers, so even if gem
signatures were implemented, building the web of trust could take some
time.

···

On Thu, Jun 24, 2004 at 11:24:08PM +0900, Hans Fugal wrote:

--
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

Actually, typing random strings in the Finder does the equivalent of
filename completion.
  -- Discussion on file completion vs. the Mac Finder

4

[ piggybacking my request onto this thread :slight_smile: ]

Could someone show a hello-world example of creating a rubygem
with an extension? I have followed the example,

http://rubygems.rubyforge.org/wiki/wiki.pl?CreateAGemInTenMinutes

using s.extension << "..." without success.

How are different platforms flagged? Do I append the gem name with
"-i686-linux" for example? How do I specify more than one extension?
Can the gem attempt to compile an extension?

I have been told gem w/ extension works in theory, but I have not
seen a concrete, working example.

···

__________________________________
Do you Yahoo!?
Yahoo! Mail is new and improved - Check it out!
http://promotions.yahoo.com/new_mail

5

Is there someplace that non-RubyForge projects can upload their gems,
so that they can also be listed with the "remotely installable" gems?
I know that this is probably a question for Rich or Tom, but wanted to
get it out there.

···

On Fri, 25 Jun 2004 01:00:11 +0900, Jim Weirich <jim@weirichhouse.org> wrote:

For now, if you create a project on RubyForge and upload a GEM file, the
gem will automatically be served by the rubyforge gemserver. Use gem -Rl
to get a list of gems.

6

Jim Weirich wrote:

A two-part response to your question...

(1) Gems can be installed with a "stub" in the standard library location. This stub requires rubygems, then requires your package with the proper
version, providing a transparent access layer for gems. Stubs are now
installed by default (controlled by an option on the gem command line).

Ok, that seems like a great solution. Guess not all the docs have been updated to reflect that on the wiki.

(2) One of the benefits of rubygems is the ability to specify some version
control over what gems get used with your library/application. If you
wish to take advantage of version control, then you necessarily become
dependent on the gems.

Naturally. I like the idea of consolidating this that you gave.

If gems is successful, it will
grow quickly, perhaps even exponentially. Sending them by email to an
individual is going to quickly become a bottleneck in that case. So
there needs to be an upload system, preferably with gpg signing.

For now, if you create a project on RubyForge and upload a GEM file, the
gem will automatically be served by the rubyforge gemserver. Use gem -Rl
to get a list of gems.

Ah, nifty. That's the spirit.

7

There's unfortunately no security mechanism in RubyGems atm.; this is
somewhat difficult due to RubyGems "distributed" nature, where the
packaging work is pushed down to upstream developers, so even if gem
signatures were implemented, building the web of trust could take some
time.

A good start would be working in signatures and letting people do with that whatever seems natural. For example, if I were to release a gem I'd like to sign it and provide my public key on the official project website, or the fingerprint of it which can be got from a public keyserver. That requires a manual check by the user to be of any use, but once the infrastructure is there other things can evolve, like some repositories would have an upload system where developers must sign the gem and they are allowed to upload with their signature for a set of packages. Packages can be uploaded by possibly more than one developer. The repository provides, along with the gems, a keyring of the public keys. The client downloads a package and checks its signature against that keyring. Or maybe the client doesn't even need to verify, because it's the upload process that really benefits from the security measures. (if someone can modify what gem you download they can probably modify what keyring you get... otoh if the keyring itself were signed by the repository ...)

8

[ piggybacking my request onto this thread :slight_smile: ]

Could someone show a hello-world example of creating a rubygem
with an extension? I have followed the example,

http://rubygems.rubyforge.org/wiki/wiki.pl?CreateAGemInTenMinutes

using s.extension << "..." without success.

Could you be more specific about the "without success" part? I somehow
managed to get this to work the first time (I guess by dumb luck) when
experimenting with making a source gem for FXRuby last night. My only
concern was that when I actually installed the gem, it did all of the
configuration & build silently; if something had gone wrong, I
wouldn't have had any way to see the error(s).

How are different platforms flagged? Do I append the gem name with
"-i686-linux" for example?

I don't see any "binary" gems (i.e. gems containing precompiled
extensions) in the list at http://gems.rubyforge.org. I have the
feeling this naming convention is still up in the air.

How do I specify more than one extension?

[Don't know, suspect that you can't at this time.]

Can the gem attempt to compile an extension?

If it's a source gem (i.e. one that defines an "extension" attribute
in its Specification) it gets compiled when the user tries to install
the gem.

I have been told gem w/ extension works in theory, but I have not
seen a concrete, working example.

I should be able to point you to one by the end of this weekend, if
all goes well. The plan is to release FXRuby 1.2 as gem(s).

···

On Fri, 25 Jun 2004 02:58:12 +0900, Jeff Mitchell <quixoticsycophant@yahoo.com> wrote:

9

(1) Gems can be installed with a "stub" in the standard library location.
This stub requires rubygems, then requires your package with the proper
version, providing a transparent access layer for gems. Stubs are now
installed by default (controlled by an option on the gem command line).

Let me make this really clear with an example session:

  $ ruby -r ruby-doom -e ''
  /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:28:in `require_gem': (LoadError)
  Could not find RubyGem ruby-doom (0.0.0)
          from /usr/local/lib/ruby/site_ruby/1.8/ruby-doom.rb:11

  $ gem -Ri ruby-doom
  Attempting remote installation of 'ruby-doom'
  Successfully installed ruby-doom-0.8

  $ ruby -r ruby-doom -e ''

  $

(2) One of the benefits of rubygems is the ability to specify some version
control over what gems get used with your library/application. If you
wish to take advantage of version control, then you necessarily become
dependent on the gems.

We are still evolving the "Best Practices" for gems, but I suspect that a
lot of projects will collect their "require_gems" in one area for better
version control. (I don't want to say I want version ">= 2.1" in more
than one place!) So I suspect require_gem with version specs to become
centralized and normal requires to continue to appear in whatever file
needs them.

Once the version control is centralized, it is easy to make it optional.
For example, just bury the require_gems in a rescue block and if it fails,
continue normally.

That's a really good win-win approach! This means the code will
happily run whether rubygems is installed or not, and it's easy to
maintain. Not only that, but it captures all the dependencies in one
place, so you can document (for non-RubyGems users) what other
packages they'll need installed.

Now, can this be taken a step further? Those dependencies are kept in
the gem spec anyway; why duplicate them in code? Is there some way we
can reuse the info in that spec? Is that something worth pursuing?

Point two, for gems to be really useful pretty much all the common
libraries at least need to be gems. That's normal, and what I'd call
critical mass for a packaging system.

Yeah, at some point we (the developers) should have a "working bee" to
create as many gems as possible :slight_smile:

Gavin

···

On Friday, June 25, 2004, 2:00:11 AM, Jim wrote:

10

Lyle Johnson wrote:

Is there someplace that non-RubyForge projects can upload their gems,
so that they can also be listed with the "remotely installable" gems?
I know that this is probably a question for Rich or Tom, but wanted to
get it out there.

From an outsider's perspective (i.e., I'm not Rich or Tom), you could always
create a RubyForge project to act as a release mirror for your project.

Curt

11

I didn't realize this either. Someone ought to write a definitive tutorial on
Gems and maintain it with each release. The examples I found left a lot of
things unexplained and it was a painful session of trial-and-error to get my
gems working properly, and even then, I didn't know there was a way to bypass
the "require_gem" statement.

If any more work is done on Gems, I strongly recommend it be on good
documentation, and then maintain those docs to update it for the changes made
in the future.

  Sean O'Dell

···

On Thursday 24 June 2004 10:18, Hans Fugal wrote:

Jim Weirich wrote:
> A two-part response to your question...
>
> (1) Gems can be installed with a "stub" in the standard library location.
> This stub requires rubygems, then requires your package with the proper
> version, providing a transparent access layer for gems. Stubs are now
> installed by default (controlled by an option on the gem command line).

Ok, that seems like a great solution. Guess not all the docs have been
updated to reflect that on the wiki.

12

Ruby comes with OpenSSL built-in now, so this need not be a manual
check.

···

Hans Fugal (hfugal@wencor.com) wrote:

>
>There's unfortunately no security mechanism in RubyGems atm.; this is
>somewhat difficult due to RubyGems "distributed" nature, where the
>packaging work is pushed down to upstream developers, so even if gem
>signatures were implemented, building the web of trust could take some
>time.
>

A good start would be working in signatures and letting people do with
that whatever seems natural. For example, if I were to release a gem I'd
like to sign it and provide my public key on the official project
website, or the fingerprint of it which can be got from a public
keyserver. That requires a manual check by the user to be of any use,
but once the infrastructure is there other things can evolve, like some
repositories would have an upload system where developers must sign the
gem and they are allowed to upload with their signature for a set of
packages.

--
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

13

>There's unfortunately no security mechanism in RubyGems atm.; this is
>somewhat difficult due to RubyGems "distributed" nature, where the
>packaging work is pushed down to upstream developers, so even if gem
>signatures were implemented, building the web of trust could take some
>time.
>

A good start would be working in signatures and letting people do with
that whatever seems natural.

What I meant is that the more technical part of the problem is easily solved
(using openssl or the upcoming Crypt::RSA), but you still have to build
the web of trust (key signing parties, etc).

For example, if I were to release a gem I'd
like to sign it and provide my public key on the official project
website, or the fingerprint of it which can be got from a public
keyserver. That requires a manual check by the user to be of any use,

Need not be manual.

···

On Fri, Jun 25, 2004 at 02:28:09AM +0900, Hans Fugal wrote:

but once the infrastructure is there other things can evolve, like some
repositories would have an upload system where developers must sign the
gem and they are allowed to upload with their signature for a set of
packages. Packages can be uploaded by possibly more than one developer.
The repository provides, along with the gems, a keyring of the public

                                                  =====================
You cannot accept libs. + public keys if you haven't verified that the
key was really created by whoever claims it.

keys. The client downloads a package and checks its signature against
that keyring. Or maybe the client doesn't even need to verify, because

Once you have implemented a signature system and have established the
web of trust, it's makes no sense not to let the end-user verify it
himself: online repositories can be cracked.

--
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

<doogie> Thinking is dangerous. It leads to ideas.
  -- Seen on #Debian

14

Could be doable; perhaps a CGI script that accepted the files, verified
that they were well-formed Gems, etc... might be a good thing to post
here:

http://rubyforge.org/tracker/?atid=578&group_id=126&func=browse

Yours,

Tom

···

On Thu, 2004-06-24 at 12:14, Lyle Johnson wrote:

Is there someplace that non-RubyForge projects can upload their gems,
so that they can also be listed with the "remotely installable" gems?
I know that this is probably a question for Rich or Tom, but wanted to
get it out there.

15

> [ piggybacking my request onto this thread :slight_smile: ]
>
> Could someone show a hello-world example of creating a rubygem
> with an extension? I have followed the example,
>
> http://rubygems.rubyforge.org/wiki/wiki.pl?CreateAGemInTenMinutes
>
> using s.extension << "..." without success.

Could you be more specific about the "without success" part?

Well there were several problems and I was hoping to mooch off a
working example rather than slosh through it. But I've gotten a
basic example working now.

General questions:

(1) s.require_path can only be one path? Errors occurred when I made
it an array and when I made it colon-separated paths.

(2) myclass.so is installed into gems/myclass-9.8.7/lib. That's not
where it should go. myclass is part of a more complex project with
a top-level directory and several subdirectories. When the extension
lies in ext/mypackage/myclass/extconf.rb, it is still installed into
the root of gems/myclass-9.8.7/lib. Can I control this somehow?

(1) and (2) could be lumped together under "rubygems strips paths to
make them incompatible with a conventional install."

(3) [repeat from before] Is there a convention for prebuilt
extensions? It would be nice if something like 'gem --use-binary -i
myclass' would retrieve the prebuilt gem for the host platform if it
exists, for example myclass-i686-linux-ruby18.

(4) Are extensions guaranteed to be compatible among 1.8.x
installs? If not, I suggest renaming libmsvcrt-ruby18.a to
libmsvcrt-ruby181.a.

I somehow managed to get this to work the first time (I guess by
dumb luck) when experimenting with making a source gem for FXRuby
last night. My only concern was that when I actually installed the
gem, it did all of the configuration & build silently; if something
had gone wrong, I wouldn't have had any way to see the error(s).

Hm? I see the all the compile commands happening. Also, there is a
gem_make.out file in the directory with extconf.rb.

···

--- Lyle Johnson <lyle.johnson@gmail.com> wrote:

On Fri, 25 Jun 2004 02:58:12 +0900, Jeff Mitchell > <quixoticsycophant@yahoo.com> wrote:

__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - Send 10MB messages!
http://promotions.yahoo.com/new_mail

16

Sean O'Dell said:

···

I didn't realize this either. Someone ought to write a definitive
tutorial on
Gems and maintain it with each release. The examples I found left a lot
of
things unexplained and it was a painful session of trial-and-error to get
my
gems working properly, and even then, I didn't know there was a way to
bypass
the "require_gem" statement.

If any more work is done on Gems, I strongly recommend it be on good
documentation, and then maintain those docs to update it for the changes
made
in the future.

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

17

That would work too - and you wouldn't have to wait for anyone to put
together that CGI script :slight_smile:

Yours,

Tom

···

On Thu, 2004-06-24 at 12:38, Curt Hibbs wrote:

> Is there someplace that non-RubyForge projects can upload their gems,
> so that they can also be listed with the "remotely installable" gems?
> I know that this is probably a question for Rich or Tom, but wanted to
> get it out there.

>From an outsider's perspective (i.e., I'm not Rich or Tom), you could always
create a RubyForge project to act as a release mirror for your project.

18

What I meant is that the more technical part of the problem is easily solved
(using openssl or the upcoming Crypt::RSA), but you still have to build
the web of trust (key signing parties, etc).

Which is also what I meant. The hard part here is not the technical, but the social.

For example, if I were to release a gem I'd like to sign it and provide my public key on the official project website, or the fingerprint of it which can be got from a public keyserver. That requires a manual check by the user to be of any use,

Need not be manual.

Either the repository keeps the public keys, some super-AI web scraper figures them out from project home pages, or it has to be manual. My above example was assuming neither repository having public keys nor any other automatic method of public key discovery, so in that case it has to be manual.

You cannot accept libs. + public keys if you haven't verified that the
key was really created by whoever claims it.

Right, keyring management would be separate and more carefully controlled. You can go to the extent that Debian does and require physical contact and ID and whatnot, or find some medium that people are comfortable with at a level somewhat below that. (that may be the best you can get though) Accepting the key with the gem wouldn't be any less secure than not signing them at all, of course, although it might give people a false sense of security.

Once you have implemented a signature system and have established the
web of trust, it's makes no sense not to let the end-user verify it
himself: online repositories can be cracked.

Yes, if the user has a local cache of the web of trust or relies on external public key servers. But if the user verifies it himself by downloading a public key from the repository there's no benefit because as you said, repositories can be cracked. I think you're right, though, have the verification done on the user's machine. It's more fun that way, anyhow. :slight_smile:

19

Jim Weirich said:

<nothing!>

I hate it when I hit send by accident!

Sean O'Dell said:

I didn't realize this either. Someone ought to write a definitive
tutorial on Gems and maintain it with each release. The examples
I found left a lot of things unexplained and it was a painful
session of trial-and-error to get my gems working properly, and
even then, I didn't know there was a way to bypass the "require_gem"
statement.

If any more work is done on Gems, I strongly recommend it be on good
documentation, and then maintain those docs to update it for the changes
made in the future.

We are currently reworking the command structure of the gem command. It
grew to be a confusing hodge-podge of options and suboptions. The new gem
command will follow the cvs model (e.g. gem install ...) and the options
will be clearly identified for each gem command. The internals will be
much cleaner as well.

At that time we will revisit the documentation and make sure it gets
updated. Documentation is high on our priority list.

Thanks for the feedback.

···

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

20

As a temporary fix, why not have the URLs at which gems can be
downloaded be derived from a secure hash (i.e., SHA digest) of the
file? It doesn't guarantee that your downloaded gem listing hasn't
been affected, but at least a given URL, once distributed as the
download location, can easily be checked -- if the actual hash of the
downloaded gem is the same as the hash component of the URL, you know
that the version you downloaded is the one advertised.

Going further, imagine a system with multiple repositories that
aggregate some reasonable number of packages -- let's say that
ruby-lang.org, rubyforge.org, and sourceforge.net all have public
package repositories set up, with a single public key for each
repository. If the SHA hashes for each gem are part of the repository
listing and URL at which it is downloaded, and the package listing
file is signed by the repository administrator(s), then you can have a
fairly secure distribution channel without a complex "web of trust".
Just get the (presumably well-known and mirrored all over the place)
public key for a repository, and you can download any package you like
from it without much worry over tampering.

Anyone see any obvious attacks or holes?

Lennon