Including other people's Ruby libs in your package

ruby-talk
1

I'm writing some code that depends on MIME::Types, and works better with (but does not depend on) BlueCloth.

I'd like to package my code as both a Gem and as a more conventional "install using cp" bundle.

What sort of project directory layout might work best?

Gem will go fetch dependencies, so I think I'm covered there (I can leave these other files out of the gem and list them as dependencies), but what might be some good installation techniques when not using a Ruby gem?

I'm thinking now of putting the third-party Ruby files into a local subdirectory of the installed code, and munging the load path in my code, prior to calling require, to add this local directory. If the user already has mime/types.rb, great, the global version gets called. Otherwise 'require' should be able to find the copy installed in the relative directory.

Thoughts?

James

2

I'm thinking now of putting the third-party Ruby files into a local
subdirectory of the installed code, and munging the load path in my
code, prior to calling require, to add this local directory. If the
user already has mime/types.rb, great, the global version gets called.
Otherwise 'require' should be able to find the copy installed in the
relative directory.

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.

It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

Ari

3

i use this layout

a top level 'include' file. one versioned, one not. this allows me install
multiple versions and do like 'require "package-0.0.0" when i want to link to a
specific version. doing a 'require "package"' will always pick up the latest
greatest.

   lib/package-0.0.0.rb
   lib/package.rb

     unless defined? $__package__
       module Package
         LIBNAME = "package"
         VERSION = "0.0.0"
         LIBVER = "#{ LIBNAME }-#{ VERSION }"
         DIRNAME = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR
         ROOTDIR = File.dirname(DIRNAME)
         LIBDIR = File.join(DIRNAME, LIBVER) + File::SEPARATOR

         require LIBDIR + 'a'
         require LIBDIR + 'b'
       end
       $__package__ = __FILE__
     end

actual package impl lives in a versioned directory (for same reason as above).
these files set alot of the same parameters as the include files. these vars
allow me to do relative requires and also allow mutual requies. this is
extremely nice for unit testing since each class can require independantly the
files it needs with no fear of double requires do the the 'require "a" -
require "a.rb"' bug.

   lib/package-0.0.0/a.rb

     unless defined? $__package_a__
       module Package
         LIBDIR = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR unless
           defined? LIBDIR

         require LIBDIR + 'b'

         class A
         end
       end
     $__package_a__ = __FILE__
     end

   lib/package-0.0.0/b.rb

     unless defined? $__package_b__
       module Package
         LIBDIR = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR unless
           defined? LIBDIR

         require LIBDIR + 'a'

         class B
         end
       end
     $__package_b__ = __FILE__
     end

this seems a pain but i have code generator scripts to do the work.

cheers.

-a

···

On Fri, 30 Jul 2004, James Britt wrote:

What sort of project directory layout might work best?

--

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
A flower falls, even though we love it;
and a weed grows, even though we do not love it. --Dogen

===============================================================================

4

What sort of application is it? (I'm curious, of course, as I wrote
MIME::Types :slight_smile:

With Ruwiki, we depended on Algorithm::Diff -- and we will be
depending on Diff::LCS -- and I will be packaging Diff::LCS with
Ruwiki. This is because Ruwiki is intended to be standalone, except
for the Ruby core libraries. Ruwiki ensures that its versions will
be loaded in preference to system versions by modifying $LOAD_PATH
such that the Ruwiki library directory is first.

If and when I make it "installable" rather than an "unpack and run"
program, I will do further work with the install.rb such that I do a
"require 'diff/lcs'" (and any other dependencies); if it is not
found, then I will install Diff::LCS. If it is found, then I will
check to make sure that it is at least version 1.1. If not, then I
will prompt the user for replacement. Now, the real problem is that
Ruwiki -- as of right now -- only includes those parts of Diff::LCS
which are necessary to run Ruwiki. Thus, if I take that route, I
will probably offer multiple packages.

An alternative -- and perhaps better way for the .tar.gz concept --
is to provide the required and optional packages as .tar.gz packages
within one of your distribution packages. This means that
OS-specific packagers don't have to strip out the code by hand, and
you don't have to worry about those tests.

  bin/
  dep/required/mime-types-1.13.tar.gz
  dep/optional/bluecloth-0.3.2.tar.gz
  ...

That said, if you have a standalone situation as we have with
Ruwiki, feel free to include MIME::Types or any of my other
packages. I have RubyForge projects for MIME::Types, Text::Format,
and Transaction::Simple, now -- I just haven't announced them.

-austin

···

On Fri, 30 Jul 2004 06:18:18 +0900, James Britt <jamesunderbarb@neurogami.com> wrote:

I'm writing some code that depends on MIME::Types, and works
better with (but does not depend on) BlueCloth.

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

5

Aredridel wrote:

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.

It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

I've considered this, but find it potentially too annoying to the user.

I would prefer that people be able to just download a tarball, extract it, and copy over a directory to the libs dir. And it just works.

James

6

Austin Ziegler wrote:

I'm writing some code that depends on MIME::Types, and works
better with (but does not depend on) BlueCloth.

What sort of application is it? (I'm curious, of course, as I wrote
MIME::Types :slight_smile:

Well, if you *must* know, it takes either a string of XHTML or plain-text and produces a MIME multipart/alternative message body. The code looks at the given string and automagically creates the alternative rendering (given, XHTML, it also creates a plain-text version, and vice versa. Works best when handed a BlueCloth string, but the plain-to-peanuts conversion is configurable).

If the resulting (or given) XHTML contains local references to images, they are encoded and added to the message body as multipart/related; the img src value is replace with a cid: ref to the encoded image segment.

MIME::Types is used to figure out the MIME type of the image.

With Ruwiki, we depended on Algorithm::Diff -- and we will be
depending on Diff::LCS -- and I will be packaging Diff::LCS with
Ruwiki. This is because Ruwiki is intended to be standalone, except
for the Ruby core libraries. Ruwiki ensures that its versions will
be loaded in preference to system versions by modifying $LOAD_PATH
such that the Ruwiki library directory is first.

I've got a similar deal with Blogtari (also stand-alone, with no assumptions about user permissions other than their on httpdocs or public_html directory), which includes RedCloth and BlueCloth, but it will try to first load them from the standard path.

If and when I make it "installable" rather than an "unpack and run"
program, I will do further work with the install.rb such that I do a
"require 'diff/lcs'" (and any other dependencies); if it is not
found, then I will install Diff::LCS. If it is found, then I will
check to make sure that it is at least version 1.1. If not, then I
will prompt the user for replacement. Now, the real problem is that
Ruwiki -- as of right now -- only includes those parts of Diff::LCS
which are necessary to run Ruwiki. Thus, if I take that route, I
will probably offer multiple packages.

Blogtari has its own installation routine. Everything is packed up using a hacked version of Tar2Rubyscript (quite slick); there's code in there to prompt the user for configuration details, as well as try to managed upgrades without stomping existing user config data or templates. But nothing to verify available dependencies.

An alternative -- and perhaps better way for the .tar.gz concept --
is to provide the required and optional packages as .tar.gz packages
within one of your distribution packages. This means that
OS-specific packagers don't have to strip out the code by hand, and
you don't have to worry about those tests.

  bin/
  dep/required/mime-types-1.13.tar.gz
  dep/optional/bluecloth-0.3.2.tar.gz
  ...

That's a really good idea. I think my biggest gripe with packages with multiple dependencies is that they send you off on a series of hunt and fetch operations. (This is *especially* annoying if you've loaded your laptop with assorted libs to examine for a presentation you plan on writing on the plane ride to a conference. Or so I hear.)

So simply bundling the required libs in such a way that the user at least has them to install is a good compromise. And I like the directory tree.

That said, if you have a standalone situation as we have with
Ruwiki, feel free to include MIME::Types or any of my other
packages. I have RubyForge projects for MIME::Types, Text::Format,
and Transaction::Simple, now -- I just haven't announced them.

Sweet. Thanks.

James

···

On Fri, 30 Jul 2004 06:18:18 +0900, James Britt > <jamesunderbarb@neurogami.com> wrote:

-austin

7

James Britt wrote:

Aredridel wrote:

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.
It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

I've considered this, but find it potentially too annoying to the user.

I would prefer that people be able to just download a tarball, extract it, and copy over a directory to the libs dir. And it just works.

On the other hand, it can be very annoying for a user as well when they already have a *newer* version of that same dependency installed, and then they install your library and suddenly things break.

If nothing else, provide two versions of your package: one *with* dependencies, and one without. I'm in the camp that prefers applications NOT be shipped with their dependencies included, but I can understand that others may prefer the other way.

The moral of the story is that there isn't a perfect solution to this, unfortunately. :frowning: Rubygems' dependency feature comes close, though.

···

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

"I use octal until I get to 8, and then I switch to decimal."

8

Jamis Buck wrote:

James Britt wrote:

Aredridel wrote:

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.
It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

I've considered this, but find it potentially too annoying to the user.

I would prefer that people be able to just download a tarball, extract it, and copy over a directory to the libs dir. And it just works.

On the other hand, it can be very annoying for a user as well when they already have a *newer* version of that same dependency installed, and then they install your library and suddenly things break.

That's right. So let's don't do that. :slight_smile:

That question is, What's the best way?

One approach is change $: to look in some specified directory, as a last resort, if the standard path does not turn up the needed lib.

So, for example, if BlueCLoth is already installed, then the pre-packaged copy is never loaded because a "correctly" installed version has been found.

If my lib is installed in /foo/bar and my class file does
    $:.push "/foo/bar/dependencies" # or something along those lines
then my special directory should be the last place Ruby looks for a file. Or no?

If nothing else, provide two versions of your package: one *with* dependencies, and one without. I'm in the camp that prefers applications NOT be shipped with their dependencies included, but I can understand that others may prefer the other way.

The moral of the story is that there isn't a perfect solution to this, unfortunately. :frowning: Rubygems' dependency feature comes close, though.

For Gems, it might be nice (or feature-creep overkill) to specify that, if a user refuses to install dependency Foo, then fail the installation, but if they refuse pseudo-dependency Bar, that's OK, keep going with a "You're going to miss some good stuff" warning.

9

I've had this on the list for a while, but I'm still not sure if it's
feature creep or a good feature to add. I have what I think is a
healthy uneasiness about adding anything to the gem spec, but
something like this may just find it way in.

Chad

···

On Fri, 30 Jul 2004 08:04:25 +0900, James Britt <jamesunderbarb@neurogami.com> wrote:

For Gems, it might be nice (or feature-creep overkill) to specify that,
if a user refuses to install dependency Foo, then fail the installation,
but if they refuse pseudo-dependency Bar, that's OK, keep going with a
"You're going to miss some good stuff" warning.

10

Chad Fowler wrote:

For Gems, it might be nice (or feature-creep overkill) to specify that,
if a user refuses to install dependency Foo, then fail the installation,
but if they refuse pseudo-dependency Bar, that's OK, keep going with a
"You're going to miss some good stuff" warning.

I've had this on the list for a while, but I'm still not sure if it's
feature creep or a good feature to add. I have what I think is a
healthy uneasiness about adding anything to the gem spec, but
something like this may just find it way in.

An idea I floated on another list (and somewhat off-topic for the main discussion there at the time) was having some way to vet code before automatically installing it.

Run a set of sanity-check tests over the proposed code (for example, look for references to $SAFE, alterations of base classes, upgrades/downgrades of existing files, and the use of camelCase for method names), then report a score of some sort.

The user can then decide that maybe the installation isn't a good idea, or it presents some risk, or would fill the hard drive, and cancel the installation.

There have been a number of times I installed Perl code using the CPAN shell, astounded by the number of extra libs that were installed as dependencies. I thought I had a new screen saver.

Is there some simple way to plug in additional operations along the RubyGems installation process path, such that people can write a class and tell RubyGems, "When you get to this step, run this code, passing in the directories of the extracted but not-yet-installed gems"?

Then write an XML (oops, I mean YAML) file that defines a customized processing path.

Or something. But then folks who wanted to embark on feeping creaturism could have a field day.

James

···

On Fri, 30 Jul 2004 08:04:25 +0900, James Britt > <jamesunderbarb@neurogami.com> wrote: