Gems is over engineered

ruby-talk
1

There has been a lot of talk on core about Gems and how it interrelates
to various OS (file hierarchy) philosophies. Of significant issue is
the "DATADIR problem". Gems doesn't support a DATA dir at the moment.
Of course they are working on it and fitting it into the Gems
"philosophy" --which is that everything is stored in gems and the
outside looks there for its needs. For example, a BIN file actually is
a kind of stub that requires a file from its gem. Gems used to use LIB
stubs too, but that was replaced with a #require hack. Now how will
they fit DATA dir into this scheme? That's what they are trying to
figure out.

One of the problems with this, besides that it really irks other
distribution managers b/c it poopoos on their philosophies, is that it
adds Gems' versioning to all of it. So now you have not only versioned
libs, but bins and data too.

So I've been thinking about this and I've come to the conclusion that
all versioning belongs to the LIB only! A data dir is only relavent if
the data is not subject to versioning (which is atypical), otherwise it
belongs to the lib. This probably rubs not only Gems people wrong but
Debian people a little as well.

But think about it.

Consider the bin dir. Gems' use of wrapping the exe is clever but is
rather useless with regards to versioning. How are you going to tell
the exe to use a different version? ... So it's up to the designer of
the exe to add that if he really wants it, manually writing a stub is
easy enough, or somehthing that has a "--ver" option. Gems can't really
help. Debian deals with the same issue by using symlinks. ruby ->
ruby1.8 for example. Its' a nicer solution. But Gems can't use that
b/c it needs to be cross-platform and Windows does't have symlinks. So
were back to convention.

As for the data dir, what does it really matter is if the data is in
with the lib or outside? It really only needs to be outside if it is in
some way "shareable" and if that is the case versioning is bad indeed
since mutliple resources will call on it --which app has rightful
control? So at that point the data itself starts to become it's own
poject and probably should get an API.

In conclusion I think Gems is over-engineered. Tightly tying a
versioning distribution manager to Ruby's library require mechanics is
too much. And b/c of this will ultimately be bad for Ruby.

I believe a better solution may simply be to extend require to provide
optional parameters, among them special version constraints and then
Ruby can look for libs following a simple convention of appending
version to the lib dir name. So all versioned material goes in the lib,
the rest remains version free. It's a simple convetion, although not
ideal in some minor respects, it's simple and effective.

T.

2

There has been a lot of talk on core about Gems and how it
interrelates to various OS (file hierarchy) philosophies. Of
significant issue is the "DATADIR problem".

This is a *Ruby* problem that's accentuated by RubyGems, but is not
caused by RubyGems. The integration of RubyGems into the Ruby core is an
opportunity here to solve this for Ruby.

Gems doesn't support a DATA dir at the moment. Of course they are
working on it and fitting it into the Gems "philosophy" --which is
that everything is stored in gems and the outside looks there for its
needs.

This isn't quite correct.

For example, a BIN file actually is a kind of stub that requires a
file from its gem.

This is a stub, but it's a versioning stub. That is, the stub is aware
of versions installed and as such you only have to have a single stub
installed to be able to run multiple versions of a program offered by
Gems.

Gems used to use LIB stubs too, but that was replaced with a #require
hack.

The require hack has always been there, it was just made into something
that was cleaner. The stubs were a mistake because they didn't include
the full package.

Now how will they fit DATA dir into this scheme? That's what they are
trying to figure out.

This applies to Ruby as a whole -- remember that. Some platforms have
the harder problem that configuration is separate, too.

One of the problems with this, besides that it really irks other
distribution managers b/c it poopoos on their philosophies, is that it
adds Gems' versioning to all of it. So now you have not only versioned
libs, but bins and data too.

This is actually valid. If you read my statement to Jim Weirich on
ruby-core, you would have noted that at least for Ruwiki and
PDF::Writer, I *require* versioned data, not just version-independent
data. However, the bin versioning isn't the case as I outlined above.

So I've been thinking about this and I've come to the conclusion that
all versioning belongs to the LIB only!

This is incorrect, as noted above. Versioning can be applied to either
data or libs.

A data dir is only relavent if the data is not subject to versioning
(which is atypical), otherwise it belongs to the lib. This probably
rubs not only Gems people wrong but Debian people a little as well.

Um. I suspect that data versioning -- and remember we're talking about
"central" data, here, not user data -- is more important than you think.
If I change data formats and want people to be able to have both v2 and
v3 of my lib, they need to have *both* datafiles present, even if
they're named the same.

[...]

Consider the bin dir. Gems' use of wrapping the exe is clever but is
rather useless with regards to versioning. How are you going to tell
the exe to use a different version? ... So it's up to the designer of
the exe to add that if he really wants it, manually writing a stub is
easy enough, or somehthing that has a "--ver" option. Gems can't really
help.

Actually ... gems *does* handle it properly. Read the stubs that are in
your bin dir.

[...]

As for the data dir, what does it really matter is if the data is in
with the lib or outside?

This is where the Debian (and FreeBSD and ...) folks would disagree with
you, and I would by and large be inclined to agree. I'm told that Ruby
Gnome support is a particular mess for this.

[...]

In conclusion I think Gems is over-engineered. Tightly tying a
versioning distribution manager to Ruby's library require mechanics is
too much. And b/c of this will ultimately be bad for Ruby.

This is not a logical conclusion from your arguments, and it's certainly
not logical from the misinformation involved with your arguments. I
believe that it is precisely that versioning that makes RubyGems
*desirable* because there's nothing else even remotely suitable, and API
versioning is something that the *language* needs to deal with, not the
operating system.

I believe a better solution may simply be to extend require to provide
optional parameters, among them special version constraints and then
Ruby can look for libs following a simple convention of appending
version to the lib dir name.

I've already argued that "require 'foo-1.0'" is inappropriate (as well
as Ugly). This means that I am *always* fixed to a particular version
number in an application, rather than possibly getting a compatible
sub-version, as with RubyGems possible future capability:

  use_gem 'pdf-writer', '~> 1.1'
  require 'pdf/writer'

This will give me versions 1.1, 1.1.1, 1.1.2, all the way thorugh 1.1.9,
but *not* 1.2.0 or higher.

Sorry, but appending version to the lib dir name is completely
inappropriate and an utter mess. It looks stupid, too.

So all versioned material goes in the lib, the rest remains version
free. It's a simple convetion, although not ideal in some minor
respects, it's simple and effective.

It's not ideal in major respects. It's also neither simple nor effective
for the reasons noted above.

-austin

···

On 10/1/05, Trans <transfire@gmail.com> wrote:
--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

3

In article <1128184557.633795.12190@g43g2000cwa.googlegroups.com>,
transfire@gmail.com says...

Debian deals with the same issue by using symlinks. ruby ->
ruby1.8 for example. Its' a nicer solution. But Gems can't use that
b/c it needs to be cross-platform and Windows does't have symlinks. So
were back to convention.

Does Gems support Win95/98/ME? Windows 2000 and on DO have symlinks,
via NTFS's native "reparse points"; there's just no way to create them
with the existing command shell. SysInternals's "junction" tool works
very nicely, and IIRC there's source available for it. I've been using
them for years with no problems - in fact, they're more transparent to
apps than symlinks are.

···

--
Jay Levitt |
Wellesley, MA | I feel calm. I feel ready. I can only
Faster: jay at jay dot fm | conclude that's because I don't have a
http://www.jay.fm | full grasp of the situation. - Mark Adler

4

Austin Ziegler wrote:

> Gems doesn't support a DATA dir at the moment. Of course they are
> working on it and fitting it into the Gems "philosophy" --which is
> that everything is stored in gems and the outside looks there for its
> needs.

This isn't quite correct.

How so? Gems even caches data to the gems lib directory.

> For example, a BIN file actually is a kind of stub that requires a
> file from its gem.

This is a stub, but it's a versioning stub. That is, the stub is aware
of versions installed and as such you only have to have a single stub
installed to be able to run multiple versions of a program offered by
Gems.

Ah, the _1.0_ version parameter (I couldn't find any documentation on
this btw). I suppose that's no worse or better then symlinks. Still its
a rather trival thing fo rthe end programmer to do If they want. What
troubles me is that a manual install with setup.rb is gogin to be
different the a gems install --probably not a big deal in this case
(though I imagine thjere are edgecases, $0 comes to mind), but it does
illustrate how Gems alters things.

> One of the problems with this, besides that it really irks other
> distribution managers b/c it poopoos on their philosophies, is that it
> adds Gems' versioning to all of it. So now you have not only versioned
> libs, but bins and data too.

This is actually valid. If you read my statement to Jim Weirich on
ruby-core, you would have noted that at least for Ruwiki and
PDF::Writer, I *require* versioned data, not just version-independent
data. However, the bin versioning isn't the case as I outlined above.

The put the data with the lib. Why should it not be there if it is
versioning along with the lib. Why are .rb files special? What makes my
data and program different. lispers would tell you otherwise. The data
could just as well be encodded in Ruby. So what's the justification of
putting it elsewhere?

> So I've been thinking about this and I've come to the conclusion that
> all versioning belongs to the LIB only!

This is incorrect, as noted above. Versioning can be applied to either
data or libs.

Sure, but my point is then why separate them?

Um. I suspect that data versioning -- and remember we're talking about
"central" data, here, not user data -- is more important than you think.
If I change data formats and want people to be able to have both v2 and
v3 of my lib, they need to have *both* datafiles present, even if
they're named the same.

Right. I'm not saying any different.

> Consider the bin dir. Gems' use of wrapping the exe is clever but is
> rather useless with regards to versioning. How are you going to tell
> the exe to use a different version? ... So it's up to the designer of
> the exe to add that if he really wants it, manually writing a stub is
> easy enough, or somehthing that has a "--ver" option. Gems can't really
> help.

Actually ... gems *does* handle it properly. Read the stubs that are in
your bin dir.

You are right, I miss spoke here. The "_1.0_" paramter allows it, I
hadn't seen anything written up about this. It's a bit of an ugly hack,
but it does the job. Nonetheless, my point here is really that can be
trival for an end programmer to implement and then they have greater
control over it themselves.

> As for the data dir, what does it really matter is if the data is in
> with the lib or outside?

This is where the Debian (and FreeBSD and ...) folks would disagree with
you, and I would by and large be inclined to agree. I'm told that Ruby
Gnome support is a particular mess for this.

Ha! Ha! That's funny. You're argeesing with them now? Well, I don't see
why. I was agreeing with you!

> In conclusion I think Gems is over-engineered. Tightly tying a
> versioning distribution manager to Ruby's library require mechanics is
> too much. And b/c of this will ultimately be bad for Ruby.

This is not a logical conclusion from your arguments, and it's certainly
not logical from the misinformation involved with your arguments. I
believe that it is precisely that versioning that makes RubyGems
*desirable* because there's nothing else even remotely suitable, and API
versioning is something that the *language* needs to deal with, not the
operating system.

Sure it does, though admittedly I didn't fill in all the little steps
along the way. An what misinforamation, I had ONE fact wrong. They rest
are essentially opinion --which can't be misinformation. I'm not
arguing against versioning, just the particular approach. And I agree,
with you last stament here. But I might go further and say that the
language shouldn't limit versioning to a package distribution system.
If I want to do versioning but not use Gems how is Ruby going to help
me? Make Gems integral to Ruby and Gems becomes the only way.
VERSIONING SHOULD NOT BE TIED A PACKAGE DITRIBUTION MANGER.

> I believe a better solution may simply be to extend require to provide
> optional parameters, among them special version constraints and then
> Ruby can look for libs following a simple convention of appending
> version to the lib dir name.

I've already argued that "require 'foo-1.0'" is inappropriate (as well
as Ugly). This means that I am *always* fixed to a particular version
number in an application, rather than possibly getting a compatible
sub-version, as with RubyGems possible future capability:

You know what, the "ugly" arguments doen't fly here. I don't think its
ugly. I think it looks fine. And you're later starement is simply not
true. You're thinking inside the box. The require statment can be
adjusted to do the version lookup. By your reasoning Gems itself
wouldn't work.

Sorry, but appending version to the lib dir name is completely
inappropriate and an utter mess. It looks stupid, too.

> So all versioned material goes in the lib, the rest remains version
> free. It's a simple convetion, although not ideal in some minor
> respects, it's simple and effective.

It's not ideal in major respects. It's also neither simple nor effective
for the reasons noted above.

What reasons? You haven't given any facts or rational explination of
your opinions except a single incorrect one. "stupid" and "ugly" are
not helpful.

T.

5

Yeah, but that's platform dependent. Symlink solutions aren't necessary.

-austin

···

On 10/1/05, Jay Levitt <jay+news@jay.fm> wrote:

In article <1128184557.633795.12190@g43g2000cwa.googlegroups.com>,
transfire@gmail.com says...
> Debian deals with the same issue by using symlinks. ruby ->
> ruby1.8 for example. Its' a nicer solution. But Gems can't use that
> b/c it needs to be cross-platform and Windows does't have symlinks. So
> were back to convention.
Does Gems support Win95/98/ME? Windows 2000 and on DO have symlinks,
via NTFS's native "reparse points"; there's just no way to create them
with the existing command shell. SysInternals's "junction" tool works
very nicely, and IIRC there's source available for it. I've been using
them for years with no problems - in fact, they're more transparent to
apps than symlinks are.

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

6

Hello Jay,

them for years with no problems - in fact, they're more transparent to
apps than symlinks are.

IMHO, that is exactly what would develop into a problem once we start using
them. Applications that do directory traversal often go into endless
recursions over win32 symlinks; this has something to do with the fact that
you mention: symlinks are somewhat a blind spot, even with NTFS.

best regards
kaspar

···

--
code manufacture & ruby lab at tua.ch - und andere Domains günstig und einfach online kaufen auf top-domains.ch

7

Hi,

At Sun, 2 Oct 2005 03:14:02 +0900,
Austin Ziegler wrote in [ruby-talk:158553]:

···

On 10/1/05, Trans <transfire@gmail.com> wrote:
> There has been a lot of talk on core about Gems and how it
> interrelates to various OS (file hierarchy) philosophies. Of
> significant issue is the "DATADIR problem".

This is a *Ruby* problem that's accentuated by RubyGems, but is not
caused by RubyGems. The integration of RubyGems into the Ruby core is an
opportunity here to solve this for Ruby.

What do you consider needed for that purpose?

--
Nobu Nakada

8

Austin Ziegler wrote:

Gems doesn't support a DATA dir at the moment. Of course they are
working on it and fitting it into the Gems "philosophy" --which is
that everything is stored in gems and the outside looks there for
its needs.

This isn't quite correct.

How so? Gems even caches data to the gems lib directory.

This is a side effect of the problem not being solved under Ruby, not a
solution of Gems. That people are able to *easily* and *consistently* be
able to get at data from a gem and be assured that the data is
appropriate for the version of the gem. This is, however, one of the
things that repackagers hate *most*.

For example, a BIN file actually is a kind of stub that requires a
file from its gem.

This is a stub, but it's a versioning stub. That is, the stub is
aware of versions installed and as such you only have to have a
single stub installed to be able to run multiple versions of a
program offered by Gems.

Ah, the _1.0_ version parameter (I couldn't find any documentation on
this btw). I suppose that's no worse or better then symlinks. Still
its a rather trival thing for the end programmer to do If they want.
What troubles me is that a manual install with setup.rb is gogin to be
different the a gems install --probably not a big deal in this case
(though I imagine thjere are edgecases, $0 comes to mind), but it does
illustrate how Gems alters things.

That doesn't bother me at all. If you install with gems, you get
additional features. If you don't, you don't.

One of the problems with this, besides that it really irks other
distribution managers b/c it poopoos on their philosophies, is that
it adds Gems' versioning to all of it. So now you have not only
versioned libs, but bins and data too.

This is actually valid. If you read my statement to Jim Weirich on
ruby-core, you would have noted that at least for Ruwiki and
PDF::Writer, I *require* versioned data, not just version-independent
data. However, the bin versioning isn't the case as I outlined above.

The put the data with the lib. Why should it not be there if it is
versioning along with the lib. Why are .rb files special? What makes my
data and program different. lispers would tell you otherwise. The data
could just as well be encodded in Ruby. So what's the justification of
putting it elsewhere?

As I said above, it is the fact that the data is *with* the lib that
drives the repackagers nuts. I don't have as big an issue with it -- I
use that right now.

So I've been thinking about this and I've come to the conclusion
that all versioning belongs to the LIB only!

This is incorrect, as noted above. Versioning can be applied to
either data or libs.

Sure, but my point is then why separate them?

Hrm. Because FreeBSD and Debian and ... various other platforms demand
that they be separated or they're not right.

Um. I suspect that data versioning -- and remember we're talking
about "central" data, here, not user data -- is more important than
you think. If I change data formats and want people to be able to
have both v2 and v3 of my lib, they need to have *both* datafiles
present, even if they're named the same.

Right. I'm not saying any different.

Okay. But this is part of the reason why a solution is needed -- and
it's a *Ruby* problem.

Consider the bin dir. Gems' use of wrapping the exe is clever but is
rather useless with regards to versioning. How are you going to tell
the exe to use a different version? ... So it's up to the designer
of the exe to add that if he really wants it, manually writing a
stub is easy enough, or somehthing that has a "--ver" option. Gems
can't really help.

Actually ... gems *does* handle it properly. Read the stubs that are
in your bin dir.

You are right, I miss spoke here. The "_1.0_" paramter allows it, I
hadn't seen anything written up about this. It's a bit of an ugly
hack, but it does the job. Nonetheless, my point here is really that
can be trival for an end programmer to implement and then they have
greater control over it themselves.

Hmm. I *would* like to be be able to specify a stub that gets installed
rather than the default, but I think that the existing stub is
sufficient for most peoples' purposes.

As for the data dir, what does it really matter is if the data is in
with the lib or outside?

This is where the Debian (and FreeBSD and ...) folks would disagree
with you, and I would by and large be inclined to agree. I'm told
that Ruby Gnome support is a particular mess for this.

Ha! Ha! That's funny. You're argeesing with them now? Well, I don't
see why. I was agreeing with you!

I agree with them that there needs to be an API to get at the data
cleanly. Simply doing data relative to File.dirname(__FILE__) is ...
fragile. I disagree with them on other points.

In conclusion I think Gems is over-engineered. Tightly tying a
versioning distribution manager to Ruby's library require mechanics
is too much. And b/c of this will ultimately be bad for Ruby.

This is not a logical conclusion from your arguments, and it's
certainly not logical from the misinformation involved with your
arguments. I believe that it is precisely that versioning that makes
RubyGems *desirable* because there's nothing else even remotely
suitable, and API versioning is something that the *language* needs
to deal with, not the operating system.

Sure it does, though admittedly I didn't fill in all the little steps
along the way. An what misinforamation, I had ONE fact wrong.

They rest are essentially opinion --which can't be misinformation. I'm
not arguing against versioning, just the particular approach. And I
agree, with you last stament here. But I might go further and say that
the language shouldn't limit versioning to a package distribution
system. If I want to do versioning but not use Gems how is Ruby going
to help me? Make Gems integral to Ruby and Gems becomes the only way.
VERSIONING SHOULD NOT BE TIED A PACKAGE DITRIBUTION MANGER.

Um. Sorry, but do you actually have a solution that works, then? One
that *doesn't* tie versioning to package management? The idea of *not*
having the two pieces together is a bit silly, at that, as package
management is centrally about versioning, and API version management is
lacking in every language that *I* know (and that's quite a few). This
really does offer Ruby an opportunity to tie things together that work
well.

I believe a better solution may simply be to extend require to
provide optional parameters, among them special version constraints
and then Ruby can look for libs following a simple convention of
appending version to the lib dir name.

I've already argued that "require 'foo-1.0'" is inappropriate (as
well as Ugly). This means that I am *always* fixed to a particular
version number in an application, rather than possibly getting a
compatible sub-version, as with RubyGems possible future capability:

You know what, the "ugly" arguments doen't fly here. I don't think its
ugly. I think it looks fine. And you're later starement is simply not
true. You're thinking inside the box. The require statment can be
adjusted to do the version lookup. By your reasoning Gems itself
wouldn't work.

Oh, bollocks. The "ugly" argument flies quite well, especially when
we're talking about a language that we both consider quite beautiful,
although apparently for different reasons.

You can accuse me of thinking inside of the box, but the reality is that
both of these problems are interrelated -- and RubyGems is suited
perfectly for the combined problem.

So all versioned material goes in the lib, the rest remains version
free. It's a simple convetion, although not ideal in some minor
respects, it's simple and effective.

It's not ideal in major respects. It's also neither simple nor
effective for the reasons noted above.

What reasons? You haven't given any facts or rational explination of
your opinions except a single incorrect one. "stupid" and "ugly" are
not helpful.

"Incorrect." Sorry, it's not incorrect -- if I have to do:

  require 'foo-1.0'

I have tied myself irrevocably to version 1.0. If I can do

  require 'foo'

Then I'll get whatever the latest version is installed. If I can do:

  require 'foo', '~> 1.1'

Then I'll get the various versions as indicated. This is *best* handled
by something that doesn't actually have to scan *directories* to see if
there's an appropriate version, but can scan a quick database of
installed packages ... just like RubyGems does.

Package managers *always* have to do version management. The only
difference between RubyGems and other package managers is that Chad,
Jim, and the others who designed it also decided to handle API
versioning *at the same time*, since the API version and the package
version are often closely related.

Again, if you can point me to a solution that is *easy* and *not ugly*,
I'll evaluate it and, if appropriate, champion it. RubyGems solves two
major problems for Ruby, and I think it solves it elegantly.

-austin

···

On 10/1/05, Trans <transfire@gmail.com> wrote:
--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

9

Hi,

At Sun, 2 Oct 2005 03:14:02 +0900,
Austin Ziegler wrote in [ruby-talk:158553]:

There has been a lot of talk on core about Gems and how it
interrelates to various OS (file hierarchy) philosophies. Of
significant issue is the "DATADIR problem".

This is a *Ruby* problem that's accentuated by RubyGems, but is not
caused by RubyGems. The integration of RubyGems into the Ruby core is an
opportunity here to solve this for Ruby.

What do you consider needed for that purpose?

There is a need for Ruby to ease the separation of portable
libraries, architecture-dependent libraries, data files, and
documentation for those platforms which expect this. In short, we
need ruby's CONFIG hash to specify more places for things to go,
(even if for many platforms they end up pointing at the same place).
This would allow people to specify directories for things which are
shared between machines, and those that are machine specific, areas
which are likely to grow (like /var) and those of constant size, and
those areas which need read/write access and those that only need to
be read.

Quite which combinations will make the most people happy, I cannot
say.

--
Nobu Nakada

         HTH
         Hugh

···

On Wed, 5 Oct 2005, nobu.nokada@softhome.net wrote:

On 10/1/05, Trans <transfire@gmail.com> wrote:

10

Something more robust than CONFIG['data'] that can also deal with
multiple packages data and multiple versions of those packages.

I'm still trying to figure out a sane API for this concept. As of right
now, I don't have one.

-austin

···

On 10/5/05, nobu.nokada@softhome.net <nobu.nokada@softhome.net> wrote:

At Sun, 2 Oct 2005 03:14:02 +0900,
Austin Ziegler wrote in [ruby-talk:158553]:

On 10/1/05, Trans <transfire@gmail.com> wrote:

There has been a lot of talk on core about Gems and how it
interrelates to various OS (file hierarchy) philosophies. Of
significant issue is the "DATADIR problem".

This is a *Ruby* problem that's accentuated by RubyGems, but is not
caused by RubyGems. The integration of RubyGems into the Ruby core is
an opportunity here to solve this for Ruby.

What do you consider needed for that purpose?

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

11

Actually, there's some reasoning behind the madness:
- We try to support users that want read-only for certain directories
- We try to make it possible for users to share as much as possible between
different architectures

lib is by definition architecture specific (unshared) and can be read only,
while share is architecture neutral and possible to share. Both should be
able to be read-only except during install. Data that vary should be put in
/var.

This was originally intended to protect the share/ and lib/ directories from
corruption due to the amount of changes. Nowadays, it is more important that
it let the user take backup of static data (lib/, share/) only occasionally
and varying data frequently, and can have the varying data on different
kinds of media (e.g, RAID-0+1 instead of RAID-0 with backup).

There's reasons for each and every one of the directory recommendations,
some important, some not so important. I'd be happy to go through them with
you (tomorrow or tuesday) if you're so inclined. I don't know of any central
repository for the information.

Eivind.

···

On 10/2/05, Austin Ziegler <halostatue@gmail.com> wrote:

>>> So I've been thinking about this and I've come to the conclusion
>>> that all versioning belongs to the LIB only!
>> This is incorrect, as noted above. Versioning can be applied to
>> either data or libs.
> Sure, but my point is then why separate them?

Hrm. Because FreeBSD and Debian and ... various other platforms demand
that they be separated or they're not right.

--
Hazzle free packages for Ruby?
RPA is available from http://www.rubyarchive.org/

12

Austin Ziegler wrote:

That doesn't bother me at all. If you install with gems, you get
additional features. If you don't, you don't.

Fine. And say I use system X to get similar features. Then what? Will
Gems play nice? If Gems goes into core will that still fly? It becomes
a problem, because Gems has hijacked #require. So it's Gems Way or no
way, but it needs to be the Ruby Way or no way --which I think is
broader then what Gems is proposing.

As I said above, it is the fact that the data is *with* the lib that
drives the repackagers nuts. I don't have as big an issue with it -- I
use that right now.

Just wrap it in an .rb file, they won't know the difference :wink: Hey,
Lisp guys do it all the time!

But on this topic BTW, I find it interesting that Ruby libs are
installwed to a special site_ruby dir but data files that go with them
are put right into /usr/share, or what have you, with every thing else.
Seems to me this bolsters my point about shared data having a life of
it's own. But I also wonder, does site_ruby exist simply so Ruby knows
were to look for libs?

Hmm. I *would* like to be be able to specify a stub that gets installed
rather than the default, but I think that the existing stub is
sufficient for most peoples' purposes.

For most it is. But I agree with you. I think it's better just have a
little tool that can do this for you. Say you put a bin/ subdir in your
lib/ dir and a script wraps it automaically and puts it in /bin for
you, or something like that.

I agree with them that there needs to be an API to get at the data
cleanly. Simply doing data relative to File.dirname(__FILE__) is ...
fragile. I disagree with them on other points.

Well, that's easy enough. And I agree. __FILE__ just looks hackish to
begin with.

Um. Sorry, but do you actually have a solution that works, then? One
that *doesn't* tie versioning to package management? The idea of *not*
having the two pieces together is a bit silly, at that, as package
management is centrally about versioning, and API version management is
lacking in every language that *I* know (and that's quite a few). This
really does offer Ruby an opportunity to tie things together that work
well.

In fact, I may have a possible solution. And you are right, it is an
oppurtunity for Ruby, but one that should be considered extremely
carfully.

Oh, bollocks. The "ugly" argument flies quite well, especially when
we're talking about a language that we both consider quite beautiful,
although apparently for different reasons.

A wise Rubyists, once said "just don't look there". This applies here
as well. Your doing it aready if you really think about it.
Nonetheless, feel free to develop without the version in place and
simply append it upon distribution --you never have to see it.

You can accuse me of thinking inside of the box, but the reality is that
both of these problems are interrelated -- and RubyGems is suited
perfectly for the combined problem.

RubyGems addresses the problem. But perfectly?

While we want the the package manager to know about versions in order
to handle dependencies, the package manager doesn't need to dictate how
the API handles version constraint. (Imagine if apt-get tried that!)
Yet few ever really try to add API versioing to the software b/c they
assume it has to be the package manages domain, not theirs. Catch-22.

"Incorrect." Sorry, it's not incorrect -- if I have to do:

  require 'foo-1.0'

I have tied myself irrevocably to version 1.0. If I can do

  require 'foo'

Then I'll get whatever the latest version is installed. If I can do:

  require 'foo', '~> 1.1'

Then I'll get the various versions as indicated. This is *best* handled
by something that doesn't actually have to scan *directories* to see if
there's an appropriate version, but can scan a quick database of
installed packages ... just like RubyGems does.

Come on! Gems can do it but nothing else can? That's exactly what I
meant by inside the box. Require can be adjusted to handle the version
appendege and you know it. Isn't that essentially what Gems does?
Except the libs are tucked away in another specail gems directory. So
you got your *ugly* versioned directory names after all, you just don't
look there!

Package managers *always* have to do version management. The only
difference between RubyGems and other package managers is that Chad,
Jim, and the others who designed it also decided to handle API
versioning *at the same time*, since the API version and the package
version are often closely related.

I can understand that. They came at from on end and went on down. Now
consider it from the other end and work your way up. That's a more
interesting appraoch b/c it gets you to deal with API versioning first
and thus automatically makes it independent of the
packaging/installation mechinism.

Again, if you can point me to a solution that is *easy* and *not ugly*,
I'll evaluate it and, if appropriate, champion it. RubyGems solves two
major problems for Ruby, and I think it solves it elegantly.

To be honest I think RubyGems is pretty cool too. But I can't be
comfrotable unless John Doe can install his wares with a tar.gz or
..zip or apt-get or whatever and still take advantage of API versioning.
That should be the goal and the opporunity. Installing software is
realtively easy, doing the above is where the challenge lies.

T.

13

Hugh Sasse wrote:

> Hi,
>
> At Sun, 2 Oct 2005 03:14:02 +0900,
> Austin Ziegler wrote in [ruby-talk:158553]:
>>> There has been a lot of talk on core about Gems and how it
>>> interrelates to various OS (file hierarchy) philosophies. Of
>>> significant issue is the "DATADIR problem".
>>
>> This is a *Ruby* problem that's accentuated by RubyGems, but is not
>> caused by RubyGems. The integration of RubyGems into the Ruby core is an
>> opportunity here to solve this for Ruby.
>
> What do you consider needed for that purpose?

There is a need for Ruby to ease the separation of portable
libraries, architecture-dependent libraries, data files, and
documentation for those platforms which expect this. In short, we
need ruby's CONFIG hash to specify more places for things to go,
(even if for many platforms they end up pointing at the same place).
This would allow people to specify directories for things which are
shared between machines, and those that are machine specific, areas
which are likely to grow (like /var) and those of constant size, and
those areas which need read/write access and those that only need to
be read.

Quite which combinations will make the most people happy, I cannot
say.

If you think about it, we already have a mostly determined project
directory structure dictated by a target distribution, namely FHS
--what setup.rb/install.rb need to work. So FHS is our LCD. In a way
that's too bad. It would be nice if we could be free of such
constraint. In fact I can't even imagine being a Windows user (knowing
next to nothing about Linux) and understanding the arrangement.

So if I put a data or doc file next to the .rb file it goes with in
lib/, obviously that's considered bad by FHS. In fact setup.rb won't
even install it!

And there are other difficulties. I especially find it difficult to
manage a large project in that I can't organize my lib/ structure
independent of how the end-user will access the files inside. I could
have a script rearrange it before distribution, but then I can't easily
test my scripts from their development location because Ruby's require
path is too "frigid".

T.

···

On Wed, 5 Oct 2005, nobu.nokada@softhome.net wrote:
>> On 10/1/05, Trans <transfire@gmail.com> wrote:

14

Eivind Eklund wrote:

Actually, there's some reasoning behind the madness:
- We try to support users that want read-only for certain directories
- We try to make it possible for users to share as much as possible between
different architectures

I realize there's method behind tha madness. But I would point you to
Gobolinux for interesting couter-argumetns.

lib is by definition architecture specific (unshared) and can be read only,
while share is architecture neutral and possible to share. Both should be
able to be read-only except during install. Data that vary should be put in
/var.

Hmm... by that definition then most of site_ruby should be in share/.

This was originally intended to protect the share/ and lib/ directories from
corruption due to the amount of changes. Nowadays, it is more important that
it let the user take backup of static data (lib/, share/) only occasionally
and varying data frequently, and can have the varying data on different
kinds of media (e.g, RAID-0+1 instead of RAID-0 with backup).

Though I really wonder how many people do such a thing. Handling high
traffic/shared areas like home/ and var/ separately is a no brainer,
but splitting hairs over share/ and lib/, not to mention the local
variety. it gets to point where the "solution" is more trouble then the
problem. For some time now I've been convinced that we need to move
beyond these location based catagories --doing so would allows much
more fluid constraints and access.

There's reasons for each and every one of the directory recommendations,
some important, some not so important. I'd be happy to go through them with
you (tomorrow or tuesday) if you're so inclined. I don't know of any central
repository for the information.

A rundown posted here ceratainly might be helpful to many. I'm familiar
with the much of it, I've quickly read over the FHS. And I am a Debian
user. I just think it's getting dated. Actually I worry about MS' WinFS
for Linux' sake. It may be behind schedule, but that's what MS is
working on to be sure.

T.

15

Hugh Sasse wrote:

What do you consider needed for that purpose?

         [...]

documentation for those platforms which expect this. In short, we
need ruby's CONFIG hash to specify more places for things to go,

         [...]

Quite which combinations will make the most people happy, I cannot
say.

If you think about it, we already have a mostly determined project
directory structure dictated by a target distribution, namely FHS
--what setup.rb/install.rb need to work. So FHS is our LCD. In a way

I don't know how/if this conflicts with other platforms. I don't
use many.

that's too bad. It would be nice if we could be free of such
constraint. In fact I can't even imagine being a Windows user (knowing
next to nothing about Linux) and understanding the arrangement.

It would certainly be different under Windows. And OpenVMS I
suspect.

So if I put a data or doc file next to the .rb file it goes with in
lib/, obviously that's considered bad by FHS. In fact setup.rb won't
even install it!

Yes, it is a problem.

And there are other difficulties. I especially find it difficult to
manage a large project in that I can't organize my lib/ structure
independent of how the end-user will access the files inside. I could
have a script rearrange it before distribution, but then I can't easily
test my scripts from their development location because Ruby's require
path is too "frigid".

If we add things to CONFIG, maybe we need to add methods to
something, maybe $: or CONFIG itself, such as #find_docs(thing),
#find_lib(alibrary), #find_for_this_arch(binary). In that way (only)
the correct areas would be searched. Making path handling more like
accessing an object would seem to bring benefits of encapsulation.

That paragraph was simply brainstorming. I'm not really sure how the
ideal interface should look, but if we can make it look the same for
everyone it would help. I mean, I think you wouldn't mind where
your docs or library end up as long as you can get at them *easily* on
the target. Is that right? If the access works, it could even be a
database and not matter? Then we could make what is encapsulated
different for each group of [Debian, Windows, Solaris] users....

T.

         Hugh [who was sure his sanity was here *somewhere*! :-)]

···

On Thu, 6 Oct 2005, Trans wrote:

On Wed, 5 Oct 2005, nobu.nokada@softhome.net wrote:

16

Eivind Eklund wrote:
> Actually, there's some reasoning behind the madness:
> - We try to support users that want read-only for certain directories
> - We try to make it possible for users to share as much as possible between
> different architectures

I realize there's method behind tha madness. But I would point you to
Gobolinux for interesting couter-argumetns.

[Gobolinux use a one-dir-per-package model. -Eivind]

I think both models should be supported. Gentoo use the same model as
GoboLinux, I believe. The problem I have is with people coming and
telling me "I am going to force all FreeBSD and Debian and Red Hat and
etc users to go with my completely incompatible setup, because your
work is too complex for me and I've tried out packaging 0.5% as many
packages as you have for 1% of the number of users you have and my way
is better and there's no reason for you to do all your stuff cause
mine has worked fine so all your complexity is just stupid."

I'm not going to tell Gobolinux not to experiment with this. I think
Gobolinux experimenting with this is GOOD. If that works out really
really well, it will take over the Linux packaging world, and we'll
just kill the *BSD and Debian and PLD and etc packaging models and the
Linux Filesystem Hierarchy Standard, migrating to a system that works
a la Gobolinux and RubyGems.

Until we do, however, I think that having Ruby implemented in a way
that makes it possible to follow the standards reasonably well is
necessary.

And until I stop hearing complaints that it's impossible to package
Rails and RailsGen properly (which I last heard from a new repackager
a few days ago), I won't think this problem is solved.

> lib is by definition architecture specific (unshared) and can be read only,
> while share is architecture neutral and possible to share. Both should be
> able to be read-only except during install. Data that vary should be put in
> /var.

Hmm... by that definition then most of site_ruby should be in share/.

Yes. site_ruby is (AFAIK) inherited from site_perl, and I think that
predated the proper share/ splitting.

> This was originally intended to protect the share/ and lib/ directories from
> corruption due to the amount of changes. Nowadays, it is more important that
> it let the user take backup of static data (lib/, share/) only occasionally
> and varying data frequently, and can have the varying data on different
> kinds of media (e.g, RAID-0+1 instead of RAID-0 with backup).

Though I really wonder how many people do such a thing.

Not many for each particular case. The ones that do are often very
much power users, however, and contribute disproportionally to the
system, and additionally there is a much larger number that don't use
it, yet feel more comfortable with the system because it's there in
case they need it.

Handling high
traffic/shared areas like home/ and var/ separately is a no brainer,
but splitting hairs over share/ and lib/, not to mention the local
variety. it gets to point where the "solution" is more trouble then the
problem. For some time now I've been convinced that we need to move
beyond these location based catagories --doing so would allows much
more fluid constraints and access.

I don't care that much about share/ and lib/ either, and I don't know
of any OS that care much about keeping architecture independent data
out of lib (though everybody tries to keep architecture dependent data
out of share/). The share/ and lib/ split is just an optimization
when it comes to data files. It's more important for documentation
and examples, so the users can find the docs when they look for them.

> There's reasons for each and every one of the directory recommendations,
> some important, some not so important. I'd be happy to go through them with
> you (tomorrow or tuesday) if you're so inclined. I don't know of any central
> repository for the information.

A rundown posted here certainly might be helpful to many.

I would prefer to go through with Austin privately and then do the
public rundown afterwards; initial discussions with too many
participants tend to be less than fruitful.

I'm familiar
with the much of it, I've quickly read over the FHS. And I am a Debian
user. I just think it's getting dated.

I think it's important to go through and find exactly what parts are
worth support and which parts we'll just want to say "So long with
that". After having gone through and found what we think we can drop,
I'd like to discuss it with some more repackagers, to make sure we get
a good solution.

Actually I worry about MS' WinFS for Linux' sake. It may be behind schedule, but that's
what MS is working on to be sure.

I'm not particularly worried. KDE is working on technology that fill
similar goals, GNOME has some already, ReiserFS include some parts,
and overall I don't think it makes that much of a difference.

If anything, I'm worried by .NET. This kind of infrastructure could
become a security must on a more hostile Internet, and that could put
a several year dent in Open Source technology (and even more in open
source trust.)

Eivind.

···

On 10/3/05, Trans <transfire@gmail.com> wrote:

17

Interesting brainstorming Hugh.

I think it may help if we break this down to the fundamentals. Imagine
if we had no $LOAD_PATH. Imagine we had to spell everything out in
absolutes. But we don't know where exactly our files are going to end
up on the end system. How do we account for this? Well, we need a map
--a map of aliases my scripts will use to refer to it's contents (and
others can use to refer to it) that point to actual locations.

I've worked out a partial solution. I'll release it soon. But it only
does the mapping internally for the lib/ dir. To go beyond that really
means modifying Ruby to handle this more gracefully as well as adding a
tool to help generate distribution specific maps based on the alias map
provided by the developers.

In essense giving Ruby a file registry system.

T.

18

Eivind Eklund wrote:

[Gobolinux use a one-dir-per-package model. -Eivind]

I think both models should be supported. Gentoo use the same model as
GoboLinux, I believe. The problem I have is with people coming and
telling me "I am going to force all FreeBSD and Debian and Red Hat and
etc users to go with my completely incompatible setup, because your
work is too complex for me and I've tried out packaging 0.5% as many
packages as you have for 1% of the number of users you have and my way
is better and there's no reason for you to do all your stuff cause
mine has worked fine so all your complexity is just stupid."

I'm not going to tell Gobolinux not to experiment with this. I think
Gobolinux experimenting with this is GOOD. If that works out really
really well, it will take over the Linux packaging world, and we'll
just kill the *BSD and Debian and PLD and etc packaging models and the
Linux Filesystem Hierarchy Standard, migrating to a system that works
a la Gobolinux and RubyGems.

Until we do, however, I think that having Ruby implemented in a way
that makes it possible to follow the standards reasonably well is
necessary.

And until I stop hearing complaints that it's impossible to package
Rails and RailsGen properly (which I last heard from a new repackager
a few days ago), I won't think this problem is solved.

Actually that's interesting. Perhaps the authors of the packages
themselves should have some sort of specfile to designate information
about their files which another build tool could use to build the
layout for particular distribution's needs. So the build tool could
have different adapters, FHS adaptor, Gentoo Adapter, Windows
Adaptor... not sure how the specfile mihgt look though.

> Hmm... by that definition then most of site_ruby should be in share/.

Yes. site_ruby is (AFAIK) inherited from site_perl, and I think that
predated the proper share/ splitting.

Interesting.

> Though I really wonder how many people do such a thing.

Not many for each particular case. The ones that do are often very
much power users, however, and contribute disproportionally to the
system, and additionally there is a much larger number that don't use
it, yet feel more comfortable with the system because it's there in
case they need it.

That makes sense. Though I imagine, even in their case some of it is
overkill. Plus consider that even if it is all quite rational, if it is
just too complex how effective is it gogin to be anyway?

I would prefer to go through with Austin privately and then do the
public rundown afterwards; initial discussions with too many
participants tend to be less than fruitful.

Okay.

I think it's important to go through and find exactly what parts are
worth support and which parts we'll just want to say "So long with
that". After having gone through and found what we think we can drop,
I'd like to discuss it with some more repackagers, to make sure we get
a good solution.

Quite resonable.

> Actually I worry about MS' WinFS for Linux' sake. It may be behind schedule, but that's
> what MS is working on to be sure.

I'm not particularly worried. KDE is working on technology that fill
similar goals, GNOME has some already, ReiserFS include some parts,
and overall I don't think it makes that much of a difference.

I'm not sure. Perhaps. Sometimes it seems like their just adding layer
upon layer rather reworking inot something original.

If anything, I'm worried by .NET. This kind of infrastructure could
become a security must on a more hostile Internet, and that could put
a several year dent in Open Source technology (and even more in open
source trust.)

That's true too :frowning:

Thanks for these well put replys.

T.

P.S. Where does one put demo/example files in Debian/FHS?

19

Interesting brainstorming Hugh.

Thank you.

I think it may help if we break this down to the fundamentals. Imagine
if we had no $LOAD_PATH. Imagine we had to spell everything out in
absolutes. But we don't know where exactly our files are going to end
up on the end system. How do we account for this? Well, we need a map
--a map of aliases my scripts will use to refer to it's contents (and
others can use to refer to it) that point to actual locations.

Completely agree...

I've worked out a partial solution. I'll release it soon. But it only
does the mapping internally for the lib/ dir. To go beyond that really
means modifying Ruby to handle this more gracefully as well as adding a
tool to help generate distribution specific maps based on the alias map
provided by the developers.

Yes, it would have to be part of Ruby itself. Bringing it into the
whole Object hierarchy would probably allow some really fun things
to happen. For example, one of the Repackager's problems is how to
distribute the files over a heterogeneous network, with machines
having different architectures. Well, if one can get documentation
and libraries by calling some methods, they could be overridden to
get the objects (files?) from a server instead. How far could one
reduce Ruby so that only the networking-with-authentication code ran
locally, and the rest of the language was grabbed from the server?
Stdlib in a TupleSpace.....?

In essense giving Ruby a file registry system.

Yes, though I'd want to avoid the word "registry" because of how
Microsoft have given the term associations: "Do Not Touch",
"Danger[, Will Robinson]!" :slight_smile:

T.

         Thank you,
         Hugh

···

On Thu, 6 Oct 2005, Trans wrote:

20

Eivind Eklund wrote:
> [Gobolinux use a one-dir-per-package model. -Eivind]
>
> I think both models should be supported. Gentoo use the same model as
> GoboLinux, I believe.

No, it doesn't. Gentoo uses a FHS-like model like all other Unix/Linux.

[...]

P.S. Where does one put demo/example files in Debian/FHS?

/usr/share/doc/packagename/examples/

···

On 05/10/05 at 05:36 +0900, Trans wrote:
--

Lucas Nussbaum
lucas@lucas-nussbaum.net http://www.lucas-nussbaum.net/ |
jabber: lucas@nussbaum.fr GPG: 1024D/023B3F4F |