[ANN] pallet 1.2.1

ruby-talk
1

I've been writing a multiplatform packaging tool for the company I work
for, dubbed "pallet". I recently got the go-ahead to release it on
RubyForge under the LGPL, so have spent the past day or so getting it up
on there.

Essentially, pallet lets you use a simple Rakefile stanza (akin to the
existing gemspec stanza) to define multiple packaging formats. Right now,
gem and dpkg are the only formats supported, although adding .tar.gz
should be trivial, and .rpm should be simple enough for anyone who's built
an RPM.

An example of the format, from pallet's own Rakefile:

  Pallet.new('pallet', Pallet::VERSION) do |p|
  
    p.author = 'Stephen Touset'
    p.email = 'stouset@damballa.com'
    p.summary = 'Automated packaging tool for different languages and OSes'
  
    p.packages << Pallet::Gem.new(p) do |gem|
      gem.depends = ['rake']
      gem.requirements = ['fakeroot', 'dpkg-dev']
      gem.files.include FileList['share/**/*']
    end
  
    p.packages << Pallet::Deb.new(p) do |deb|
      deb.architecture = 'all'
      deb.depends = ['rake', 'fakeroot', 'dpkg-dev']
      deb.files = [InstallSpec['lib', '/usr/lib/ruby/1.8'],
                          InstallSpec['share', '/usr/share/pallet'],]
      deb.commands.document = 'rake doc'
    end
  
  end

It doesn't support every single possible feature of every packaging
system, but it's close enough for 99% of projects. Nearly thirty of our
own internal software projects were ported over to it as part of our build
system in a matter of hours by two developers with no previous experience
with either the packaging systems or pallet.

One nice ability is that it should require no more than maybe ten lines to
build a Debian package that simply contains a gem that is
installed/uninstalled from rubygems by postinst/prerm scripts in the
Debian package. In a future release, I expect to make this a trivial
use-case, through something like

  p.packages << Pallet::Deb.new(p, :include => :gem)

It is extensively rdoc'd, available as "pallet" from rubygems, and can be
downloaded as a dpkg or a gem from http://rubyforge.org/projects/pallet/

Happy packing!

···

--
Stephen Touset <stephen@touset.org>

2

As part of this release, I'd like to extend a (limited) offer to
personally package up a few -- maybe five to ten, time-willing -- willing
open source projects. Please reply to this message if you have a project or
library that would benefit from being packaged as a .deb (preferably also
a .gem). Higher-profile projects will be dealt with first.

Since I didn't include it in the original post, I want to clarify that
pallet can be used to package software from _any_ language and integrate
into any build system -- not just Ruby using Rake. Most of the projects we
have at the company I developed this for are written in C built with
Makefiles or Java build with ant. None of these required any extra
effort by our developers to package.

···

On Tue, 01 May 2007 12:53:18 -0500, Stephen Touset wrote:

I've been writing a multiplatform packaging tool for the company I work
for, dubbed "pallet". I recently got the go-ahead to release it on
RubyForge under the LGPL, so have spent the past day or so getting it up
on there.

--
Stephen Touset <stephen@touset.org>

3

I'd definitely be interested in RPM.

Right now I'm using a different approach: gem2spec takes a gem and turns it
into an RPM spec file. Unlike the native .deb packages you're generating
under Pallet, the components are installed under /usr/lib/ruby/gems/1.8 not
/usr/lib/ruby/1.8.

$ rpm -ql rubygem-rake
/usr/bin/rake
/usr/lib/ruby/gems/1.8/cache/rake-0.7.1.gem
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/rdoc
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/rdoc/classes
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/rdoc/classes/CompositePublisher.html
...
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/ri
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/ri/CompositePublisher
/usr/lib/ruby/gems/1.8/doc/rake-0.7.1/ri/CompositePublisher/add-i.yaml
...
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/CHANGES
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/MIT-LICENSE
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/README
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/Rakefile
..
/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/test/test_test_task.rb
/usr/lib/ruby/gems/1.8/specifications/rake-0.7.1.gemspec

With Pallet, you can tell it what files to bundle together to build a
package. But what about the initial fetch/untar/configure/build/install
steps? It would be very attractive to write these once in ruby (or rake),
but I don't think Pallet provides rules for this. Are there existing
libraries of rake tasks to do these steps?

Regards,

Brian.

P.S. Another tool I've used occasionally is Debian's "alien" (e.g. to
install the oracle-instantclient RPMs on an Ubuntu box). I've never tried
using alien to convert a .deb to .rpm though.

···

On Wed, May 02, 2007 at 02:55:07AM +0900, Stephen Touset wrote:

Essentially, pallet lets you use a simple Rakefile stanza (akin to the
existing gemspec stanza) to define multiple packaging formats. Right now,
gem and dpkg are the only formats supported, although adding .tar.gz
should be trivial, and .rpm should be simple enough for anyone who's built
an RPM.

4

Hi Stephen,

* Stephen Touset <stephen@touset.org> [01.05.2007]:

I've been writing a multiplatform packaging tool for the company I work
for, dubbed "pallet". I recently got the go-ahead to release it on
RubyForge under the LGPL, so have spent the past day or so getting it up
on there.

oh, this is great! I just grabbed it and like it. :slight_smile:

one small note:

the created debian files contain references to what i think is one of
your personal projects: non-free/damballa (see debian/files for
examples). Maybe this can be customized when you debianize a project.

Thanks a lot for making this public!

Cheers,

- Steph.

5

Stephen:

I've been writing a multiplatform packaging tool for the company I
work
for, dubbed "pallet". I recently got the go-ahead to release it on
RubyForge under the LGPL, so have spent the past day or so getting it
up
on there.

*snip*

Since I didn't include it in the original post, I want to clarify that
pallet can be used to package software from _any_ language and integrate
into any build system -- not just Ruby using Rake. Most of the projects
we
have at the company I developed this for are written in C built with
Makefiles or Java build with ant. None of these required any extra
effort by our developers to package.

This sounds extremely interesting Stephen: I would expect quite a lots
of interst, but there don't seem to be replies on the list. I wonder if
people haven't quite understood what you've released? Is it some kind of
meta package description that can be compiled in to vareious different
kinds of package? Or have I got completely the wrong end of the stick?

:slight_smile:

Cheers,
  Benjohn

···

On Tue, 01 May 2007 12:53:18 -0500, Stephen Touset wrote:

6

I've been writing a multiplatform packaging tool for the company I work
for, dubbed "pallet".

[...]

Right now, gem and dpkg are the only formats supported, although adding .tar.gz should be trivial, and .rpm should be simple enough for anyone who's built an RPM.

Have you been talking to Debian's Ruby packagers? Mutating this into something like "rake dpkg", that would automagically turn any ruby gem into a Debian package would be brilliant :-o
*t

···

On Wed, 2 May 2007, Stephen Touset wrote:

--
-----------------------------------------------------------
   Tomas Pospisek
   http://sourcepole.com - Linux & Open Source Solutions
-----------------------------------------------------------

7

Oops. I still have a little work to do to remove some of the company-
specific assumptions. Thanks for the heads-up. Please let me know if
you find anything else!

···

On May 2, 9:54 am, Stephan Mueller <d...@web.de> wrote:

one small note:

the created debian files contain references to what i think is one of
your personal projects: non-free/damballa (see debian/files for
examples). Maybe this can be customized when you debianize a project.

--
Stephen Touset <stephen@touset.org>

8

Not of rake tasks, but you could easily write a rake task which downloads
the file to somewhere predictable.

···

On Wed, 02 May 2007 18:46:15 +0900, Brian Candler wrote:

Right now I'm using a different approach: gem2spec takes a gem and turns it
into an RPM spec file. Unlike the native .deb packages you're generating
under Pallet, the components are installed under /usr/lib/ruby/gems/1.8 not
/usr/lib/ruby/1.8.

*snip*

With Pallet, you can tell it what files to bundle together to build a
package. But what about the initial fetch/untar/configure/build/install
steps? It would be very attractive to write these once in ruby (or rake),
but I don't think Pallet provides rules for this. Are there existing
libraries of rake tasks to do these steps?

--
Stephen Touset <stephen@touset.org>

9

I did send an email to the debian-ruby-extras mailing list, but
haven't heard anything back.

I'm about to start writing (time-willing) a Pallet::Gem2Deb module,
which should simply wrap around the Pallet::Deb builder. Basically,
feed it the task to build a gem, and it will handle the whole thing
for you -- including extracting relevant metadata from the Gem and
applying it automagically.

···

On May 4, 4:25 am, Tomas Pospisek's Mailing Lists <t...@sourcepole.ch> wrote:

On Wed, 2 May 2007, Stephen Touset wrote:
> I've been writing a multiplatform packaging tool for the company I work
> for, dubbed "pallet".
[...]
> Right now, gem and dpkg are the only formats supported, although adding
> .tar.gz should be trivial, and .rpm should be simple enough for anyone
> who's built an RPM.

Have you been talking to Debian's Ruby packagers? Mutating this into
something like "rake dpkg", that would automagically turn any ruby gem
into a Debian package would be brilliant :-o

10

* stephen@touset.org <stephen@touset.org> [04.05.2007]:

Oops. I still have a little work to do to remove some of the company-
specific assumptions. Thanks for the heads-up. Please let me know if
you find anything else!

i've using pallet now for 3 or 4 projects here and still like it. :slight_smile: I
am thinking about enhance pallet in a way that allows it to rebuild all
debian package control files on the fly, when the user whishes to do so.
This would allow to get packages with an up-to-date package version etc.
without handediting these files. What do you think about that?

Maybe a simple template engine could be used to handle build these files
on the fly and give the package maintainer full control on this.

Cheers,

Steph

11

Sure, but I was thinking of using it as a generic way to package third-party
software and use it to generate, say, .deb and .rpm

RPM is what I'm more familiar with. It has a 'spec' file which directs the
build. But it has sensible default macros for:
- untar source into a work directory
- apply patch(es)
- configure
- make
- make install

You can override any of these, but the defaults work in the majority of
cases.

Now, if I had to then package the same software as .deb, I'd have to write
(and first learn) Debian control files.

If I could specify the package building once - in Ruby - that would be
great.

Regards,

Brian.

···

On Sat, May 05, 2007 at 07:45:10AM +0900, Stephen Touset wrote:

> With Pallet, you can tell it what files to bundle together to build a
> package. But what about the initial fetch/untar/configure/build/install
> steps? It would be very attractive to write these once in ruby (or rake),
> but I don't think Pallet provides rules for this. Are there existing
> libraries of rake tasks to do these steps?

Not of rake tasks, but you could easily write a rake task which downloads
the file to somewhere predictable.

12

> With Pallet, you can tell it what files to bundle together to build a
> package. But what about the initial fetch/untar/configure/build/install
> steps? It would be very attractive to write these once in ruby (or rake),
> but I don't think Pallet provides rules for this. Are there existing
> libraries of rake tasks to do these steps?

Not of rake tasks, but you could easily write a rake task which downloads
the file to somewhere predictable.

Sure, but I was thinking of using it as a generic way to package third-party
software and use it to generate, say, .deb and .rpm

That's what I really aim this project to deal with. It'd be exceedingly
nice if upstream developers would take it upon themselves to package their
software in multiple formats, but barring that it should be easy for a
distro maintainer to repackage it.

Debian isn't so hot on packing gems inside of dpkgs and just having the
dpkg be a frontend for "gem install foo", but I don't personally mind, and
other Debian-derived distros (read: Ubuntu) might feel differently as well.

RPM is what I'm more familiar with. It has a 'spec' file which directs the
build. But it has sensible default macros for:
- untar source into a work directory
- apply patch(es)
- configure
- make
- make install

You can override any of these, but the defaults work in the majority of
cases.

I was going to try and tackle writing an RPM backend sometime the next
month, but would have to first learn the entire RPM building process. If
you'd like to write a Pallet::RPM class to generate and build a spec file,
I'd be more than happy to accept it.

Automatically downloading from remote sources, if done at all, would
probably come after I start supporting more package formats and write some
Pallet::Gem2(Deb|RPM|foo) classes to wrap Gems inside of OS-native packages.

···

On Sun, 06 May 2007 03:20:10 +0900, Brian Candler wrote:

On Sat, May 05, 2007 at 07:45:10AM +0900, Stephen Touset wrote:

--
Stephen Touset <stephen@touset.org>

13

Glad you like it!

The files are already templated, but I made the decision to leave the
debian/ directory on the filesystem after a build, for the case where a
package maintainer might want to generate the build structure once, then
hand-edit it and maintain it on his own from then on.

In retrospect, that should probably be done through a separate target
(rake packages:perm_deb, perhaps?). As it stands now, just run a "rake
clobber" or "rake clobber:deb" to get rid of the debian directory and
build again from scratch.

···

On Sun, 06 May 2007 01:38:06 +0900, Stephan Mueller wrote:

* stephen@touset.org <stephen@touset.org> [04.05.2007]:

Oops. I still have a little work to do to remove some of the company-
specific assumptions. Thanks for the heads-up. Please let me know if
you find anything else!

i've using pallet now for 3 or 4 projects here and still like it. :slight_smile: I
am thinking about enhance pallet in a way that allows it to rebuild all
debian package control files on the fly, when the user whishes to do so.
This would allow to get packages with an up-to-date package version etc.
without handediting these files. What do you think about that?

Maybe a simple template engine could be used to handle build these files
on the fly and give the package maintainer full control on this.

--
Stephen Touset <stephen@touset.org>

14

OK, so I'm not entirely clear how Pallet works then.

Does it generate "how to" build rules for the native packaging system - e.g.
spec files for RPM, whatever the equivalent is for Debian (control.tar.gz)?
And then you use the native packaging system to actually create the package?

Or does it configure, build and install the package itself, and then only
instruct the native packaging system to assemble those files into the
finished package?

Regards,

Brian.

···

On Mon, May 07, 2007 at 11:20:04PM +0900, Stephen Touset wrote:

> RPM is what I'm more familiar with. It has a 'spec' file which directs the
> build. But it has sensible default macros for:
> - untar source into a work directory
> - apply patch(es)
> - configure
> - make
> - make install
>
> You can override any of these, but the defaults work in the majority of
> cases.

I was going to try and tackle writing an RPM backend sometime the next
month, but would have to first learn the entire RPM building process. If
you'd like to write a Pallet::RPM class to generate and build a spec file,
I'd be more than happy to accept it.

Automatically downloading from remote sources, if done at all, would
probably come after I start supporting more package formats and write some
Pallet::Gem2(Deb|RPM|foo) classes to wrap Gems inside of OS-native packages.

15

OK, so I'm not entirely clear how Pallet works then.

Does it generate "how to" build rules for the native packaging system - e.g.
spec files for RPM, whatever the equivalent is for Debian (control.tar.gz)?
And then you use the native packaging system to actually create the
package?

Yes. And, for debian, that's a whole slew of files in a "debian/"
directory, plus a self-executing Makefile called "debian/rules". It's
kind of a PITA, actually. :slight_smile:

Or does it configure, build and install the package itself, and then
only instruct the native packaging system to assemble those files into
the finished package?

No, it only writes the package-specific configuration (config file, slew
of files in some directory, whatever) files, then instructs the package
system to do the whole thing itself.

Arguably, the latter approach would be cleaner from a software design
point of view, and likely where the project will head in the future. For
now, I wanted to get something released and working. Actual Ruby libraries
to completely generate .deb/.rpm/.ebuild/.foo would be very useful,
though. Sadly, I don't have infinite time, and I'm nowhere near as
productive as _why.

···

On Tue, 08 May 2007 01:51:45 +0900, Brian Candler wrote:

--
Stephen Touset <stephen@touset.org>

16

Uhm - as far as I know you only need like - two - of them: debian/control and debian/rules. The others are optional in various degrees (such as debian/copyright is reasonable if you want to distribute the package, isn't it?)
*t

···

On Tue, 8 May 2007, Stephen Touset wrote:

On Tue, 08 May 2007 01:51:45 +0900, Brian Candler wrote:

OK, so I'm not entirely clear how Pallet works then.

Does it generate "how to" build rules for the native packaging system - e.g.
spec files for RPM, whatever the equivalent is for Debian (control.tar.gz)?
And then you use the native packaging system to actually create the
package?

Yes. And, for debian, that's a whole slew of files in a "debian/"
directory, plus a self-executing Makefile called "debian/rules". It's
kind of a PITA, actually. :slight_smile:

--
-----------------------------------------------------------
   Tomas Pospisek
   http://sourcepole.com - Linux & Open Source Solutions
-----------------------------------------------------------

17

Right. Unfortunately, to do lots of things programmatically, you need
plenty more than that in conjunction with debhelper.

  debian/changelog => changelog
  debian/compat => debhelper compatibility version
  debian/config => debconf configuration
  debian/control => package information
  debian/copyright => copyright information
  debian/dirs => directories that should be created on the target
  debian/docs => location of documentation
  debian/files => files to be installed in the target
  debian/rules => build script

Plus a few others, depending on whether or not a few options are used.

···

On Tue, 08 May 2007 14:40:54 +0900, Tomas Pospisek's Mailing Lists wrote:

On Tue, 8 May 2007, Stephen Touset wrote:

On Tue, 08 May 2007 01:51:45 +0900, Brian Candler wrote:

OK, so I'm not entirely clear how Pallet works then.

Does it generate "how to" build rules for the native packaging system - e.g.
spec files for RPM, whatever the equivalent is for Debian (control.tar.gz)?
And then you use the native packaging system to actually create the
package?

Yes. And, for debian, that's a whole slew of files in a "debian/"
directory, plus a self-executing Makefile called "debian/rules". It's
kind of a PITA, actually. :slight_smile:

Uhm - as far as I know you only need like - two - of them: debian/control
and debian/rules. The others are optional in various degrees (such as
debian/copyright is reasonable if you want to distribute the package,
isn't it?)

--
Stephen Touset <stephen@touset.org>

--
Stephen Touset <stephen@touset.org>

18

Take those easy. debhelper has some 61 helper programms of which most are configurable or use arguments *only*. So it's also your choice which way to go and also *if* to use those at all. Neither of those, except for debian/control and debian/rules are mandatory. Actually you can even make a package without debian/rules, it will be installable if you build it right however Debian itself will not accept it since it will not be autobuildable. Please correct me if I'm wrong.
*t

···

On Wed, 9 May 2007, Stephen Touset wrote:

On Tue, 08 May 2007 14:40:54 +0900, Tomas Pospisek's Mailing Lists wrote:

On Tue, 8 May 2007, Stephen Touset wrote:

On Tue, 08 May 2007 01:51:45 +0900, Brian Candler wrote:

OK, so I'm not entirely clear how Pallet works then.

Does it generate "how to" build rules for the native packaging system - e.g.
spec files for RPM, whatever the equivalent is for Debian (control.tar.gz)?
And then you use the native packaging system to actually create the
package?

Yes. And, for debian, that's a whole slew of files in a "debian/"
directory, plus a self-executing Makefile called "debian/rules". It's
kind of a PITA, actually. :slight_smile:

Uhm - as far as I know you only need like - two - of them: debian/control
and debian/rules. The others are optional in various degrees (such as
debian/copyright is reasonable if you want to distribute the package,
isn't it?)

Right. Unfortunately, to do lots of things programmatically, you need
plenty more than that in conjunction with debhelper.

debian/changelog => changelog
debian/compat => debhelper compatibility version
debian/config => debconf configuration
debian/control => package information
debian/copyright => copyright information
debian/dirs => directories that should be created on the target
debian/docs => location of documentation
debian/files => files to be installed in the target
debian/rules => build script

Plus a few others, depending on whether or not a few options are used.

--
-----------------------------------------------------------
   Tomas Pospisek
   http://sourcepole.com - Linux & Open Source Solutions
-----------------------------------------------------------