Rbconfig.rb and DESTDIR confusion while cross-compiling Ruby gems

Hello,

I'm trying to cross-compile Nokogiri from one Linux platform to
another. After a lot of digging around (especially since I'm new to
Ruby), I went with the following approach:

1) Cross-compile a version of Ruby for the target platform. This
cross-compiled Ruby will contain a file called rbconfig.rb that
contains build information (compiler flags used, paths to standard
directories, etc.). This build information is used when building
native extensions.

2) Configure (produce a Makefile for) Nokogiri's native extension by
running the following command in Nokogiri's root (with irrelevant
flags left out):

    ruby -Cext/nokogiri -I<dir_of_rbconfig.rb> extconf.rb ...

The 'ruby' used here is the native one, not the cross-compiled one,
but rbconfig.rb comes from the cross-compiled 'ruby'. The native and
cross-compiled rubies are the same version. extconf.rb uses mkmf to
build the Makefile, and mkmf in turn uses build information from
rbconfig.rb.

3) Build the native extension as follows:

    make -Cext/nokogiri

This approach seems to almost work, but runs into trouble for one
reason: The cross-compiled Ruby is installed into a custom sysroot
directory (with e.g. the main ruby executable at
/home/ulf/target-sysroot/usr/bin/ruby), while the paths in rbconfig.rb
seem to assume installation into / (where the ruby executable would be
at /usr/bin/ruby).

The variable DESTDIR (not to be confused with $(DESTDIR), which comes
from the Makefile) in rbconfig.rb looks promising, but is not
sufficient as it is not prepended to e.g. CONFIG["libdir"].

The generated Makefile also has a DESTDIR. Just setting DESTDIR in the
Makefile won't work though as mkmf needs correct paths when looking
for headers and the like during configuration.

After the above mind dump, my questions are as follows:

- Is rbconfig.rb's DESTDIR meant to solve problems like this? What
exactly is its purpose?

- Why is there an additional DESTDIR in the Makefile? Is its purpose different?

Thanks,
Ulf

Have you tried using the tooling that nokogiri uses?

It tries to make cross-compiling clean and easy.

···

On Nov 3, 2015, at 11:09, Ulf Magnusson <ulfalizer@gmail.com> wrote:

Hello,

I'm trying to cross-compile Nokogiri from one Linux platform to
another. After a lot of digging around (especially since I'm new to
Ruby), I went with the following approach:

1) Cross-compile a version of Ruby for the target platform. This
cross-compiled Ruby will contain a file called rbconfig.rb that
contains build information (compiler flags used, paths to standard
directories, etc.). This build information is used when building
native extensions.

2) Configure (produce a Makefile for) Nokogiri's native extension by
running the following command in Nokogiri's root (with irrelevant
flags left out):

   ruby -Cext/nokogiri -I<dir_of_rbconfig.rb> extconf.rb ...

The 'ruby' used here is the native one, not the cross-compiled one,
but rbconfig.rb comes from the cross-compiled 'ruby'. The native and
cross-compiled rubies are the same version. extconf.rb uses mkmf to
build the Makefile, and mkmf in turn uses build information from
rbconfig.rb.

3) Build the native extension as follows:

   make -Cext/nokogiri

Yup, I've already looked at rake-compiler, and it'd obviously be the
cleanest solution if it could be made to work. It seems to be heavily
geared towards cross-compiling for Windows though (do a search for
"mingw" in the readme and sources), and making it use an existing
cross-compiler for another system is tricky. rake-compiler seems to
use the approach described above too (perhaps with some additional
tweaks).

I just discovered that the path and DESTDIR trouble I've been having
is due to using Ruby 1.9.3 (which unfortunately seems to be a
requirement in this case). Later Ruby versions change some paths in
rbconfig.rb so that they're based on $(prefix) rather than $(DESTDIR),
which makes mkmf happy and makes things compile for me.

Just for posterity, here's an ugly patch for Ruby 1.9.3 that overrides
some rbconfig.rb paths (by changing mkconfig.rb, which generates
rbconfig.rb) to be like in later Ruby versions. This is enough to get
native extensions cross-compiling for me (and hopefully working,
though I haven't had time to verify yet):

···

On Fri, Nov 6, 2015 at 12:15 AM, Ryan Davis <ryand-ruby@zenspider.com> wrote:

On Nov 3, 2015, at 11:09, Ulf Magnusson <ulfalizer@gmail.com> wrote:

Hello,

I'm trying to cross-compile Nokogiri from one Linux platform to
another. After a lot of digging around (especially since I'm new to
Ruby), I went with the following approach:

1) Cross-compile a version of Ruby for the target platform. This
cross-compiled Ruby will contain a file called rbconfig.rb that
contains build information (compiler flags used, paths to standard
directories, etc.). This build information is used when building
native extensions.

2) Configure (produce a Makefile for) Nokogiri's native extension by
running the following command in Nokogiri's root (with irrelevant
flags left out):

   ruby -Cext/nokogiri -I<dir_of_rbconfig.rb> extconf.rb ...

The 'ruby' used here is the native one, not the cross-compiled one,
but rbconfig.rb comes from the cross-compiled 'ruby'. The native and
cross-compiled rubies are the same version. extconf.rb uses mkmf to
build the Makefile, and mkmf in turn uses build information from
rbconfig.rb.

3) Build the native extension as follows:

   make -Cext/nokogiri

Have you tried using the tooling that nokogiri uses?

GitHub - rake-compiler/rake-compiler: Provide a standard and simplified way to build and package Ruby C and Java extensions using Rake as glue.

It tries to make cross-compiling clean and easy.

---
tool/mkconfig.rb | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/tool/mkconfig.rb b/tool/mkconfig.rb
index aa54512..d563697 100755
--- a/tool/mkconfig.rb
+++ b/tool/mkconfig.rb
@@ -218,6 +218,21 @@ end

print(*v_fast)
print(*v_others)
+
+print <<EOS
+ # Start of manual overrides
+ CONFIG["man_dir"] = "$(prefix)/share/man"
+ CONFIG["libdir"] = "$(exec_prefix)/lib"
+ CONFIG["infodir"] = "$(prefix)/share/info"
+ CONFIG["includedir"] = "$(prefix)/include"
+ CONFIG["sharedstatedir"] = "$(prefix)/com"
+ CONFIG["libexecdir"] = "$(prefix)/lib/ruby"
+ CONFIG["sbindir"] = "$(exec_prefix)/sbin"
+ CONFIG["bindir"] = "$(exec_prefix)/bin"
+ CONFIG["exec_prefix"] = "$(prefix)"
+ # End of manual overrides
+EOS
+
print <<EOS
   CONFIG["rubylibdir"] = "$(rubylibprefix)#{path_version}"
   CONFIG["archdir"] = "$(rubylibdir)/$(arch)"
--
2.1.4

Just for completeness, I also had to fix a path issue with
OpenEmbedded's Ruby:
http://lists.openembedded.org/pipermail/openembedded-devel/2015-November/104350.html

Cheers,
Ulf

Hello,

I'm trying to cross-compile Nokogiri from one Linux platform to
another. After a lot of digging around (especially since I'm new to
Ruby), I went with the following approach:

1) Cross-compile a version of Ruby for the target platform. This
cross-compiled Ruby will contain a file called rbconfig.rb that
contains build information (compiler flags used, paths to standard
directories, etc.). This build information is used when building
native extensions.

2) Configure (produce a Makefile for) Nokogiri's native extension by
running the following command in Nokogiri's root (with irrelevant
flags left out):

   ruby -Cext/nokogiri -I<dir_of_rbconfig.rb> extconf.rb ...

The 'ruby' used here is the native one, not the cross-compiled one,
but rbconfig.rb comes from the cross-compiled 'ruby'. The native and
cross-compiled rubies are the same version. extconf.rb uses mkmf to
build the Makefile, and mkmf in turn uses build information from
rbconfig.rb.

3) Build the native extension as follows:

   make -Cext/nokogiri

Have you tried using the tooling that nokogiri uses?

GitHub - rake-compiler/rake-compiler: Provide a standard and simplified way to build and package Ruby C and Java extensions using Rake as glue.

It tries to make cross-compiling clean and easy.

Yup, I've already looked at rake-compiler, and it'd obviously be the
cleanest solution if it could be made to work. It seems to be heavily
geared towards cross-compiling for Windows though (do a search for
"mingw" in the readme and sources), and making it use an existing
cross-compiler for another system is tricky. rake-compiler seems to
use the approach described above too (perhaps with some additional
tweaks).

I just discovered that the path and DESTDIR trouble I've been having
is due to using Ruby 1.9.3 (which unfortunately seems to be a
requirement in this case). Later Ruby versions change some paths in
rbconfig.rb so that they're based on $(prefix) rather than $(DESTDIR),
which makes mkmf happy and makes things compile for me.

Just for posterity, here's an ugly patch for Ruby 1.9.3 that overrides
some rbconfig.rb paths (by changing mkconfig.rb, which generates
rbconfig.rb) to be like in later Ruby versions. This is enough to get
native extensions cross-compiling for me (and hopefully working,
though I haven't had time to verify yet):

Update: The cross-compiled native extensions seem to run fine.

Cheers,
Ulf

···

On Fri, Nov 6, 2015 at 10:26 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:

On Fri, Nov 6, 2015 at 12:15 AM, Ryan Davis <ryand-ruby@zenspider.com> wrote:

On Nov 3, 2015, at 11:09, Ulf Magnusson <ulfalizer@gmail.com> wrote:

---
tool/mkconfig.rb | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/tool/mkconfig.rb b/tool/mkconfig.rb
index aa54512..d563697 100755
--- a/tool/mkconfig.rb
+++ b/tool/mkconfig.rb
@@ -218,6 +218,21 @@ end

print(*v_fast)
print(*v_others)
+
+print <<EOS
+ # Start of manual overrides
+ CONFIG["man_dir"] = "$(prefix)/share/man"
+ CONFIG["libdir"] = "$(exec_prefix)/lib"
+ CONFIG["infodir"] = "$(prefix)/share/info"
+ CONFIG["includedir"] = "$(prefix)/include"
+ CONFIG["sharedstatedir"] = "$(prefix)/com"
+ CONFIG["libexecdir"] = "$(prefix)/lib/ruby"
+ CONFIG["sbindir"] = "$(exec_prefix)/sbin"
+ CONFIG["bindir"] = "$(exec_prefix)/bin"
+ CONFIG["exec_prefix"] = "$(prefix)"
+ # End of manual overrides
+EOS
+
print <<EOS
   CONFIG["rubylibdir"] = "$(rubylibprefix)#{path_version}"
   CONFIG["archdir"] = "$(rubylibdir)/$(arch)"
--
2.1.4

Just for completeness, I also had to fix a path issue with
OpenEmbedded's Ruby:
http://lists.openembedded.org/pipermail/openembedded-devel/2015-November/104350.html

Cheers,
Ulf