Autopackage GNOME Launch Box Tutorial

Binary portability

Binary portability is about making sure your compiled programs work on many computers. There are some simple things you can do to ensure the installation experience is as close to "Just Works" as possible no matter what distribution your users run.

Let's take a look at how many libraries Launch Box needs.

[mike@redwood gnome-launch-box]$ ldd src/gnome-launch-box | wc -l
	75

75?! Oh my god that's a lot of libraries. That can't possibly be right, such a simple program shouldn't have 75 dependencies. Let's dig a little deeper. Looking through the list shows many libraries that the application clearly does not use: for instance libXrandR is linked into the program despite the fact that Launch Box does not modify your screen resolution. What's going on here?

Well, there are actually two issues. The first is that the ldd program shows you every library loaded into a process image, not every library that the program directly requests. We're seeing not just the dependencies of the program but its dependencies dependencies and so on. We can see only what Launch Box itself directly needs with the following command:

[mike@redwood gnome-launch-box]$ objdump -x src/gnome-launch-box | grep NEEDED

This is such a handy pattern that I usually make it a shell function. This shows you the ELF DT_NEEDED entries for the binary: these are the libraries that it directly requests and must be available on every system your binary runs on.

[mike@redwood gnome-launch-box]$ objdump -x src/gnome-launch-box | grep NEEDED
  NEEDED      libgnome-menu.so.0
  NEEDED      libgnome-desktop-2.so.2
  NEEDED      libgnomeui-2.so.0
  NEEDED      libSM.so.6
  NEEDED      libICE.so.6
  NEEDED      libstartup-notification-1.so.0
  NEEDED      libbonoboui-2.so.0
  NEEDED      libgnomecanvas-2.so.0
  NEEDED      libart_lgpl_2.so.2
  NEEDED      libpangoft2-1.0.so.0
  NEEDED      libgtk-x11-2.0.so.0
  NEEDED      libgdk-x11-2.0.so.0
  NEEDED      libatk-1.0.so.0
  NEEDED      libgdk_pixbuf-2.0.so.0
  NEEDED      libpangoxft-1.0.so.0
  NEEDED      libpangox-1.0.so.0
  NEEDED      libpango-1.0.so.0
  NEEDED      libgobject-2.0.so.0
  NEEDED      libebook-1.2.so.3
  NEEDED      libedataserver-1.2.so.4
  NEEDED      libgnome-2.so.0
  NEEDED      libpopt.so.0
  NEEDED      libxml2.so.2
  NEEDED      libpthread.so.0
  NEEDED      libz.so.1
  NEEDED      libgnomevfs-2.so.0
  NEEDED      libbonobo-2.so.0
  NEEDED      libgconf-2.so.4
  NEEDED      libbonobo-activation.so.4
  NEEDED      libORBit-2.so.0
  NEEDED      libm.so.6
  NEEDED      libgmodule-2.0.so.0
  NEEDED      libdl.so.2
  NEEDED      libgthread-2.0.so.0
  NEEDED      libglib-2.0.so.0
  NEEDED      libc.so.6

Ouch - 36 libraries is a lot better than 75, but it's an awful lot of dependencies. And they still don't look right: Launch Box uses its own IPC system from libbacon, not Bonobo. So why is libbonobo-activation being linked in? Is it even being used? With modern glibc linkers, you can find out whether any libraries are in the DT_NEEDED list but aren't actually used:

[mike@redwood gnome-launch-box]$ ldd -u -r src/gnome-launch-box
Unused direct dependencies:

        /usr/X11R6/lib/libSM.so.6
        /usr/X11R6/lib/libICE.so.6
        /usr/lib/libstartup-notification-1.so.0
        /usr/lib/libbonoboui-2.so.0
        /usr/lib/libgnomecanvas-2.so.0
        /usr/lib/libart_lgpl_2.so.2
        /usr/lib/libpangoft2-1.0.so.0
        /usr/lib/libatk-1.0.so.0
        /usr/lib/libpangoxft-1.0.so.0
        /usr/lib/libpangox-1.0.so.0
        /usr/lib/libpango-1.0.so.0
        /usr/lib/libedataserver-1.2.so.4
        /usr/lib/libgnome-2.so.0
        /usr/lib/libpopt.so.0
        /usr/lib/libz.so.1
        /usr/lib/libbonobo-2.so.0
        /usr/lib/libgconf-2.so.4
        /usr/lib/libbonobo-activation.so.4
        /usr/lib/libORBit-2.so.0
        /lib/libm.so.6
        /usr/lib/libgmodule-2.0.so.0
        /lib/libdl.so.2
        /usr/lib/libgthread-2.0.so.0

Interesting! 23 of those 36 libraries aren't actually used at all! They're being added by pkg-config files and similar scripts that list not only the library in the LDFLAGS, but also the libraries dependencies. This is necessary on some non-GNU linkers which can't follow library dependencies down the tree, but on Linux it's just a nuisance.

We can immediately increase the number of systems our binary runs on by eliminating these so-called bogus dependencies. Why should we risk the user seeing an error message about a library they have, yet don't need?

We can use apbuild to transparently fix this problem. In fact, as we shall see in a moment, apbuild can automatically fix many silly binary portability problems like this one. Apbuild is an easy to use GCC wrapper script that modifies the parameters passed to GCC in such a way that it increases the number of Linux systems your binaries can run on - for free! It solves many other problems, such as glibc symbol versions, non-versioned headers, issues with GNU exception handling. For C++ binaries it can even double compile the binary with two different compilers which autopackage can then use to ensure the user will always get a binary of the right ABI.

It's recommended that you use apbuild with the latest versions of the GNU toolchain. In particular, a recent version of binutils.

[mike@redwood gnome-launch-box]$ make CC=apgcc CXX=apg++
        .....
[mike@redwood gnome-launch-box]$ objdump -x src/gnome-launch-box | grep NEEDED
          NEEDED      libpthread.so.0
          NEEDED      libgnome-menu.so.0
          NEEDED      libgnome-desktop-2.so.2
          NEEDED      libgnomeui-2.so.0
          NEEDED      libgtk-x11-2.0.so.0
          NEEDED      libgdk-x11-2.0.so.0
          NEEDED      libgdk_pixbuf-2.0.so.0
          NEEDED      libgobject-2.0.so.0
          NEEDED      libebook-1.2.so.3
          NEEDED      libxml2.so.2
          NEEDED      libgnomevfs-2.so.0
          NEEDED      libglib-2.0.so.0
          NEEDED      libc.so.6

That's a lot better: now we can see what libraries our program really needs.
Now it's time for a dependency audit.

Dealing with dependencies

Let's go through each library in turn and figure out how many users are going to have them.

libpthread.so.0: This is a part of glibc, everyone will have it.
libgnome-menu.so.0: This library comes with GNOME 2.10, but it's unstable and might break backwards compatibility at any time: even within point releases or security updates!
libgnome-desktop-2.so.2: This is a part of GNOME 2.10 (the desktop platform), and it's not a part of the GNOME developer platform. Again, it could break backwards compatibility at any time. We can't rely on it being there.
libgnomeui-2.so.0: According to Launch Boxes configure.ac, any version of this will do. It's also a part of every GNOME release since version 2.0, so it's OK to link against this dynamically. It's pretty likely to be on the users system, or at least, very easy to get.
libgtk/libgdk/libgdk_pixbuf/libgobject/libglib: These are not seperate dependencies, but one single dependency: GTK+. It is a widespread library and almost everybody has it. Still, upgrading GTK+ for end users is very difficult, it really means a distribution upgrade so users (and even developers!) may well not have the latest version. It's worth trying to make your app run against older GTK+ versions; that way you can get more users (and as such, more developers).
libebook-1.2.so.3: This is a component of Evolution Data Server. You can install e-d-s without having Evolution proper installed, but most people are unlikely to have it unless they also use Evo. Launch Box is so useful as a quick way to run apps we really don't want to make this a hard dependency. It should work properly if Evo isn't installed, but automatically get the extra features if libebook is there.
libxml2.so.2: Unfortunately, there's no check in configure.ac for this library so we have no idea what version is needed. This sort of thing is why autopackages should be written by the same developers that wrote the original software. Fortunately, libxml2 is very widespread, it's a safe bet that users have this (assuming no usage of bleeding edge features).
libgnomevfs-2.so.0: The VFS comes with every copy of GNOME, so it's likely that it'll be there. It's also quite stable. On the other hand, Launch Box requires the GNOME 2.10 version of it, which is very new. Not many users will have this yet. Ah well, c'est la vie.
libc.so.6: The GNU C library is a very special library. You can't tell what version of it you need by looking purely at the DT_NEEDED entries, nor by looking at the source. Sometimes you will be automatically given dependencies on very new versions of the C library, even if it's for no reason. Apbuild fixes this automatically.

relaytool

Let's make Launch Box work even if libebook-1.2.so is missing. The traditional way to do this is to use dlopen() and dlsym(), but this interface is rather cumbersome. Because weak linking is so vital to reducing dependencies and making software easier to install, the autopackage project provides a program which makes using dlopen much easier.

Behind the scenes, relaytool generates and compiles a C file that can stand in for the symbols you use in the real library. When your program starts, an ELF constructor function dlopens the library and sets things up so that when you use a symbol from that library it's routed either to a relaytool generated stub (if the library is missing) or the real thing. The actual implementation involves a bit of assembly and I won't go into it here.

Firstly we need to make it optional at compile time. This isn't too hard, we just add an automake conditional to switch off the two lb-module-evolution files, surround the init code in the module manager with a C preprocessor conditional, and test for it separately in the configure script:

PKG_CHECK_MODULES(EVO, libebook-1.2, have_ebook=true, have_ebook=false)
AM_CONDITIONAL(EBOOK, test x$have_ebook = xtrue)
if test x$have_ebook = xtrue; then
   AC_DEFINE(HAVE_EBOOK, 1, [Have libebook-1.2.so])
fi

... then we can separate the source files containing the Evo plugin out in Makefile.am:

if EBOOK
gnome_launch_box_SOURCES += lb-module-evolution.c lb-module-evolution.h
endif

... and make sure the plugin won't be loaded if it's missing:

#ifdef HAVE_EBOOK
	manager->modules = g_list_append (manager->modules,
					  g_object_new (LB_TYPE_MODULE_EVOLUTION,
							NULL));
#endif

So far, this is nothing special. Now if Evolution isn't available at compile time, it will still build. But compile time isn't good enough. Users want binaries just as flexible as source, and that's what we'll give them.

Relaytool is designed to work in conjunction with pkg-config. It provides a simple M4 macro you can use in your configure script. Once apbuild is installed, you should be able to add this after the PKG_CHECK_MODULES call returns succesfully:

RELAYTOOL("ebook-1.2", LB_LIBS, LB_CFLAGS, )

The final parameter is some code to run if weak linking is possible. At the moment, that simply means, if relaytool is installed. If it isn't installed then you will get a hard link as usual. The two middle parameters are the names of shell variables containing the linker and compiler commands, respectively. The RELAYTOOL macro will modify these as necessary.

Now, in the code, you need to find where the library is being used and protect any calls into the library with runtime conditionals. We might need to restructure the program so that the codepaths which use the library will always be compiled in. In this instance, it's easy. We need to put the RELAYTOOL_EBOOK_1_2 macro into the source, protected by an ifdef so if the library is missing completely it's ignored. The name of the macro is generated from the library name. At compile time the macro will be taken from the CFLAGS and expand to provide libebook_1_2_is_present and libebook_1_2_symbol_is_present() which you can use to avoid calling the stubs accidentally.:

#ifdef HAVE_EBOOK
/* the following line is a macro that provides the relaytool _is_present symbols */
RELAYTOOL_EBOOK_1_2
#endif

....

#ifdef HAVE_EBOOK
	if (libebook_1_2_is_present)
		manager->modules = g_list_append (manager->modules,
						  g_object_new (LB_TYPE_MODULE_EVOLUTION,
								NULL));
#endif

That's it! That's all we have to do. There's no need to modify code beyond adding relaytool to the build system and adding a runtime check as shown. If you call a function in a library which isn't present, then your program will abort with an error identifying which function was called.

Relaytool works for variables too: if the library is missing the value will always be -1. It is also capable of weak linking C++ libraries. Finally, you can "relay" a subset of functionality in a library you are hard linking to. This can be used to easily use new APIs that may or may not be available in the users copy of a library (eg, the new GTK+ file chooser).

Summary

And now we're finished! Let's take a look at all the things we've done:

Dependency audit: we have determined what the program really needs, and specified them as either required or optional dependencies.
Soft linking: by using relaytool, we have turned libebook into an optional runtime depedency. This makes life much easier for end users!

A final note: glibc dependency

One thing we didn't handle in this tutorial is glibc dependency. Have you ever seen errors along the lines of "undefined symbol GLIBC_2.3 in /lib/libc.so" when you try to run a program? This problem (and the fix) are explained by the APbuild web page. Autoconf/automake projects that use the prepareBuild function (this will be explained in step 4) will automatically be compiled using APBuild. But if your project does not use autoconf/automake (and thus doesn't use prepareBuild) then don't forget to compile your source code using APBuild!