Porting to Modular X HOWTO

1.0 2006-01-03 Background

Introduction

You may be wondering, why in the world did one nice, easy xorg-x11 package turn into almost 300 separate ones? And you'd certainly be justified in this. =) It's not something Gentoo is doing independently of upstream X.Org; they're splitting up all the packages into separate releases, and we're just following along.

You can read up on the details at the Migrating to Modular X HOWTO page.

Dependency checking

We want to enumerate the dependencies as finely as possible, so people don't to have unused cruft lying all over their systems for things they don't even use, such as XPrint. So you want to depend directly upon the library and header packages you need rather than any sort of virtual.

Also, we can't guarantee that people will still have subpackages installed just because they have a metabuild installed, and eliminating that possibility of breakage will save us all a lot of time that would be spent marking bugs INVALID.

If I find any packages depending upon any of the metabuilds there will be, I won't hesitate to hassle and haze the maintainer unto eternity.

First step is to either install modular X or find somebody who has it installed. This can be done in a chroot -- there's no need to actually run X, you just need to have its files available for dependency checking.

$ wget http://dev.gentoo.org/~dberkholz/scripts/linking_libs \
	http://dev.gentoo.org/~dberkholz/scripts/included_headers \
	http://dev.gentoo.org/~betelgeuse/scripts/deputils/checkdeps.rb \
	http://dev.gentoo.org/~betelgeuse/scripts/deputils/pkgutil.rb

Do not use gentoolkit-0.2.1_pre9, it produces invalid output. See https://bugs.gentoo.org/show_bug.cgi?id=111501.

The first script, linking_libs, checks a compilation log of your package for all the libraries it links with, and prints out the packages those libraries belong to. To get a compilation log, you may set PORT_LOGDIR in /etc/make.conf or use output redirection.

$ ls /var/log/portage/*gdm* -l
-rw-r--r-- 1 root portage 556551 2006-01-09 11:41 /var/log/portage/8430-gdm-2.8.0.7.log
-rw-r--r-- 1 root portage    855 2006-01-09 11:42 /var/log/portage/8431-gdm-2.8.0.7.log
$ linking_libs /var/log/portage/8430-gdm-2.8.0.7.log

The second script, included_headers, scans the unpacked source of your package for lines starting with #include, and grabs out include files contained in <>. As of 9 Sept. 2005, it does work for ""-style includes as well as <>.

The third script, checkdeps.rb, scans the installed files of a package using scanelf from pax-utils. This is useful for binary packages or packages that otherwise hide the compilation output. It's a Ruby script, so it depends on dev-lang/ruby as well as app-misc/pax-utils. The fourth script, pkgutil.rb, is a dependency of checkdeps.rb.

Running the first two scripts should get you a good list of packages for both RDEPEND (for libraries) and DEPEND (headers and libraries). If a library shows up in the RDEPEND list that doesn't show up in the DEPEND list, be suspicious; it may be a "false dependency," (a library that's linked against for no reason). One known example of a true dependency like this is libXt; many packages check for libXt headers to check for X.

Occasionally the relative header search in included_headers will find the wrong header, because there are multiple ones named the same, and thus will return an incorrect package. Usually this is quite obvious, such as when Windows headers make it think app-emulation/wine is a dependency.

If you specify the -d option, occasionally you'll run into a library or header that comes up "Not found!" This may mean you've uninstalled a header the package needs since compiling it, or it's an optional header you aren't using. In the library case, it could mean you uninstalled the lib or perhaps it was only an internally-built static library that was never installed.

It's worth spending the time to figure out whether libraries or headers that aren't found need to be added to the dependency list, particularly in the header case since you don't need the headers installed to run the scan.

In some cases, packages require an actual X server of some sort. But if they don't actually require it during the installation, I encourage you to not put it into the RDEPEND. This breaks headless X installations, where the programs actually run on another machine so only need local libraries and headers.

There are already a number of X servers in the tree, so if you do need an X server, please include them all. Modular X's servers are in xorg-server; there's a DirectFB server at xdirectfb; kdrive provides tiny X servers; and of course old <xorg-x11-7. Do specify the version restriction on xorg-x11, to ensure an X server instead of a metabuild. In the near future, I anticipate kdrive moving to xserver. If you do require an X server, please contact a member of the X team. We may create a virtual for it if a sufficient number of packages require it.

Dependency structure

On to actually adding the dependencies -- you'll want a structure like this:

RDEPEND="|| ( ( x11-libs/libXfoo
			x11-libs/libXbar
			blah? ( x11-libs/libXblah )
			whatever else shows up in the library run
		)
		virtual/x11
	)

DEPEND="${RDEPEND}
	|| ( ( x11-proto/fooproto
			blah? ( x11-proto/blahproto )
			whatever else shows up in both script runs
		)
		virtual/x11
	)

Some of the results produced by the scripts will be redundant. If the RDEPEND of one library includes another, you don't need to put both into the dependencies. If the DEPEND of a library includes a proto, you do need it in the DEPEND list of the package. Likely candidates for libraries that pull in lots of other libraries are libXaw, libXmu, libXpm, libXcursor, libXt. Just do emerge -ep $library | grep lib[SIX]. Also keep in mind that other packages like gtk+ have been ported to modular dependencies, so they can pull in needed libraries as well. Two separate USE flags both will bring in X, but one is not dependent on the other. In this case, you will need to either duplicate the X dependency section or define it elsewhere and include it as ${X_DEPEND}. If defined elsewhere, remember to also include the parts specific to each USE flag.

The goal here is defaulting to the new, modular X, but allowing people to also fulfill the dependencies with the old, monolithic xorg-x11 package. We're dropping virtual/x11 entirely to encourage the enumeration of proper dependencies.

For the initial run through the tree, the porting task force only fixes the newest ebuild available to ~arch users, and anything newer (KEYWORDS=-* or package.mask). Individual package maintainers may choose to do this and allow the unported ebuilds to gradually fade from the tree, but they will probably want to port all their ebuilds at once. Repoman will soon die on any ebuild with a hard dependency on virtual/x11.

This should allow ~arch users to use the modular X by default while sending non-~arch users to virtual/x11.

Dealing with USE flags

Many people have the xinerama USE flag deactivated. But if, when you're running included_headers, x11-proto/xineramaproto shows up as a dependency, then add it to DEPEND behind the xinerama USE, and add x11-libs/libXinerama to RDEPEND behind the xinerama USE.

Caleb raised a good point, and that is, how do you deal with all the USE flags on packages that have a ton of optional X library dependencies? In many cases, it will make sense to always force on or off flags. Also, you can make this easier by grouping flags. Make sure you're naming the flags by their functions and not by the libraries or packages they use.

Getting your fixes in the tree

If you're a developer, commit them. If not, file a new bug, assigned to the package maintainers (in metadata.xml). Have it block bug #112675. Attach a patch to fix the dependencies.