GLEP:33
Title:Eclass Restructure/Redesign
Version:1.1
Last-Modified:2005/02/16 21:30:44
Author:John Mylchreest <johnm at gentoo.org>, Brian Harring <ferringb at gentoo.org>
Status:Draft
Type:Standards Track
Content-Type:text/x-rst
Created:29-Jan-2005
Post-History:29-Jan-2005

Contents

Abstract

For any design, the transition from theoretical to applied exposes inadequacies in the original design. This document is intended to document, and propose a revision of the current eclass setup to address current eclass inadequacies.

This document proposes several thing- the creation of ebuild libraries, 'elibs', a narrowing of the focus of eclasses, a move of eclasses w/in the tree, the addition of changelogs, and a way to allow for simple eclass gpg signing. In general, a large scale restructuring of what eclasses are and how they're implemented. Essentially version two of the eclass setup.

Terminology

From this point on, the proposed eclass setup will be called 'new eclasses', the existing crop (as of this writing) will be referenced as 'old eclasses'. The destinction is elaborated on within this document.

Motivation and Rationale

Eclasses within the tree currently are a bit of a mess- they're forced to maintain backwards compatability w/ all previous functionality. In effect, their api is constant, and can only be added to- never changing the existing functionality. This obviously is quite limiting, and leads to cruft accrueing in eclasses as a eclasses design is refined. This needs to be dealt with prior to eclass code reaching a critical mass where they become unmanagable/fragile (recent pushes for eclass versioning could be interpretted as proof of this).

Beyond that, eclasses were originally intended as a method to allow for ebuilds to use a pre-existing block of code, rather then having to duplicate the code in each ebuild. This is a good thing, but there are ill effects that result from the current design. Eclasses inherit other eclasses to get a single function- in doing so, modifying the the exported 'template' (default src_compile, default src_unpack, various vars, etc). All the eclass designer was after was reusing a function, not making their eclass sensitive to changes in the template of the eclass it's inheriting. The eclass designer -should- be aware of changes in the function they're using, but shouldn't have to worry about their default src_* and pkg_* functions being overwritten, let alone the env changes.

Addressing up front why a collection of eclass refinements are being rolled into a single set of changes, parts of this proposal -could- be split into multiple phases. Why do it though? It's simpler for developers to know that the first eclass specification was this, and that the second specification is that, rather then requiring them to be aware of what phase of eclass changes is in progress.

By rolling all changes into one large change, a line is intentionally drawn in the sand. Old eclasses allowed for this, behaved this way. New eclasses allow for that, and behave this way. This should reduce misconceptions about what is allowed/possible with eclasses, thus reducing bugs that result from said misconceptions.

Specification.

The various parts of this proposal are broken down into a set of changes and elaborations on why a proposed change is preferable. It's advisable to the reader that this be read serially, rather then jumping around.

Ebuild Libraries (elibs for short)

As briefly touched upon in Motivation and Rationale, the original eclass design allowed for the eclass to modify the metadata of an ebuild, metadata being the DEPENDS, RDEPENDS, SRC_URI, IUSE, etc, vars that are required to be constant, and used by portage for dep resolution, fetching, etc. Using the earlier example, if you're after a single function from an eclass (say epatch from eutils), you -don't- want the metadata modifications the eclass you're inheriting might do. You want to treat the eclass you're pulling from as a library, pure and simple.

A new directory named elib should be added to the top level of the tree to serve as a repository of ebuild function libraries. Rather then relying on using the source command, an 'elib' function should be added to portage to import that libraries functionality. The reason for the indirection via the function is mostly related to portage internals, but it does serve as an abstraction such that (for example) zsh compatability hacks could be hidden in the elib function.

Elib's will be collections of bash functions- they're not allowed to do anything in the global scope aside from function definition, and any -minimal- initialization of the library that is absolutely needed. Additionally, they cannot modify any ebuild functions- src_compile, src_unpack fex. Since they are required to not modify the metadata keys, nor in any way affect the ebuild aside from providing functionality, they can be conditionally pulled in. They also are allowed to pull in other elibs, but strictly just elibs- no eclasses, just other elibs. A realworld example would be the eutils eclass.

Portage, since the elib's don't modify metadata, isn't required to track elibs as it tracks eclasses. Thus a change in an elib doesn't result in half the tree forced to be regenerated/marked stale when changed (this is more of an infra benefit, although regen's that take too long due to eclass changes have been known to cause rsync issues due to missing timestamps). The only thing portage will do for elibs, aside from provide the elib function, is track what elibs have been loaded thus far, and load an elib only if it hasn't been loaded once already. An implication of this (if it wasn't clear from the elib description) is that elibs cannot change their exported api dependant on the api (as some eclass do for example).

Regarding maintainability of elibs, it should be a less of a load then old eclasses. One of the major issues with old eclasses is that their functions are quite incestuous- they're bound tightly to the env they're defined in. This makes eclass functions a bit fragile- the restrictions on what can, and cannot be done in elibs will address this, making functionality less fragile (thus a bit more maintainable).

There is no need for backwards compatability with elibs- they just must work against the current tree. Thus elibs can be removed when the tree no longer needs them. The reasons for this are explained below.

Structuring of the elibs directory will be exactly the same as that of the new eclass directory (detailed below), sans a different extension.

The reduced role of Eclasses, and a clarification of existing Eclass requirements

Since elibs are now intended on holding common bash functionality, the focus of eclasses should be in defining an appropriate template for ebuilds. For example, defining common DEPENDS, RDEPENDS, src_compile functions, src_unpack, etc. Additionally, eclasses should pull in any elibs they need for functionality.

Eclass functionality that isn't directly related to the metadata, or src_* and pkg_* funcs should be shifted into elibs to allow for maximal code reuse. This however isn't a hard requirement, merely a strongly worded suggestion.

Previously, it was 'strongly' suggested by developers to avoid having any code executed in the global scope that wasn't required. This suggestion is now a requirement. Execute only what must be executed in the global scope. Any code executed in the global scope that is related to configuring/building the package must be placed in pkg_setup. Metadata keys (already a rule, but now stated as an absolute requirement to clarify it) must be constant. The results of metadata keys exported from an ebuild on system A, must be exactly the same as the keys exported on system B.

If an eclass (or ebuild for that matter) violates this constant requirement, it leads to portage doing the wrong thing for rsync users- for example, wrong deps pulled in, leading to compilation failure.

If the existing metadata isn't flexible enough for what is required for a package, the parsing of the metadata is changed to address that. Cases where the constant requirement is violated are known, and a select few are allowed- these are exceptions to the rule that are required due to inadequacies in portage. In other words, those few exceptions are allowed because it's the only way to do it at this time. Any case where it's determined the constant requirement may need to be violated the dev must make it aware to the majority of devs, and the portage devs- violation of the constant rule has far reaching effects.

It's quite likely there is a way to allow what you're attempting- if you just go and do it, the rsync users (our userbase) suffer the results of compilation failures and unneeded deps being pulled in.

After that stern reminder, back to new eclasses. Defining INHERITED and ECLASS within the eclass is no longer required. Portage already handles those vars if they aren't defined.

As with elibs, it's no longer required backwards compatability be maintained indefinitely- compatability must be maintained against the current tree, but just that. As such new eclasses (the true distinction of new vs old is elaborated in the next section) can be removed from the tree once they're no longer in use.

The end of backwards compatability...

With current eclasses, once the eclass is in use, it's api can no longer be changed, nor can the eclass ever be removed from the tree. This is why we still have ancient eclasses that are completely unused sitting in the tree, for example inherit.eclass . The reason for this, not surprisingly is a portage deficiency- on unmerging an installed ebuild, portage used the eclass from the current tree.

For a real world example of this, if you merged a glibc 2 years back, whatever eclasses it used must still be compatible, or you may not be able to unmerge the older glibc version during an upgrade to a newer version. So either the glibc maintainer is left with the option of leaving people using ancient versions out in the rain, or maintaining an ever increasing load of backwards compatability cruft in any used eclasses.

Binpkgs suffer a similar fate. Merging of a binpkg pulls needed eclasses from the tree, so you may not be able to even merge a binpkg if the eclasses api has changed. If the eclass was removed, you can't even merge the binpkg, period.

The next major release of portage will address this- the environment that the ebuild was built in already contains the eclasses functions, as such the env can be re-used rather then relying on the eclass. In other words, binpkgs and installed ebuilds will no longer go and pull needed eclasses from the tree, they'll use the 'saved' version of the eclass they were built/merged with.

So the backwards compatability requirement for users of the next major portage version (and beyond) isn't required. All the cruft can be dropped.

The problem is that there will be users using older versions of portage that don't support this functionality. So backwards compatability must be maintained for them. Additionally, earlier versions of portage haven't always handled the env correctly- for broken saved envs, the eclasses backwards compatability is still required. Waiting N months preserving backwards compatability in current eclasses, then dropping the support isn't much of an option. There always are stragglers who don't upgrade, beyond that, there is the possibility of cases where users -will- upgrade, but still be bitten (broken saved envs from earlier portage installations). More importantly, it doesn't provide a route to upgrade/fix things if a user lags behind, exempting trying to find a compatabile version of the eclass in viewcvs (assuming it hasn't been sent to the attic already). Obviously, that isn't acceptable.

With the next major portage release, it will be possible to drop backwards compatability for eclasses, and all lingering cruft. What is needed is a way to take full advantage of this functionality, without completely screwing over the unfortunates and those who don't upgrade.

Unfortunately, the creation of new eclasses within the tree has an additional snag due to portage. The existing inherit function that is used to pull in old eclasses- basically, whatever it's passed (inherit kernel or inherit kernel/kernel) it will pull in (kernel.eclass, and kernel/kernel.eclass respectively). So even if the new eclasses were implemented within a subdirectory of the eclass dir in the tree, all current portage versions would still be able to access them.

In other words, these new eclasses would in effect, be old eclasses since older portage versions could still access them.

Tree restructuring.

There are only two way to block the existing (as of this writing) inherit functionality from accessing the new eclasses- either change the extension of eclasses to something other then 'eclass', or to have them stored in a seperate subdirectory of the tree then eclass.

The latter is preferable, and the proposed solution. Reasons are- the current eclass directory is already overgrown. Structuring of the new eclass dir (clarified below) will allow for easier signing, ChangeLogs, and grouping of eclasses. New eclasses allow for something akin to a clean break and have new capabilities/requirements, thus it's advisable to start with a clean directory, devoid of all cruft from the old eclass implementation.

If it's unclear as to why the old inherit function cannot access the new eclasses, please reread the previous section. It's unfortunately a requirement to take advantage of all that the next major portage release will allow.

The proposed directory sructure is ${PORTDIR}/include/{eclass,elib}. Something like ${PORTDIR}/new-eclass, or ${PORTDIR}/eclass-ng could be used (although many would cringe at the -ng), but such a name is unwise. Consider the possibility (likely a fact) that new eclasses someday may be found lacking, and refined further (version three as it were). Or perhaps we want to add yet more functionality with direct relation to sourcing new files, and we would then need to further populate ${PORTDIR}.

The new-eclass directory will be (at least) 2 levels deep- for example:

::
kernel/ kernel/linux-info.eclass kernel/linux-mod.eclass kernel/kernel-2.6.eclass kernel/kernel-2.4.eclass kernel/ChangeLog kernel/Manifest

No eclasses will be allowed in the base directory- grouping of new eclasses will be required to help keep things tidy, and for the following reasons. Grouping of eclasses allows for the addition of ChangeLogs that are specific to that group of eclasses, grouping of files/patches as needed, and allows for saner/easier signing of eclasses- basically, you can just stick a signed Manifest file w/in that grouping, thus providing the information portage needs to ensure no files are missing, and that nothing has been tainted.

The elib directory will be structured in the same way, for the same reasons.

Repoman will have to be extended to work within new eclass and elib groups, and to handle signing and commiting. This is intentional, and a good thing. This gives repoman the possibility of doing sanity checks on elibs/new eclasses. It won't solve developers doing dumb things with eclasses (no technological solution would, exempting a tazering), but it will give us a way to automate checks to try and prevent honest mistakes from slipping through and breaking things for our users.

The start of a different phase of backwards compatability

As clarified above, new eclasses will exist in a seperate directory that will be intentionally inaccessible to the inherit function. As such, users of older portage versions will have to upgrade to merge any ebuild that uses elibs/new eclasses. A depend on the next major portage version would address transparently handle this for rsync users.

There still is the issue of users who haven't upgraded to the required portage version. This is a minor concern frankly- portage releases include new functionality, and bug fixes. If they won't upgrade, it's assumed they have their reasons and are big boys, thus able to handle the complications themselves.

The real issue is broken envs, whether in binpkgs, or for installed packages. Two options exist- either the old eclasses are left in the tree indefinitely, or they're left for N months, then shifted out of the tree, and into a tarball that can be merged.

Shifting them out of the tree is advisable for several reasons- less cruft in the tree, but more importantly the fact that they are not signed (thus an angle for attack). Note that the proposed method of eclass signing doesn't even try to address them. Frankly, it's not worth the effort supporting two variations of eclass signing, when the old eclass setup isn't designed to allow for easy signing.

If this approach is taken, then either the old eclasses would have to be merged to an overlay directory's eclass directory (ugly), or to a safe location that portage's inherit function knows to look for (less ugly).

For users who do not upgrade within the window of N months while the old eclasses are in the tree, as stated, it's assumed they know what they are doing. If they specifically block the new portage version, as the ebuilds in the tree migrate to the new eclasses, they will have less and less ebuilds available to them. If they tried injecting the new portage version (lieing to portage, essentially), portage would bail since it cannot find the new eclass. Note that for them to even get to this point, they'd have to somehow disable the DEPEND on a new version of portage- either hack up the ebuild, or do an injection. Essentially they'd have to actively try to sidestep sanity checks implemented to make the shift over from old to new transparent. If they've disabled/sidestepped our attempt at a transparent migration, they can deal with the repercussions of it.

What is a bit more annoying is that once the old eclasses are out of the tree, users will lose the ability to unmerge any installed ebuild that used an old eclass, further users will lose the ability to merge any tbz2 that uses old eclasses.

They however will not be left out in the rain. For merging old eclass binpkgs, and unmerging installed packages, they can merge the old eclass compat ebuild. The compat ebuild provides the missing eclasses, thus providing that lost functionality.

The intention isn't to force them to upgrade, hence the ability to restore the lost functionality. The intention is to clean up the existing mess, and allow us to move forward. The saying "you've got to break a few eggs to make an omelete" is akin, exempting the fact we're providing a way to make the eggs whole again (the king's men would've loved such an option).

It's advisable that once all old eclasses are no longer in use in the tree, the old eclass package is added to system default. Remember that even those who have upgraded to a portage version that handles the env correctly, may run into instances where an installed packages env is corrupt. For new bootstraps (which automatically upgrade portage right off the bat), an injection of the compat package would be advisable- unless they downgrade portage, they will never need the old eclasses.

Migrating to the new setup

As has been done in the past whenever a change in the tree results in ebuilds requiring a specific version of portage, as ebuilds migrate to the new eclasses, they should depend on a version of portage that supports it. From the users viewpoint, this transparently handles the migration.

This isn't so transparent for devs or a particular infrastructure server however. Devs, due to them using cvs for their tree, lack the pregenerated cache rsync users have. Devs will have to be early adopters of the new portage. Older portage versions won't be able to access the new eclasses, thus the local cache generation for that ebuild will fail, ergo the depends on a newer portage version won't transparently handle it for them.

Additionally, prior to any ebuilds in the tree using the new eclasses, the infrastructure server that generates the cache for rsync users will have to either be upgraded to a version of portage supporting new eclasses, or patched. The former being much more preferable then the latter for the portage devs.

Beyond that, an appropriate window for old eclasses to exist in the tree must be determined, and prior to that window passing an ebuild must be added to the tree so users can get the old eclasses if needed.

For eclass devs to migrate from old to new, it is possible for them to just transfer the old eclass into an appropriate grouping in the new eclass directory, although it's advisable they cleanse all cruft out of the eclass. You can migrate ebuilds gradually over to the new eclass, and don't have to worry about having to support ebuilds from X years back.

Essentially, you have a chance to nail the design perfectly/cleanly, and have a window in which to redesign it. It's humbly suggested eclass devs take advantage of it. :)

Backwards Compatibility

All backwards compatability issues are addressed inline, but a recap is offered- it's suggested that if the a particular compatability issue is questioned/worried over, the reader read the relevant section. There should be a more in depth discussion of the issue, along with a more extensive explanation of the potential solutions, and reasons for the choosen solution.

To recap:

New eclasses and elib functionality will be tied to a specific portage
version. A DEPENDs on said portage version should address this for rsync
users who refuse to upgrade to a portage version that supports the new
eclasses/elibs and will gradually be unable to merge ebuilds that use said
functionality.  It is their choice to upgrade, as such, the gradual
'thinning' of available ebuilds should they block the portage upgrade is
their responsibility.

Old eclasses at some point in the future should be removed from the tree,
and released in a tarball/ebuild. This will cause installed ebuilds that
rely on the old eclass to be unable to unmerge to behave as expected, with
the same applying for merging of binpkgs.
        
This eclass ebuild should be a system depends target to make the transition
transparent. Future portage ebuilds, and the old eclass compat ebuild should
not inherit any eclasses. The reason for this is that in doing so, it may
block upgrade paths. At least for portage, this already is something of a
known issue for ebuild functionality- due to what it is/provides, it must
essentially be standalone, and cannot benefit from any eclass/elib
functionality.