/[gentoo]/xml/htdocs/proj/en/glep/glep-0033.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0033.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (show annotations) (download)
Tue Sep 5 20:54:30 2006 UTC (7 years, 11 months ago) by g2boojum
Branch: MAIN
Changes since 1.5: +6 -5 lines
File MIME type: text/plain
updates

1 GLEP: 33
2 Title: Eclass Restructure/Redesign
3 Version: $Revision: 1.5 $
4 Last-Modified: $Date: 2005/09/15 21:02:11 $
5 Author: Brian Harring <ferringb@gentoo.org>, John Mylchreest <johnm@gentoo.org>
6 Status: Moribund
7 Type: Standards Track
8 Content-Type: text/x-rst
9 Created: 29-Jan-2005
10 Post-History: 29-Jan-2005 6-Mar-2005 15-Sep-2005 5-Sep-2006
11
12 Status
13 ======
14
15 Approved by the Gentoo Council on 15 September 2005. As of Sept. 2006
16 this GLEP is on hold, pending future revisions.
17
18 Abstract
19 ========
20
21 For any design, the transition from theoretical to applied exposes inadequacies
22 in the original design. This document is intended to document, and propose a
23 revision of the current eclass setup to address current eclass inadequacies.
24
25 This document proposes several things- the creation of ebuild libraries, 'elibs',
26 a narrowing of the focus of eclasses, a move of eclasses w/in the tree, the
27 addition of changelogs, and a way to allow for simple eclass gpg signing.
28 In general, a large scale restructuring of what eclasses are and how they're
29 implemented. Essentially version two of the eclass setup.
30
31
32 Terminology
33 ===========
34
35 From this point on, the proposed eclass setup will be called 'new eclasses', the
36 existing crop (as of this writing) will be referenced as 'old eclasses'. The
37 distinction is elaborated on within this document.
38
39
40 Motivation and Rationale
41 ========================
42
43 Eclasses within the tree currently are a bit of a mess- they're forced to
44 maintain backwards compatibility w/ all previous functionality. In effect,
45 their api is constant, and can only be added to- never changing the existing
46 functionality. This obviously is quite limiting, and leads to cruft accruing in
47 eclasses as a eclasses design is refined. This needs to be dealt with prior to
48 eclass code reaching a critical mass where they become unmanageable/fragile
49 (recent pushes for eclass versioning could be interpreted as proof of this).
50
51 Beyond that, eclasses were originally intended as a method to allow for ebuilds
52 to use a pre-existing block of code, rather then having to duplicate the code in
53 each ebuild. This is a good thing, but there are ill effects that result from
54 the current design. Eclasses inherit other eclasses to get a single function- in
55 doing so, modifying the the exported 'template' (default src_compile, default
56 src_unpack, various vars, etc). All the eclass designer was after was reusing a
57 function, not making their eclass sensitive to changes in the template of the
58 eclass it's inheriting. The eclass designer -should- be aware of changes in the
59 function they're using, but shouldn't have to worry about their default src_*
60 and pkg_* functions being overwritten, let alone the env changes.
61
62 Addressing up front why a collection of eclass refinements are being rolled into
63 a single set of changes, parts of this proposal -could- be split into multiple
64 phases. Why do it though? It's simpler for developers to know that the first
65 eclass specification was this, and that the second specification is that,
66 rather then requiring them to be aware of what phase of eclass changes is in
67 progress.
68
69 By rolling all changes into one large change, a line is intentionally drawn in
70 the sand. Old eclasses allowed for this, behaved this way. New eclasses allow
71 for that, and behave this way. This should reduce misconceptions about what is
72 allowed/possible with eclasses, thus reducing bugs that result from said
73 misconceptions.
74
75 A few words on elibs- think of them as a clear definition between behavioral
76 functionality of an eclass, and the library functionality. Eclass's modify
77 template data, and are the basis for other ebuilds- elibs, however are *just*
78 common bash functionality.
79
80 Consider the majority of the portage bin/* scripts- these all are candidates for
81 being added to the tree as elibs, as is the bulk of eutils.
82
83
84 Specification
85 =============
86
87 The various parts of this proposal are broken down into a set of changes and
88 elaborations on why a proposed change is preferable. It's advisable to the
89 reader that this be read serially, rather then jumping around.
90
91
92 Ebuild Libraries (elibs for short)
93 ----------------------------------
94
95 As briefly touched upon in Motivation and Rationale, the original eclass design
96 allowed for the eclass to modify the metadata of an ebuild, metadata being the
97 DEPENDS, RDEPENDS, SRC_URI, IUSE, etc, vars that are required to be constant,
98 and used by portage for dep resolution, fetching, etc. Using the earlier
99 example, if you're after a single function from an eclass (say epatch from
100 eutils), you -don't- want the metadata modifications the eclass you're
101 inheriting might do. You want to treat the eclass you're pulling from as a
102 library, pure and simple.
103
104 A new directory named elib should be added to the top level of the tree to serve
105 as a repository of ebuild function libraries. Rather then relying on using the
106 source command, an 'elib' function should be added to portage to import that
107 libraries functionality. The reason for the indirection via the function is
108 mostly related to portage internals, but it does serve as an abstraction such
109 that (for example) zsh compatibility hacks could be hidden in the elib function.
110
111 Elib's will be collections of bash functions- they're not allowed to do anything
112 in the global scope aside from function definition, and any -minimal-
113 initialization of the library that is absolutely needed. Additionally, they
114 cannot modify any ebuild template functions- src_compile, src_unpack. Since they are
115 required to not modify the metadata keys, nor in any way affect the ebuild aside
116 from providing functionality, they can be conditionally pulled in. They also
117 are allowed to pull in other elibs, but strictly just elibs- no eclasses, just
118 other elibs. A real world example would be the eutils eclass.
119
120 Portage, since the elib's don't modify metadata, isn't required to track elibs
121 as it tracks eclasses. Thus a change in an elib doesn't result in half the tree
122 forced to be regenerated/marked stale when changed (this is more of an infra
123 benefit, although regen's that take too long due to eclass changes have been
124 known to cause rsync issues due to missing timestamps).
125
126 Elibs will not be available in the global scope of an eclass, or ebuild- nor during the
127 depends phase (basically a phase that sources the ebuild, to get its metadata). Elib
128 calls in the global scope will be tracked, but the elib will not be loaded till just before
129 the setup phase (pkg_setup). There are two reasons for this- first, it ensures elibs are
130 completely incapable of modifying metadata. There is no room for confusion, late loading
131 of elibs gives you the functionality for all phases, except for depends- depends being the
132 only phase that is capable of specifying metadata. Second, as an added bonus, late
133 loading reduces the amount of bash sourced for a regen- faster regens. This however is minor,
134 and is an ancillary benefit of the first reason.
135
136 There are a few further restrictions with elibs--mainly, elibs to load can only be specified
137 in either global scope, or in the setup, unpack, compile, test, and install phases. You can
138 not load elibs in prerm, postrm, preinst, and postinst. The reason being, for \*rm phases,
139 installed pkgs will have to look to the tree for the elib, which allows for api drift to cause
140 breakage. For \*inst phases, same thing, except the culprit is binpkgs.
141
142 There is a final restriction--elibs cannot change their exported api dependent on the api
143 (as some eclass do for example). The reason mainly being that elibs are loaded once--not
144 multiple times, as eclasses are.
145
146 To clarify, for example this is invalid.
147 ::
148
149 if [[ -n ${SOME_VAR} ]]; then
150 func x() { echo "I'm accessible only via tweaking some var";}
151 else
152 func x() { echo "this is invalid, do not do it."; }
153 fi
154
155
156 Regarding maintainability of elibs, it should be a less of a load then old
157 eclasses. One of the major issues with old eclasses is that their functions are
158 quite incestuous- they're bound tightly to the env they're defined in. This
159 makes eclass functions a bit fragile- the restrictions on what can, and cannot
160 be done in elibs will address this, making functionality less fragile (thus a
161 bit more maintainable).
162
163 There is no need for backwards compatibility with elibs- they just must work
164 against the current tree. Thus elibs can be removed when the tree no longer
165 needs them. The reasons for this are explained below.
166
167 Structuring of the elibs directory will be exactly the same as that of the new
168 eclass directory (detailed below), sans a different extension.
169
170 As to why their are so many restrictions, the answer is simple- the definition of
171 what elibs are, what they are capable of, and how to use them is nailed down as much as
172 possible to avoid *any* ambiguity related to them. The intention is to make it clear,
173 such that no misconceptions occur, resulting in bugs.
174
175 The reduced role of Eclasses, and a clarification of existing Eclass requirements
176 ---------------------------------------------------------------------------------
177
178 Since elibs are now intended on holding common bash functionality, the focus of
179 eclasses should be in defining an appropriate template for ebuilds. For example,
180 defining common DEPENDS, RDEPENDS, src_compile functions, src_unpack, etc.
181 Additionally, eclasses should pull in any elibs they need for functionality.
182
183 Eclass functionality that isn't directly related to the metadata, or src_* and
184 pkg_* funcs should be shifted into elibs to allow for maximal code reuse. This
185 however isn't a hard requirement, merely a strongly worded suggestion.
186
187 Previously, it was 'strongly' suggested by developers to avoid having any code
188 executed in the global scope that wasn't required. This suggestion is now a
189 requirement. Execute only what must be executed in the global scope. Any code
190 executed in the global scope that is related to configuring/building the package
191 must be placed in pkg_setup. Metadata keys (already a rule, but now stated as
192 an absolute requirement to clarify it) *must* be constant. The results of
193 metadata keys exported from an ebuild on system A, must be *exactly* the same as
194 the keys exported on system B.
195
196 If an eclass (or ebuild for that matter) violates this constant requirement, it
197 leads to portage doing the wrong thing for rsync users- for example, wrong deps
198 pulled in, leading to compilation failure, or dud deps.
199
200 If the existing metadata isn't flexible enough for what is required for a
201 package, the parsing of the metadata is changed to address that. Cases where
202 the constant requirement is violated are known, and a select few are allowed-
203 these are exceptions to the rule that are required due to inadequacies in
204 portage. Any case where it's determined the constant requirement may need to be
205 violated the dev must make it aware to the majority of devs, along with the portage
206 devs. This should be done prior to committing.
207
208 It's quite likely there is a way to allow what you're attempting- if you just go
209 and do it, the rsync users (our user base) suffer the results of compilation
210 failures and unneeded deps being pulled in.
211
212 After that stern reminder, back to new eclasses. Defining INHERITED and ECLASS
213 within the eclass is no longer required. Portage already handles those vars if
214 they aren't defined.
215
216 As with elibs, it's no longer required that backwards compatibility be maintained
217 indefinitely- compatibility must be maintained against the current tree, but
218 just that. As such new eclasses (the true distinction of new vs old is
219 elaborated in the next section) can be removed from the tree once they're no
220 longer in use.
221
222
223 The end of backwards compatibility...
224 -------------------------------------
225
226 With current eclasses, once the eclass is in use, its api can no longer be
227 changed, nor can the eclass ever be removed from the tree. This is why we still
228 have *ancient* eclasses that are completely unused sitting in the tree, for
229 example inherit.eclass. The reason for this, not surprisingly, is a portage
230 deficiency: on unmerging an installed ebuild, portage used the eclass from the
231 current tree.
232
233 For a real world example of this, if you merged a glibc 2 years back, whatever
234 eclasses it used must still be compatible, or you may not be able to unmerge the
235 older glibc version during an upgrade to a newer version. So either the glibc
236 maintainer is left with the option of leaving people using ancient versions out
237 in the rain, or maintaining an ever increasing load of backwards compatibility
238 cruft in any used eclasses.
239
240 Binpkgs suffer a similar fate. Merging of a binpkg pulls needed eclasses from
241 the tree, so you may not be able to even merge a binpkg if the eclasses api has
242 changed. If the eclass was removed, you can't even merge the binpkg, period.
243
244 The next major release of portage will address this- the environment that the
245 ebuild was built in already contains the eclasses functions, as such the env can
246 be re-used rather then relying on the eclass. In other words, binpkgs and
247 installed ebuilds will no longer go and pull needed eclasses from the tree,
248 they'll use the 'saved' version of the eclass they were built/merged with.
249
250 So the backwards compatibility requirement for users of the next major portage
251 version (and beyond) isn't required. All the cruft can be dropped.
252
253 The problem is that there will be users using older versions of portage that don't
254 support this functionality- these older installations *cannot* use the
255 new eclasses, due to the fact that their portage version is incapable of
256 properly relying on the env- in other words, the varying api of the eclass will
257 result in user-visible failures during unmerging.
258
259 So we're able to do a clean break of all old eclasses, and api cruft, but we need
260 a means to basically disallow access to the new eclasses for all portage versions
261 incapable of properly handling the env requirements.
262
263 Unfortunately, we cannot just rely on a different grouping/naming convention within
264 the old eclass directory. The new eclasses must be inaccessible, and portage throws
265 a snag into this- the existing inherit function that is used to handle existing
266 eclasses. Basically, whatever it's passed (inherit kernel or inherit
267 kernel/kernel) it will pull in (kernel.eclass, and kernel/kernel.eclass
268 respectively). So even if the new eclasses were implemented within a
269 subdirectory of the eclass dir in the tree, all current portage versions would
270 still be able to access them.
271
272 In other words, these new eclasses would in effect, be old eclasses since older
273 portage versions could still access them.
274
275
276 Tree restructuring
277 ------------------
278
279 There are only two way to block the existing (as of this writing) inherit
280 functionality from accessing the new eclasses- either change the extension of
281 eclasses to something other then 'eclass', or to have them stored in a separate
282 subdirectory of the tree then eclass.
283
284 The latter is preferable, and the proposed solution. Reasons are- the current
285 eclass directory is already overgrown. Structuring of the new eclass dir
286 (clarified below) will allow for easier signing, ChangeLogs, and grouping of
287 eclasses. New eclasses allow for something akin to a clean break and have new
288 capabilities/requirements, thus it's advisable to start with a clean directory,
289 devoid of all cruft from the old eclass implementation.
290
291 If it's unclear as to why the old inherit function *cannot* access the new
292 eclasses, please reread the previous section. It's unfortunately a requirement
293 to take advantage of all that the next major portage release will allow.
294
295 The proposed directory structure is ${PORTDIR}/include/{eclass,elib}.
296 Something like ${PORTDIR}/new-eclass, or ${PORTDIR}/eclass-ng could be used
297 (although many would cringe at the -ng), but such a name is unwise. Consider the
298 possibility (likely a fact) that new eclasses someday may be found lacking, and
299 refined further (version three as it were). Or perhaps we want to add yet more
300 functionality with direct relation to sourcing new files, and we would then need
301 to further populate ${PORTDIR}.
302
303 The new-eclass directory will be (at least) 2 levels deep- for example:
304
305 ::
306 kernel/
307 kernel/linux-info.eclass
308 kernel/linux-mod.eclass
309 kernel/kernel-2.6.eclass
310 kernel/kernel-2.4.eclass
311 kernel/ChangeLog
312 kernel/Manifest
313
314 No eclasses will be allowed in the base directory- grouping of new eclasses will
315 be required to help keep things tidy, and for the following reasons. Grouping
316 of eclasses allows for the addition of ChangeLogs that are specific to that
317 group of eclasses, grouping of files/patches as needed, and allows for
318 saner/easier signing of eclasses- you can just stick a signed
319 Manifest file w/in that grouping, thus providing the information portage needs
320 to ensure no files are missing, and that nothing has been tainted.
321
322 The elib directory will be structured in the same way, for the same reasons.
323
324 Repoman will have to be extended to work within new eclass and elib groups, and
325 to handle signing and committing. This is intentional, and a good thing. This
326 gives repoman the possibility of doing sanity checks on elibs/new eclasses.
327
328 Note these checks will not prevent developers from doing dumb things with eclass-
329 these checks would only be capable of doing basic sanity checks, such as syntax checks.
330 There is no way to prevent people from doing dumb things (exempting perhaps repeated
331 applications of a cattle prod)- these are strictly automatic checks, akin to repoman's
332 dependency checks.
333
334
335 The start of a different phase of backwards compatibility
336 ---------------------------------------------------------
337
338 As clarified above, new eclasses will exist in a separate directory that will be
339 intentionally inaccessible to the inherit function. As such, users of older
340 portage versions *will* have to upgrade to merge any ebuild that uses elibs/new
341 eclasses. A depend on the next major portage version would transparently handle
342 this for rsync users.
343
344 There still is the issue of users who haven't upgraded to the required portage
345 version. This is a minor concern frankly- portage releases include new
346 functionality, and bug fixes. If they won't upgrade, it's assumed they have
347 their reasons and are big boys, thus able to handle the complications themselves.
348
349 The real issue is broken envs, whether in binpkgs, or for installed packages.
350 Two options exist- either the old eclasses are left in the tree indefinitely, or
351 they're left for N months, then shifted out of the tree, and into a tarball that
352 can be merged.
353
354 Shifting them out of the tree is advisable for several reasons- less cruft in
355 the tree, but more importantly the fact that they are not signed (thus an angle
356 for attack). Note that the proposed method of eclass signing doesn't even try
357 to address them. Frankly, it's not worth the effort supporting two variations
358 of eclass signing, when the old eclass setup isn't designed to allow for easy
359 signing.
360
361 If this approach is taken, then either the old eclasses would have to be merged
362 to an overlay directory's eclass directory (ugly), or to a safe location that
363 portage's inherit function knows to look for (less ugly).
364
365 For users who do not upgrade within the window of N months while the old
366 eclasses are in the tree, as stated, it's assumed they know what they are doing.
367 If they specifically block the new portage version, as the ebuilds in the tree
368 migrate to the new eclasses, they will have less and less ebuilds available to
369 them. If they tried injecting the new portage version (lying to portage,
370 essentially), portage would bail since it cannot find the new eclass.
371 For ebuilds that use the new eclasses, there really isn't any way to sidestep
372 the portage version requirement- same as it has been for other portage features.
373
374 What is a bit more annoying is that once the old eclasses are out of the tree,
375 if a user has not upgraded to a portage version supporting env processing, they
376 will lose the ability to unmerge any installed ebuild that used an old
377 eclass. Same cause, different symptom being they will lose the ability to merge
378 any tbz2 that uses old eclasses also.
379
380 There is one additional case that is a rarity, but should be noted- if a user
381 has suffered significant corruption of their installed package database (vdb). This is
382 ignoring the question of whether the vdb is even usable at this point, but the possibility
383 exists for the saved envs to be non usable due to either A) missing, or B) corrupted.
384 In such a case, even with the new portage capabilities, they would need
385 the old eclass compat ebuild.
386
387 Note for this to happen requires either rather... unwise uses of root, or significant
388 fs corruption. Regardless of the cause, it's quite likely for this to even become an
389 issue, the system's vdb is completely unusable. It's a moot issue at that point.
390 If you lose your vdb, or it gets seriously damaged, it's akin to lobotomizing portage-
391 it doesn't know what's installed, it doesn't know of its own files, and in general,
392 a rebuilding of the system is about the only sane course of action. The missing env is
393 truly the least of the users concern in such a case.
394
395 Continuing with the more likely scenario, users unwilling to upgrade portage will
396 *not* be left out in the rain. Merging the old eclass compat ebuild will provide
397 the missing eclasses, thus providing that lost functionality.
398
399 Note the intention isn't to force them to upgrade, hence the ability to restore the
400 lost functionality. The intention is to clean up the existing mess, and allow us
401 to move forward. The saying "you've got to break a few eggs to make an omelet"
402 is akin, exempting the fact we're providing a way to make the eggs whole again
403 (the king's men would've loved such an option).
404
405
406 Migrating to the new setup
407 --------------------------
408
409 As has been done in the past whenever a change in the tree results in ebuilds
410 requiring a specific version of portage, as ebuilds migrate to the new eclasses,
411 they should depend on a version of portage that supports it. From the users
412 viewpoint, this transparently handles the migration.
413
414 This isn't so transparent for devs or a particular infrastructure server however.
415 Devs, due to them using cvs for their tree, lack the pregenerated cache rsync
416 users have. Devs will have to be early adopters of the new portage. Older
417 portage versions won't be able to access the new eclasses, thus the local cache
418 generation for that ebuild will fail, ergo the depends on a newer portage
419 version won't transparently handle it for them.
420
421 Additionally, prior to any ebuilds in the tree using the new eclasses, the
422 infrastructure server that generates the cache for rsync users will have to
423 either be upgraded to a version of portage supporting new eclasses, or patched.
424 The former being much more preferable then the latter for the portage devs.
425
426 Beyond that, an appropriate window for old eclasses to exist in the tree must be
427 determined, and prior to that window passing, an ebuild must be added to the tree
428 so users can get the old eclasses if needed.
429
430 For eclass devs to migrate from old to new, it is possible for them to just
431 transfer the old eclass into an appropriate grouping in the new eclass directory,
432 although it's advisable they cleanse all cruft out of the eclass. You can
433 migrate ebuilds gradually over to the new eclass, and don't have to worry about
434 having to support ebuilds from X years back.
435
436 Essentially, you have a chance to nail the design perfectly/cleanly, and have a
437 window in which to redesign it. It's humbly suggested eclass devs take
438 advantage of it. :)
439
440
441 Backwards Compatibility
442 =======================
443
444 All backwards compatibility issues are addressed in line, but a recap is offered-
445 it's suggested that if the a particular compatibility issue is
446 questioned/worried over, the reader read the relevant section. There should be
447 a more in depth discussion of the issue, along with a more extensive explanation
448 of the potential solutions, and reasons for the chosen solution.
449
450 To recap:
451 ::
452
453 New eclasses and elib functionality will be tied to a specific portage
454 version. A DEPENDs on said portage version should address this for rsync
455 users who refuse to upgrade to a portage version that supports the new
456 eclasses/elibs and will gradually be unable to merge ebuilds that use said
457 functionality. It is their choice to upgrade, as such, the gradual
458 'thinning' of available ebuilds should they block the portage upgrade is
459 their responsibility.
460
461 Old eclasses at some point in the future should be removed from the tree,
462 and released in a tarball/ebuild. This will cause installed ebuilds that
463 rely on the old eclass to be unable to unmerge, with the same applying for
464 merging of binpkgs dependent on the following paragraph.
465
466 The old eclass-compat is only required for users who do not upgrade their
467 portage installation, and one further exemption- if the user has somehow
468 corrupted/destroyed their installed pkgs database (/var/db/pkg currently),
469 in the process, they've lost their saved environments. The eclass-compat
470 ebuild would be required for ebuilds that required older eclasses in such a
471 case. Note, this case is rare also- as clarified above, it's mentioned
472 strictly to be complete, it's not much of a real world scenario as elaborated
473 above.
474
475
476 Copyright
477 =========
478
479 This document has been placed in the public domain.

  ViewVC Help
Powered by ViewVC 1.1.20