/[gentoo]/xml/htdocs/proj/en/glep/glep-0033.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0033.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.1 - (show annotations) (download)
Wed Feb 16 21:30:44 2005 UTC (9 years, 4 months ago) by g2boojum
Branch: MAIN
File MIME type: text/plain
new glep

1 GLEP: 33
2 Title: Eclass Restructure/Redesign
3 Version: $Revision: 1.0 $
4 Last-Modified: $Date: 2005/02/15 00:00:00 $
5 Author: John Mylchreest <johnm@gentoo.org>, Brian Harring <ferringb@gentoo.org>
6 Status: Draft
7 Type: Standards Track
8 Content-Type: text/x-rst
9 Created: 29-Jan-2005
10 Post-History: 29-Jan-2005
11
12
13 Abstract
14 ========
15
16 For any design, the transition from theoretical to applied exposes inadequacies
17 in the original design. This document is intended to document, and propose a
18 revision of the current eclass setup to address current eclass inadequacies.
19
20 This document proposes several thing- the creation of ebuild libraries, 'elibs',
21 a narrowing of the focus of eclasses, a move of eclasses w/in the tree, the
22 addition of changelogs, and a way to allow for simple eclass gpg signing.
23 In general, a large scale restructuring of what eclasses are and how they're
24 implemented. Essentially version two of the eclass setup.
25
26
27 Terminology
28 ===========
29
30 From this point on, the proposed eclass setup will be called 'new eclasses', the
31 existing crop (as of this writing) will be referenced as 'old eclasses'. The
32 destinction is elaborated on within this document.
33
34
35 Motivation and Rationale
36 ========================
37
38 Eclasses within the tree currently are a bit of a mess- they're forced to
39 maintain backwards compatability w/ all previous functionality. In effect,
40 their api is constant, and can only be added to- never changing the existing
41 functionality. This obviously is quite limiting, and leads to cruft accrueing in
42 eclasses as a eclasses design is refined. This needs to be dealt with prior to
43 eclass code reaching a critical mass where they become unmanagable/fragile
44 (recent pushes for eclass versioning could be interpretted as proof of this).
45
46 Beyond that, eclasses were originally intended as a method to allow for ebuilds
47 to use a pre-existing block of code, rather then having to duplicate the code in
48 each ebuild. This is a good thing, but there are ill effects that result from
49 the current design. Eclasses inherit other eclasses to get a single function- in
50 doing so, modifying the the exported 'template' (default src_compile, default
51 src_unpack, various vars, etc). All the eclass designer was after was reusing a
52 function, not making their eclass sensitive to changes in the template of the
53 eclass it's inheriting. The eclass designer -should- be aware of changes in the
54 function they're using, but shouldn't have to worry about their default src_*
55 and pkg_* functions being overwritten, let alone the env changes.
56
57 Addressing up front why a collection of eclass refinements are being rolled into
58 a single set of changes, parts of this proposal -could- be split into multiple
59 phases. Why do it though? It's simpler for developers to know that the first
60 eclass specification was this, and that the second specification is that,
61 rather then requiring them to be aware of what phase of eclass changes is in
62 progress.
63
64 By rolling all changes into one large change, a line is intentionally drawn in
65 the sand. Old eclasses allowed for this, behaved this way. New eclasses allow
66 for that, and behave this way. This should reduce misconceptions about what is
67 allowed/possible with eclasses, thus reducing bugs that result from said
68 misconceptions.
69
70
71 Specification.
72 ==============
73
74 The various parts of this proposal are broken down into a set of changes and
75 elaborations on why a proposed change is preferable. It's advisable to the
76 reader that this be read serially, rather then jumping around.
77
78
79 Ebuild Libraries (elibs for short)
80 ----------------------------------
81
82 As briefly touched upon in Motivation and Rationale, the original eclass design
83 allowed for the eclass to modify the metadata of an ebuild, metadata being the
84 DEPENDS, RDEPENDS, SRC_URI, IUSE, etc, vars that are required to be constant,
85 and used by portage for dep resolution, fetching, etc. Using the earlier
86 example, if you're after a single function from an eclass (say epatch from
87 eutils), you -don't- want the metadata modifications the eclass you're
88 inheriting might do. You want to treat the eclass you're pulling from as a
89 library, pure and simple.
90
91 A new directory named elib should be added to the top level of the tree to serve
92 as a repository of ebuild function libraries. Rather then relying on using the
93 source command, an 'elib' function should be added to portage to import that
94 libraries functionality. The reason for the indirection via the function is
95 mostly related to portage internals, but it does serve as an abstraction such
96 that (for example) zsh compatability hacks could be hidden in the elib function.
97
98 Elib's will be collections of bash functions- they're not allowed to do anything
99 in the global scope aside from function definition, and any -minimal-
100 initialization of the library that is absolutely needed. Additionally, they
101 cannot modify any ebuild functions- src_compile, src_unpack fex. Since they are
102 required to not modify the metadata keys, nor in any way affect the ebuild aside
103 from providing functionality, they can be conditionally pulled in. They also
104 are allowed to pull in other elibs, but strictly just elibs- no eclasses, just
105 other elibs. A realworld example would be the eutils eclass.
106
107 Portage, since the elib's don't modify metadata, isn't required to track elibs
108 as it tracks eclasses. Thus a change in an elib doesn't result in half the tree
109 forced to be regenerated/marked stale when changed (this is more of an infra
110 benefit, although regen's that take too long due to eclass changes have been
111 known to cause rsync issues due to missing timestamps). The only thing portage
112 will do for elibs, aside from provide the elib function, is track what elibs
113 have been loaded thus far, and load an elib only if it hasn't been loaded once
114 already. An implication of this (if it wasn't clear from the elib description)
115 is that elibs cannot change their exported api dependant on the api (as some
116 eclass do for example).
117
118 Regarding maintainability of elibs, it should be a less of a load then old
119 eclasses. One of the major issues with old eclasses is that their functions are
120 quite incestuous- they're bound tightly to the env they're defined in. This
121 makes eclass functions a bit fragile- the restrictions on what can, and cannot
122 be done in elibs will address this, making functionality less fragile (thus a
123 bit more maintainable).
124
125 There is no need for backwards compatability with elibs- they just must work
126 against the current tree. Thus elibs can be removed when the tree no longer
127 needs them. The reasons for this are explained below.
128
129 Structuring of the elibs directory will be exactly the same as that of the new
130 eclass directory (detailed below), sans a different extension.
131
132
133 The reduced role of Eclasses, and a clarification of existing Eclass requirements
134 ---------------------------------------------------------------------------------
135
136 Since elibs are now intended on holding common bash functionality, the focus of
137 eclasses should be in defining an appropriate template for ebuilds. For example,
138 defining common DEPENDS, RDEPENDS, src_compile functions, src_unpack, etc.
139 Additionally, eclasses should pull in any elibs they need for functionality.
140
141 Eclass functionality that isn't directly related to the metadata, or src_* and
142 pkg_* funcs should be shifted into elibs to allow for maximal code reuse. This
143 however isn't a hard requirement, merely a strongly worded suggestion.
144
145 Previously, it was 'strongly' suggested by developers to avoid having any code
146 executed in the global scope that wasn't required. This suggestion is now a
147 requirement. Execute only what must be executed in the global scope. Any code
148 executed in the global scope that is related to configuring/building the package
149 must be placed in pkg_setup. Metadata keys (already a rule, but now stated as
150 an absolute requirement to clarify it) *must* be constant. The results of
151 metadata keys exported from an ebuild on system A, must be *exactly* the same as
152 the keys exported on system B.
153
154 If an eclass (or ebuild for that matter) violates this constant requirement, it
155 leads to portage doing the wrong thing for rsync users- for example, wrong deps
156 pulled in, leading to compilation failure.
157
158 If the existing metadata isn't flexible enough for what is required for a
159 package, the parsing of the metadata is changed to address that. Cases where
160 the constant requirement is violated are known, and a select few are allowed-
161 these are exceptions to the rule that are required due to inadequacies in
162 portage. In other words, those *few* exceptions are allowed because it's the
163 only way to do it at this time. Any case where it's determined the constant
164 requirement may need to be violated the dev must make it aware to the majority
165 of devs, and the portage devs- violation of the constant rule has far reaching
166 effects.
167
168 It's quite likely there is a way to allow what you're attempting- if you just go
169 and do it, the rsync users (our userbase) suffer the results of compilation
170 failures and unneeded deps being pulled in.
171
172 After that stern reminder, back to new eclasses. Defining INHERITED and ECLASS
173 within the eclass is no longer required. Portage already handles those vars if
174 they aren't defined.
175
176 As with elibs, it's no longer required backwards compatability be maintained
177 indefinitely- compatability must be maintained against the current tree, but
178 just that. As such new eclasses (the true distinction of new vs old is
179 elaborated in the next section) can be removed from the tree once they're no
180 longer in use.
181
182
183 The end of backwards compatability...
184 -------------------------------------
185
186 With current eclasses, once the eclass is in use, it's api can no longer be
187 changed, nor can the eclass ever be removed from the tree. This is why we still
188 have *ancient* eclasses that are completely unused sitting in the tree, for
189 example inherit.eclass . The reason for this, not surprisingly is a portage
190 deficiency- on unmerging an installed ebuild, portage used the eclass from the
191 current tree.
192
193 For a real world example of this, if you merged a glibc 2 years back, whatever
194 eclasses it used must still be compatible, or you may not be able to unmerge the
195 older glibc version during an upgrade to a newer version. So either the glibc
196 maintainer is left with the option of leaving people using ancient versions out
197 in the rain, or maintaining an ever increasing load of backwards compatability
198 cruft in any used eclasses.
199
200 Binpkgs suffer a similar fate. Merging of a binpkg pulls needed eclasses from
201 the tree, so you may not be able to even merge a binpkg if the eclasses api has
202 changed. If the eclass was removed, you can't even merge the binpkg, period.
203
204 The next major release of portage will address this- the environment that the
205 ebuild was built in already contains the eclasses functions, as such the env can
206 be re-used rather then relying on the eclass. In other words, binpkgs and
207 installed ebuilds will no longer go and pull needed eclasses from the tree,
208 they'll use the 'saved' version of the eclass they were built/merged with.
209
210 So the backwards compatability requirement for users of the next major portage
211 version (and beyond) isn't required. All the cruft can be dropped.
212
213 The problem is that there will be users using older versions of portage that
214 don't support this functionality. So backwards compatability must be maintained
215 for them. Additionally, earlier versions of portage haven't always handled the
216 env correctly- for broken saved envs, the eclasses backwards compatability is
217 still required. Waiting N months preserving backwards compatability in current
218 eclasses, then dropping the support isn't much of an option. There always are
219 stragglers who don't upgrade, beyond that, there is the possibility of cases
220 where users -will- upgrade, but still be bitten (broken saved envs from earlier
221 portage installations). More importantly, it doesn't provide a route to
222 upgrade/fix things if a user lags behind, exempting trying to find a compatabile
223 version of the eclass in viewcvs (assuming it hasn't been sent to the attic
224 already). Obviously, that isn't acceptable.
225
226 With the next major portage release, it will be possible to drop backwards
227 compatability for eclasses, and all lingering cruft. What is needed is a way to
228 take full advantage of this functionality, without completely screwing over the
229 unfortunates and those who don't upgrade.
230
231 Unfortunately, the creation of new eclasses within the tree has an additional
232 snag due to portage. The existing inherit function that is used to pull in old
233 eclasses- basically, whatever it's passed (inherit kernel or inherit
234 kernel/kernel) it will pull in (kernel.eclass, and kernel/kernel.eclass
235 respectively). So even if the new eclasses were implemented within a
236 subdirectory of the eclass dir in the tree, all current portage versions would
237 still be able to access them.
238
239 In other words, these new eclasses would in effect, be old eclasses since older
240 portage versions could still access them.
241
242
243 Tree restructuring.
244 -------------------
245
246 There are only two way to block the existing (as of this writing) inherit
247 functionality from accessing the new eclasses- either change the extension of
248 eclasses to something other then 'eclass', or to have them stored in a seperate
249 subdirectory of the tree then eclass.
250
251 The latter is preferable, and the proposed solution. Reasons are- the current
252 eclass directory is already overgrown. Structuring of the new eclass dir
253 (clarified below) will allow for easier signing, ChangeLogs, and grouping of
254 eclasses. New eclasses allow for something akin to a clean break and have new
255 capabilities/requirements, thus it's advisable to start with a clean directory,
256 devoid of all cruft from the old eclass implementation.
257
258 If it's unclear as to why the old inherit function *cannot* access the new
259 eclasses, please reread the previous section. It's unfortunately a requirement
260 to take advantage of all that the next major portage release will allow.
261
262 The proposed directory sructure is ${PORTDIR}/include/{eclass,elib}.
263 Something like ${PORTDIR}/new-eclass, or ${PORTDIR}/eclass-ng could be used
264 (although many would cringe at the -ng), but such a name is unwise. Consider the
265 possibility (likely a fact) that new eclasses someday may be found lacking, and
266 refined further (version three as it were). Or perhaps we want to add yet more
267 functionality with direct relation to sourcing new files, and we would then need
268 to further populate ${PORTDIR}.
269
270 The new-eclass directory will be (at least) 2 levels deep- for example:
271
272 ::
273 kernel/
274 kernel/linux-info.eclass
275 kernel/linux-mod.eclass
276 kernel/kernel-2.6.eclass
277 kernel/kernel-2.4.eclass
278 kernel/ChangeLog
279 kernel/Manifest
280
281 No eclasses will be allowed in the base directory- grouping of new eclasses will
282 be required to help keep things tidy, and for the following reasons. Grouping
283 of eclasses allows for the addition of ChangeLogs that are specific to that
284 group of eclasses, grouping of files/patches as needed, and allows for
285 saner/easier signing of eclasses- basically, you can just stick a signed
286 Manifest file w/in that grouping, thus providing the information portage needs
287 to ensure no files are missing, and that nothing has been tainted.
288
289 The elib directory will be structured in the same way, for the same reasons.
290
291 Repoman will have to be extended to work within new eclass and elib groups, and
292 to handle signing and commiting. This is intentional, and a good thing. This
293 gives repoman the possibility of doing sanity checks on elibs/new eclasses.
294 It won't solve developers doing dumb things with eclasses (no technological
295 solution would, exempting a tazering), but it will give us a way to automate
296 checks to try and prevent honest mistakes from slipping through and breaking
297 things for our users.
298
299
300 The start of a different phase of backwards compatability
301 ---------------------------------------------------------
302
303 As clarified above, new eclasses will exist in a seperate directory that will be
304 intentionally inaccessible to the inherit function. As such, users of older
305 portage versions *will* have to upgrade to merge any ebuild that uses elibs/new
306 eclasses. A depend on the next major portage version would address
307 transparently handle this for rsync users.
308
309 There still is the issue of users who haven't upgraded to the required portage
310 version. This is a minor concern frankly- portage releases include new
311 functionality, and bug fixes. If they won't upgrade, it's assumed they have
312 their reasons and are big boys, thus able to handle the complications themselves.
313
314 The real issue is broken envs, whether in binpkgs, or for installed packages.
315 Two options exist- either the old eclasses are left in the tree indefinitely, or
316 they're left for N months, then shifted out of the tree, and into a tarball that
317 can be merged.
318
319 Shifting them out of the tree is advisable for several reasons- less cruft in
320 the tree, but more importantly the fact that they are not signed (thus an angle
321 for attack). Note that the proposed method of eclass signing doesn't even try
322 to address them. Frankly, it's not worth the effort supporting two variations
323 of eclass signing, when the old eclass setup isn't designed to allow for easy
324 signing.
325
326 If this approach is taken, then either the old eclasses would have to be merged
327 to an overlay directory's eclass directory (ugly), or to a safe location that
328 portage's inherit function knows to look for (less ugly).
329
330 For users who do not upgrade within the window of N months while the old
331 eclasses are in the tree, as stated, it's assumed they know what they are doing.
332 If they specifically block the new portage version, as the ebuilds in the tree
333 migrate to the new eclasses, they will have less and less ebuilds available to
334 them. If they tried injecting the new portage version (lieing to portage,
335 essentially), portage would bail since it cannot find the new eclass. Note that
336 for them to even get to this point, they'd have to somehow disable the DEPEND on
337 a new version of portage- either hack up the ebuild, or do an injection.
338 Essentially they'd have to actively try to sidestep sanity checks implemented to
339 make the shift over from old to new transparent. If they've
340 disabled/sidestepped our attempt at a transparent migration, they can deal with
341 the repercussions of it.
342
343 What is a bit more annoying is that once the old eclasses are out of the tree,
344 users will lose the ability to unmerge any installed ebuild that used an old
345 eclass, further users will lose the ability to merge any tbz2 that uses old
346 eclasses.
347
348 They however will *not* be left out in the rain. For merging old eclass
349 binpkgs, and unmerging installed packages, they can merge the old eclass compat
350 ebuild. The compat ebuild provides the missing eclasses, thus providing that
351 lost functionality.
352
353 The intention isn't to force them to upgrade, hence the ability to restore the
354 lost functionality. The intention is to clean up the existing mess, and allow us
355 to move forward. The saying "you've got to break a few eggs to make an omelete"
356 is akin, exempting the fact we're providing a way to make the eggs whole again
357 (the king's men would've loved such an option).
358
359 It's advisable that once all old eclasses are no longer in use in the tree, the
360 old eclass package is added to system default. Remember that even those who
361 have upgraded to a portage version that handles the env correctly, may run into
362 instances where an installed packages env is corrupt. For new bootstraps (which
363 automatically upgrade portage right off the bat), an injection of the compat
364 package would be advisable- unless they downgrade portage, they will never need
365 the old eclasses.
366
367
368 Migrating to the new setup
369 --------------------------
370
371 As has been done in the past whenever a change in the tree results in ebuilds
372 requiring a specific version of portage, as ebuilds migrate to the new eclasses,
373 they should depend on a version of portage that supports it. From the users
374 viewpoint, this transparently handles the migration.
375
376 This isn't so transparent for devs or a particular infrastructure server however.
377 Devs, due to them using cvs for their tree, lack the pregenerated cache rsync
378 users have. Devs will have to be early adopters of the new portage. Older
379 portage versions won't be able to access the new eclasses, thus the local cache
380 generation for that ebuild will fail, ergo the depends on a newer portage
381 version won't transparently handle it for them.
382
383 Additionally, prior to any ebuilds in the tree using the new eclasses, the
384 infrastructure server that generates the cache for rsync users will have to
385 either be upgraded to a version of portage supporting new eclasses, or patched.
386 The former being much more preferable then the latter for the portage devs.
387
388 Beyond that, an appropriate window for old eclasses to exist in the tree must be
389 determined, and prior to that window passing an ebuild must be added to the tree
390 so users can get the old eclasses if needed.
391
392 For eclass devs to migrate from old to new, it is possible for them to just
393 transfer the old eclass into an appropriate grouping in the new eclass directory,
394 although it's advisable they cleanse all cruft out of the eclass. You can
395 migrate ebuilds gradually over to the new eclass, and don't have to worry about
396 having to support ebuilds from X years back.
397
398 Essentially, you have a chance to nail the design perfectly/cleanly, and have a
399 window in which to redesign it. It's humbly suggested eclass devs take
400 advantage of it. :)
401
402
403 Backwards Compatibility
404 =======================
405
406 All backwards compatability issues are addressed inline, but a recap is offered-
407 it's suggested that if the a particular compatability issue is
408 questioned/worried over, the reader read the relevant section. There should be
409 a more in depth discussion of the issue, along with a more extensive explanation
410 of the potential solutions, and reasons for the choosen solution.
411
412 To recap:
413 ::
414
415 New eclasses and elib functionality will be tied to a specific portage
416 version. A DEPENDs on said portage version should address this for rsync
417 users who refuse to upgrade to a portage version that supports the new
418 eclasses/elibs and will gradually be unable to merge ebuilds that use said
419 functionality. It is their choice to upgrade, as such, the gradual
420 'thinning' of available ebuilds should they block the portage upgrade is
421 their responsibility.
422
423 Old eclasses at some point in the future should be removed from the tree,
424 and released in a tarball/ebuild. This will cause installed ebuilds that
425 rely on the old eclass to be unable to unmerge to behave as expected, with
426 the same applying for merging of binpkgs.
427
428 This eclass ebuild should be a system depends target to make the transition
429 transparent. Future portage ebuilds, and the old eclass compat ebuild should
430 not inherit any eclasses. The reason for this is that in doing so, it may
431 block upgrade paths. At least for portage, this already is something of a
432 known issue for ebuild functionality- due to what it is/provides, it must
433 essentially be standalone, and cannot benefit from any eclass/elib
434 functionality.
435
436
437 Copyright
438 =========
439
440 This document has been placed in the public domain.

  ViewVC Help
Powered by ViewVC 1.1.20