en/glep/glep-0060.txt

GLEP: 60
Title: Manifest2 filetypes
Version: $Revision: 1.7 $
Last-Modified: $Date: 2010/01/13 03:26:53 $
Author: Robin Hugh Johnson <robbat2@gentoo.org> 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 44
Created: November 2007
Updated: June 2008, July 2008, October 2008, January 2010
Updates: 44
Post-History: December 2009, January 2010

Abstract
========
Clarification of the Manifest2 [#GLEP44] specification, including new types to
help in the tree-signing specification.

Motivation
==========
[#GLEP44] was not entirely clear on the usage of filetype specifiers.
This document serves to provide some of the internal logic used by
Portage at the point of writing, as well as adding new types to cover
the rest of the tree, for the purposes of tree-signing coverage.

This GLEP is not mandatory for the tree-signing specification, but
instead aims to clarify the usage of the Manifest2 filetype specifiers,
and note which types signify files that are allowed to be missing from
the tree (e.g. a user excluding a package or category).

Specification
=============
General
-------
For any given directory with a Manifest file, every file located in that
directory, or a sub-directory must be listed in that Manifest file,
unless stated otherwise in the following sections. The Manifest file
must not contain an entry for itself.

Excluded files
--------------
When generating or validating a Manifest, or committing to a version
control system, the package manager should endeavour to ignore files
created by a version control system, backup files from text editors. A
non-exhaustive list is suggested here: ``CVS/``, ``.svn/``, ``.bzr/``,
``.git/``, ``.hg/``, ``.#*``, ``*.rej``, ``*.orig``, ``*.bak``, ``*~``.

Additionally, for a transitional Manifest1->Manifest2 system, old-style
digest files located in a 'files/' directory, may be excluded from
Manifest2 generation, or included with a type of MISC.

Under strict security conditions, the exclusion list may be ignored
during validation if the existence of a file would be considered a
security risk.

Existing filetypes:
-------------------
AUX
~~~
- The AUX type is used for all items under the 'files' subdirectory. 
- They should be verified relative to $FILESDIR. 
- The string 'files/' is left out of the Manifest line. 
- The absence of a file mentioned by AUX must be treated as an error. 
- The AUX type is intended to denote potentially executable content
  (either directly or indirectly), that must be treated an error if
  modified or absent.

EBUILD
~~~~~~
- The EBUILD type is used solely for files ending in .ebuild, or other
  suffixes as defined by the EAPI.
- The files are located in the same directory as the Manifest file. 
- The modification or absence of a file mentioned by EBUILD must be
  treated as an error.

DIST
~~~~
- The DIST type is used for distfiles
- They may be found directly via the $DISTDIR setting of the package
  manager. 
- During simple verification of a Manifest, a missing DIST file should
  not be consider as a validation error (it is however a failure to
  fetch or unpack).

MISC
~~~~
- The MISC type covers all remaining files in a directory. 
- MISC is intended to mark all content that was not used in
  some way that directly affected execution of the package manager. 
- This includes metadata.xml and ChangeLog entries, and any other purely
  informational content.
- MISC entries where the file is missing may optionally be ignored as by
  non-strict package managers.
- It should be possible to install a package while all MISC entries have
  been deleted from the tree.


New filetypes:
--------------
_INFO (new, abstract)
~~~~~~~~~~~~~~~~~~~~~
- This is the functionality of the old AUX, but does not include the
  implicit 'files/' prefix in the path, and is verified relative to the
  working directory instead of $FILESDIR.
- The modification or absence of a file listed as a _INFO-derived type 
  is not an error unless the package manager is attempting to be strict.

_CRIT (new, abstract)
~~~~~~~~~~~~~~~~~~~~~
- _CRIT is based off the _INFO type.
- The modification or absence of a file listed as a _CRIT-derived type 
  MUST be treated as an error.

EBUILD
~~~~~~
- Now derived from _CRIT.
- Otherwise unchanged.

DIST
~~~~
- Now derived from _CRIT.
- Otherwise unchanged.

MISC
~~~~
- Now derived from _INFO.
- Otherwise unchanged.

MANIFEST (new)
~~~~~~~~~~~~~~
- The MANIFEST type is explicitly to cover all nested Manifest files.
- During validation, this serves as an indicator that the package
  manager may need to check subtree Manifest file. 
- A missing MANIFEST file may be treated as a minor (e.g. excluding an
  entire category) or critical validation failure.
- The failure should be considered as critical only if files that would
  be directly covered by this Manifest are missing. Deletion of a
  category-level Manifest while preserving the packages is forbidden.
  Deletion of an entire category is not.

ECLASS (new)
~~~~~~~~~~~~
- uses _CRIT.
- This type shall be used for all eclasses only.

DATA (new)
~~~~~~~~~~
- uses _CRIT.
- The DATA type shall be used for all files that directly affect the
  package manager, such as metadata/cache/* and profiles/.

EXEC (new)
~~~~~~~~~~
- uses _CRIT.
- If the file gets sourced, executed, or causes a change (patches) in
  how something is sourced or executed, it belongs in the EXEC
  filetype.
- This filetype should be used for the scripts directories of a
  repository for important files.
- This filetype is not limited to being used in the files/
  subdirectory.

OTHER (new)
~~~~~~~~~~~
- uses _CRIT.
- All other files that are not covered by another type should be
  considered as 'OTHER'.
- Any further new filetypes should be introduced to subtract files
  from the 'OTHER' set.
- If a package manager runs into a unknown Manifest2 type, it should
  be treated as 'OTHER'.

On Bloat
--------
If repeated use of a common path prefix is considered a bloat problem, a
Manifest file should be added inside the common directory, however this
should not be done blindly, as bloat by inodes is more significant for
the majority of use cases. See also [#GLEP58] on size reductions of
Manifests.

Chosing a filetype
------------------
1. matches ``Manifest``
    => MANIFEST, stop.
2. matches ``*.ebuild``
    => EBUILD, stop.
3. matches ``*.eclass``
    => ECLASS, stop.
4. listed in SRC_URI 
    => DIST, stop.
5. matches ``files/*``
    => AUX, continue [see note].
6. matches any of ``*.sh``, ``*.bashrc``, ``*.patch``, ...
    => EXEC, stop.
7. matches any of ``metadata/cache/*``, ``profiles/``, ``package.*``, ``use.mask*``, ...
    => DATA, stop.
8. matches any of ``ChangeLog``, ``metadata.xml``, ``*.desc``, ...
    => MISC, stop.
9. not matched by any other rule 
    => OTHER, stop.

The logic behind 5, 6, 7 is ensuring that every item that by it's
presence or absence may be dangerous should always be treated strictly.
(Consider epatch given a directory of patches ``${FILESDIR}/${PV}/``,
where it blindly includes them, or alternatively, the package.mask file
or a profile being altered/missing).

The above lists of file patterns are not intended to be exhaustive,
but merely demonstrative.

Note: The AUX entries should only be generated if we are generating a
compatible Manifest that supports older versions of Portage. They should
be generated along with the new type.

Backwards Compatibility
=======================
For generation of existing package Manifests, the AUX entries must
continue to be present for the standard Portage deprecation cycle.
The new entries may be included already in all Manifest files, as they
will be ignored by older Portage versions. Over time, ECLASS, DATA,
EXEC, OTHER may replace the existing AUX type.

The adoption of this proposal does also affect [#GLEP58] as part of
this GLEP series, however this GLEP was an offset of the research in
that GLEP.

Thanks to
=========
I'd like to thank the following people for input on this GLEP.
- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2

References
==========
.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
  http://www.gentoo.org/proj/en/glep/glep-0044.html     

Copyright
=========
Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0.

vim: tw=72 ts=2 expandtab:
1	cardoe	1.1	GLEP: 60
2			Title: Manifest2 filetypes
3	robbat2	1.8	Version: $Revision: 1.7 $
4			Last-Modified: $Date: 2010/01/13 03:26:53 $
5	cardoe	1.1	Author: Robin Hugh Johnson <robbat2@gentoo.org>
6			Status: Draft
7			Type: Standards Track
8			Content-Type: text/x-rst
9			Requires: 44
10			Created: November 2007
11	robbat2	1.6	Updated: June 2008, July 2008, October 2008, January 2010
12	cardoe	1.1	Updates: 44
13	robbat2	1.8	Post-History: December 2009, January 2010
14	cardoe	1.1
15			Abstract
16			========
17	robbat2	1.3	Clarification of the Manifest2 [#GLEP44] specification, including new types to
18	cardoe	1.1	help in the tree-signing specification.
19
20			Motivation
21			==========
22	robbat2	1.3	[#GLEP44] was not entirely clear on the usage of filetype specifiers.
23	cardoe	1.1	This document serves to provide some of the internal logic used by
24			Portage at the point of writing, as well as adding new types to cover
25			the rest of the tree, for the purposes of tree-signing coverage.
26
27	robbat2	1.8	This GLEP is not mandatory for the tree-signing specification, but
28			instead aims to clarify the usage of the Manifest2 filetype specifiers,
29			and note which types signify files that are allowed to be missing from
30			the tree (e.g. a user excluding a package or category).
31
32	cardoe	1.1	Specification
33			=============
34			General
35			-------
36			For any given directory with a Manifest file, every file located in that
37			directory, or a sub-directory must be listed in that Manifest file,
38			unless stated otherwise in the following sections. The Manifest file
39			must not contain an entry for itself.
40
41			Excluded files
42			--------------
43	robbat2	1.7	When generating or validating a Manifest, or committing to a version
44	cardoe	1.1	control system, the package manager should endeavour to ignore files
45			created by a version control system, backup files from text editors. A
46	robbat2	1.3	non-exhaustive list is suggested here: ``CVS/``, ``.svn/``, ``.bzr/``,
47			``.git/``, ``.hg/``, ``.#``, ``.rej``, ``.orig``, ``.bak``, ``*~``.
48	cardoe	1.1
49			Additionally, for a transitional Manifest1->Manifest2 system, old-style
50			digest files located in a 'files/' directory, may be excluded from
51			Manifest2 generation, or included with a type of MISC.
52
53			Under strict security conditions, the exclusion list may be ignored
54			during validation if the existence of a file would be considered a
55			security risk.
56
57			Existing filetypes:
58			-------------------
59			AUX
60			~~~
61			- The AUX type is used for all items under the 'files' subdirectory.
62			- They should be verified relative to $FILESDIR.
63			- The string 'files/' is left out of the Manifest line.
64			- The absence of a file mentioned by AUX must be treated as an error.
65			- The AUX type is intended to denote potentially executable content
66			(either directly or indirectly), that must be treated an error if
67			modified or absent.
68
69			EBUILD
70			~~~~~~
71			- The EBUILD type is used solely for files ending in .ebuild, or other
72			suffixes as defined by the EAPI.
73			- The files are located in the same directory as the Manifest file.
74			- The modification or absence of a file mentioned by EBUILD must be
75			treated as an error.
76
77			DIST
78			~~~~
79			- The DIST type is used for distfiles
80			- They may be found directly via the $DISTDIR setting of the package
81			manager.
82			- During simple verification of a Manifest, a missing DIST file should
83			not be consider as a validation error (it is however a failure to
84			fetch or unpack).
85
86			MISC
87			~~~~
88			- The MISC type covers all remaining files in a directory.
89			- MISC is intended to mark all content that was not used in
90			some way that directly affected execution of the package manager.
91			- This includes metadata.xml and ChangeLog entries, and any other purely
92			informational content.
93			- MISC entries where the file is missing may optionally be ignored as by
94			non-strict package managers.
95			- It should be possible to install a package while all MISC entries have
96			been deleted from the tree.
97
98
99			New filetypes:
100			--------------
101			_INFO (new, abstract)
102	robbat2	1.2	~~~~~~~~~~~~~~~~~~~~~
103	cardoe	1.1	- This is the functionality of the old AUX, but does not include the
104			implicit 'files/' prefix in the path, and is verified relative to the
105			working directory instead of $FILESDIR.
106			- The modification or absence of a file listed as a _INFO-derived type
107			is not an error unless the package manager is attempting to be strict.
108
109			_CRIT (new, abstract)
110	robbat2	1.2	~~~~~~~~~~~~~~~~~~~~~
111	cardoe	1.1	- _CRIT is based off the _INFO type.
112			- The modification or absence of a file listed as a _CRIT-derived type
113	robbat2	1.5	MUST be treated as an error.
114	cardoe	1.1
115			EBUILD
116			~~~~~~
117			- Now derived from _CRIT.
118			- Otherwise unchanged.
119
120			DIST
121			~~~~
122			- Now derived from _CRIT.
123			- Otherwise unchanged.
124
125			MISC
126			~~~~
127			- Now derived from _INFO.
128			- Otherwise unchanged.
129
130			MANIFEST (new)
131			~~~~~~~~~~~~~~
132			- The MANIFEST type is explicitly to cover all nested Manifest files.
133			- During validation, this serves as an indicator that the package
134			manager may need to check subtree Manifest file.
135	robbat2	1.7	- A missing MANIFEST file may be treated as a minor (e.g. excluding an
136	cardoe	1.1	entire category) or critical validation failure.
137			- The failure should be considered as critical only if files that would
138			be directly covered by this Manifest are missing. Deletion of a
139			category-level Manifest while preserving the packages is forbidden.
140			Deletion of an entire category is not.
141
142			ECLASS (new)
143			~~~~~~~~~~~~
144			- uses _CRIT.
145			- This type shall be used for all eclasses only.
146
147			DATA (new)
148			~~~~~~~~~~
149			- uses _CRIT.
150			- The DATA type shall be used for all files that directly affect the
151			package manager, such as metadata/cache/* and profiles/.
152
153			EXEC (new)
154			~~~~~~~~~~
155			- uses _CRIT.
156			- If the file gets sourced, executed, or causes a change (patches) in
157	robbat2	1.7	how something is sourced or executed, it belongs in the EXEC
158	cardoe	1.1	filetype.
159			- This filetype should be used for the scripts directories of a
160			repository for important files.
161	robbat2	1.5	- This filetype is not limited to being used in the files/
162			subdirectory.
163	cardoe	1.1
164	robbat2	1.4	OTHER (new)
165			~~~~~~~~~~~
166	cardoe	1.1	- uses _CRIT.
167			- All other files that are not covered by another type should be
168	robbat2	1.4	considered as 'OTHER'.
169	robbat2	1.5	- Any further new filetypes should be introduced to subtract files
170			from the 'OTHER' set.
171			- If a package manager runs into a unknown Manifest2 type, it should
172			be treated as 'OTHER'.
173	cardoe	1.1
174			On Bloat
175			--------
176			If repeated use of a common path prefix is considered a bloat problem, a
177			Manifest file should be added inside the common directory, however this
178			should not be done blindly, as bloat by inodes is more significant for
179	robbat2	1.5	the majority of use cases. See also [#GLEP58] on size reductions of
180			Manifests.
181	cardoe	1.1
182			Chosing a filetype
183			------------------
184	robbat2	1.5	1. matches ``Manifest``
185	cardoe	1.1	=> MANIFEST, stop.
186	robbat2	1.3	2. matches ``*.ebuild``
187	cardoe	1.1	=> EBUILD, stop.
188	robbat2	1.3	3. matches ``*.eclass``
189	cardoe	1.1	=> ECLASS, stop.
190			4. listed in SRC_URI
191			=> DIST, stop.
192	robbat2	1.3	5. matches ``files/*``
193	cardoe	1.1	=> AUX, continue [see note].
194	robbat2	1.3	6. matches any of ``.sh``, ``.bashrc``, ``*.patch``, ...
195	cardoe	1.1	=> EXEC, stop.
196	robbat2	1.3	7. matches any of ``metadata/cache/``, ``profiles/``, ``package.``, ``use.mask*``, ...
197	cardoe	1.1	=> DATA, stop.
198	robbat2	1.3	8. matches any of ``ChangeLog``, ``metadata.xml``, ``*.desc``, ...
199	cardoe	1.1	=> MISC, stop.
200			9. not matched by any other rule
201	robbat2	1.4	=> OTHER, stop.
202	cardoe	1.1
203			The logic behind 5, 6, 7 is ensuring that every item that by it's
204	robbat2	1.7	presence or absence may be dangerous should always be treated strictly.
205	robbat2	1.3	(Consider epatch given a directory of patches ``${FILESDIR}/${PV}/``,
206			where it blindly includes them, or alternatively, the package.mask file
207			or a profile being altered/missing).
208	cardoe	1.1
209	robbat2	1.5	The above lists of file patterns are not intended to be exhaustive,
210			but merely demonstrative.
211
212	cardoe	1.1	Note: The AUX entries should only be generated if we are generating a
213			compatible Manifest that supports older versions of Portage. They should
214			be generated along with the new type.
215
216			Backwards Compatibility
217			=======================
218			For generation of existing package Manifests, the AUX entries must
219			continue to be present for the standard Portage deprecation cycle.
220			The new entries may be included already in all Manifest files, as they
221			will be ignored by older Portage versions. Over time, ECLASS, DATA,
222	robbat2	1.4	EXEC, OTHER may replace the existing AUX type.
223	cardoe	1.1
224	robbat2	1.3	The adoption of this proposal does also affect [#GLEP58] as part of
225	cardoe	1.1	this GLEP series, however this GLEP was an offset of the research in
226			that GLEP.
227
228			Thanks to
229			=========
230			I'd like to thank the following people for input on this GLEP.
231			- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
232
233			References
234			==========
235			.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
236			http://www.gentoo.org/proj/en/glep/glep-0044.html
237
238			Copyright
239			=========
240	robbat2	1.7	Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be
241	cardoe	1.1	distributed only subject to the terms and conditions set forth in the
242			Open Publication License, v1.0.
243
244			vim: tw=72 ts=2 expandtab: