en/glep/glep-0060.txt

GLEP: 60
Title: Manifest2 filetypes
Version: $Revision: 1.1 $
Last-Modified: $Date: 2008/10/21 23:30:47 $
Author: Robin Hugh Johnson <robbat2@gentoo.org> 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 44
Created: November 2007
Updated: June 2008, July 2008
Updates: 44

Abstract
========
Clarification of the Manifest2 [GLEP44] specification, including new types to
help in the tree-signing specification.

Motivation
==========
[GLEP44] was not entirely clear on the usage of filetype specifiers.
This document serves to provide some of the internal logic used by
Portage at the point of writing, as well as adding new types to cover
the rest of the tree, for the purposes of tree-signing coverage.

Specification
=============
General
-------
For any given directory with a Manifest file, every file located in that
directory, or a sub-directory must be listed in that Manifest file,
unless stated otherwise in the following sections. The Manifest file
must not contain an entry for itself.

Excluded files
--------------
When generating or validating a Manifest, or commiting to a version
control system, the package manager should endeavour to ignore files
created by a version control system, backup files from text editors. A
non-exhaustive list is suggested here: CVS/, .svn/, .bzr/, .git/, .hg/,
.#*, *.rej, *.orig, *.bak, *~.

Additionally, for a transitional Manifest1->Manifest2 system, old-style
digest files located in a 'files/' directory, may be excluded from
Manifest2 generation, or included with a type of MISC.

Under strict security conditions, the exclusion list may be ignored
during validation if the existence of a file would be considered a
security risk.

Existing filetypes:
-------------------
AUX
~~~
- The AUX type is used for all items under the 'files' subdirectory. 
- They should be verified relative to $FILESDIR. 
- The string 'files/' is left out of the Manifest line. 
- The absence of a file mentioned by AUX must be treated as an error. 
- The AUX type is intended to denote potentially executable content
  (either directly or indirectly), that must be treated an error if
  modified or absent.

EBUILD
~~~~~~
- The EBUILD type is used solely for files ending in .ebuild, or other
  suffixes as defined by the EAPI.
- The files are located in the same directory as the Manifest file. 
- The modification or absence of a file mentioned by EBUILD must be
  treated as an error.

DIST
~~~~
- The DIST type is used for distfiles
- They may be found directly via the $DISTDIR setting of the package
  manager. 
- During simple verification of a Manifest, a missing DIST file should
  not be consider as a validation error (it is however a failure to
  fetch or unpack).

MISC
~~~~
- The MISC type covers all remaining files in a directory. 
- MISC is intended to mark all content that was not used in
  some way that directly affected execution of the package manager. 
- This includes metadata.xml and ChangeLog entries, and any other purely
  informational content.
- MISC entries where the file is missing may optionally be ignored as by
  non-strict package managers.
- It should be possible to install a package while all MISC entries have
  been deleted from the tree.


New filetypes:
--------------
_INFO (new, abstract)
~~~~~~~~~~~~~~~~~~~~~
- This is the functionality of the old AUX, but does not include the
  implicit 'files/' prefix in the path, and is verified relative to the
  working directory instead of $FILESDIR.
- The modification or absence of a file listed as a _INFO-derived type 
  is not an error unless the package manager is attempting to be strict.

_CRIT (new, abstract)
~~~~~~~~~~~~~~~~~~~~~
- _CRIT is based off the _INFO type.
- The modification or absence of a file listed as a _CRIT-derived type 
  must be treated as an error.

EBUILD
~~~~~~
- Now derived from _CRIT.
- Otherwise unchanged.

DIST
~~~~
- Now derived from _CRIT.
- Otherwise unchanged.

MISC
~~~~
- Now derived from _INFO.
- Otherwise unchanged.

MANIFEST (new)
~~~~~~~~~~~~~~
- The MANIFEST type is explicitly to cover all nested Manifest files.
- During validation, this serves as an indicator that the package
  manager may need to check subtree Manifest file. 
- A missing MANIFEST file may be treated as a minor (eg excluding an
  entire category) or critical validation failure.
- The failure should be considered as critical only if files that would
  be directly covered by this Manifest are missing. Deletion of a
  category-level Manifest while preserving the packages is forbidden.
  Deletion of an entire category is not.

ECLASS (new)
~~~~~~~~~~~~
- uses _CRIT.
- This type shall be used for all eclasses only.
- TODO: What about patches etc under eclasses/? Probably EXEC?

DATA (new)
~~~~~~~~~~
- uses _CRIT.
- The DATA type shall be used for all files that directly affect the
  package manager, such as metadata/cache/* and profiles/.

EXEC (new)
~~~~~~~~~~
- uses _CRIT.
- If the file gets sourced, executed, or causes a change (patches) in
  how something is sourced or execututed, it belongs in the EXEC
  filetype.
- This filetype should be used for the scripts directories of a
  repository for important files.

UNKNOWN (new)
~~~~~~~~~~~~~
- uses _CRIT.
- All other files that are not covered by another type should be
  considered as 'UNKNOWN'.

On Bloat
--------
If repeated use of a common path prefix is considered a bloat problem, a
Manifest file should be added inside the common directory, however this
should not be done blindly, as bloat by inodes is more significant for
the majority of use cases. 

Chosing a filetype
------------------
1. matches Manifest 
    => MANIFEST, stop.
2. matches *.ebuild 
    => EBUILD, stop.
3. matches *.eclass 
    => ECLASS, stop.
4. listed in SRC_URI 
    => DIST, stop.
5. matches files/* 
    => AUX, continue [see note].
6. matches {*.sh,*.bashrc,*.patch,...} 
    => EXEC, stop.
7. matches {metadata/cache/*,profiles/,package.*,use.mask*,...} 
    => DATA, stop.
8. matches {ChangeLog,metadata.xml,*.desc,...} 
    => MISC, stop.
9. not matched by any other rule 
    => UNKNOWN, stop.

The logic behind 5, 6, 7 is ensuring that every item that by it's
presence or absense may be dangerous should always be treated strictly.
(Consider epatch given a directory of patches ${FILESDIR}/${PV}/, where
it blindly includes them, or alternatively, the package.mask file or a
profile being altered/missing).

Note: The AUX entries should only be generated if we are generating a
compatible Manifest that supports older versions of Portage. They should
be generated along with the new type.

Backwards Compatibility
=======================
For generation of existing package Manifests, the AUX entries must
continue to be present for the standard Portage deprecation cycle.
The new entries may be included already in all Manifest files, as they
will be ignored by older Portage versions. Over time, ECLASS, DATA,
EXEC, UNKNOWN may replace the existing AUX type.

The adoption of this proposal does also affect [GLEPxx+1] as part of
this GLEP series, however this GLEP was an offset of the research in
that GLEP.

Thanks to
=========
I'd like to thank the following people for input on this GLEP.
- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2

References
==========
.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
  http://www.gentoo.org/proj/en/glep/glep-0044.html     

Copyright
=========
Copyright (c) 2007 by Robin Hugh Johnson. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0.

vim: tw=72 ts=2 expandtab:
1	cardoe	1.1	GLEP: 60
2			Title: Manifest2 filetypes
3	robbat2	1.2	Version: $Revision: 1.1 $
4			Last-Modified: $Date: 2008/10/21 23:30:47 $
5	cardoe	1.1	Author: Robin Hugh Johnson <robbat2@gentoo.org>
6			Status: Draft
7			Type: Standards Track
8			Content-Type: text/x-rst
9			Requires: 44
10			Created: November 2007
11			Updated: June 2008, July 2008
12			Updates: 44
13
14			Abstract
15			========
16			Clarification of the Manifest2 [GLEP44] specification, including new types to
17			help in the tree-signing specification.
18
19			Motivation
20			==========
21			[GLEP44] was not entirely clear on the usage of filetype specifiers.
22			This document serves to provide some of the internal logic used by
23			Portage at the point of writing, as well as adding new types to cover
24			the rest of the tree, for the purposes of tree-signing coverage.
25
26			Specification
27			=============
28			General
29			-------
30			For any given directory with a Manifest file, every file located in that
31			directory, or a sub-directory must be listed in that Manifest file,
32			unless stated otherwise in the following sections. The Manifest file
33			must not contain an entry for itself.
34
35			Excluded files
36			--------------
37			When generating or validating a Manifest, or commiting to a version
38			control system, the package manager should endeavour to ignore files
39			created by a version control system, backup files from text editors. A
40			non-exhaustive list is suggested here: CVS/, .svn/, .bzr/, .git/, .hg/,
41			.#, .rej, .orig, .bak, *~.
42
43			Additionally, for a transitional Manifest1->Manifest2 system, old-style
44			digest files located in a 'files/' directory, may be excluded from
45			Manifest2 generation, or included with a type of MISC.
46
47			Under strict security conditions, the exclusion list may be ignored
48			during validation if the existence of a file would be considered a
49			security risk.
50
51			Existing filetypes:
52			-------------------
53			AUX
54			~~~
55			- The AUX type is used for all items under the 'files' subdirectory.
56			- They should be verified relative to $FILESDIR.
57			- The string 'files/' is left out of the Manifest line.
58			- The absence of a file mentioned by AUX must be treated as an error.
59			- The AUX type is intended to denote potentially executable content
60			(either directly or indirectly), that must be treated an error if
61			modified or absent.
62
63			EBUILD
64			~~~~~~
65			- The EBUILD type is used solely for files ending in .ebuild, or other
66			suffixes as defined by the EAPI.
67			- The files are located in the same directory as the Manifest file.
68			- The modification or absence of a file mentioned by EBUILD must be
69			treated as an error.
70
71			DIST
72			~~~~
73			- The DIST type is used for distfiles
74			- They may be found directly via the $DISTDIR setting of the package
75			manager.
76			- During simple verification of a Manifest, a missing DIST file should
77			not be consider as a validation error (it is however a failure to
78			fetch or unpack).
79
80			MISC
81			~~~~
82			- The MISC type covers all remaining files in a directory.
83			- MISC is intended to mark all content that was not used in
84			some way that directly affected execution of the package manager.
85			- This includes metadata.xml and ChangeLog entries, and any other purely
86			informational content.
87			- MISC entries where the file is missing may optionally be ignored as by
88			non-strict package managers.
89			- It should be possible to install a package while all MISC entries have
90			been deleted from the tree.
91
92
93			New filetypes:
94			--------------
95			_INFO (new, abstract)
96	robbat2	1.2	~~~~~~~~~~~~~~~~~~~~~
97	cardoe	1.1	- This is the functionality of the old AUX, but does not include the
98			implicit 'files/' prefix in the path, and is verified relative to the
99			working directory instead of $FILESDIR.
100			- The modification or absence of a file listed as a _INFO-derived type
101			is not an error unless the package manager is attempting to be strict.
102
103			_CRIT (new, abstract)
104	robbat2	1.2	~~~~~~~~~~~~~~~~~~~~~
105	cardoe	1.1	- _CRIT is based off the _INFO type.
106			- The modification or absence of a file listed as a _CRIT-derived type
107			must be treated as an error.
108
109			EBUILD
110			~~~~~~
111			- Now derived from _CRIT.
112			- Otherwise unchanged.
113
114			DIST
115			~~~~
116			- Now derived from _CRIT.
117			- Otherwise unchanged.
118
119			MISC
120			~~~~
121			- Now derived from _INFO.
122			- Otherwise unchanged.
123
124			MANIFEST (new)
125			~~~~~~~~~~~~~~
126			- The MANIFEST type is explicitly to cover all nested Manifest files.
127			- During validation, this serves as an indicator that the package
128			manager may need to check subtree Manifest file.
129			- A missing MANIFEST file may be treated as a minor (eg excluding an
130			entire category) or critical validation failure.
131			- The failure should be considered as critical only if files that would
132			be directly covered by this Manifest are missing. Deletion of a
133			category-level Manifest while preserving the packages is forbidden.
134			Deletion of an entire category is not.
135
136			ECLASS (new)
137			~~~~~~~~~~~~
138			- uses _CRIT.
139			- This type shall be used for all eclasses only.
140			- TODO: What about patches etc under eclasses/? Probably EXEC?
141
142			DATA (new)
143			~~~~~~~~~~
144			- uses _CRIT.
145			- The DATA type shall be used for all files that directly affect the
146			package manager, such as metadata/cache/* and profiles/.
147
148			EXEC (new)
149			~~~~~~~~~~
150			- uses _CRIT.
151			- If the file gets sourced, executed, or causes a change (patches) in
152			how something is sourced or execututed, it belongs in the EXEC
153			filetype.
154			- This filetype should be used for the scripts directories of a
155			repository for important files.
156
157			UNKNOWN (new)
158			~~~~~~~~~~~~~
159			- uses _CRIT.
160			- All other files that are not covered by another type should be
161			considered as 'UNKNOWN'.
162
163			On Bloat
164			--------
165			If repeated use of a common path prefix is considered a bloat problem, a
166			Manifest file should be added inside the common directory, however this
167			should not be done blindly, as bloat by inodes is more significant for
168			the majority of use cases.
169
170			Chosing a filetype
171			------------------
172			1. matches Manifest
173			=> MANIFEST, stop.
174			2. matches *.ebuild
175			=> EBUILD, stop.
176			3. matches *.eclass
177			=> ECLASS, stop.
178			4. listed in SRC_URI
179			=> DIST, stop.
180			5. matches files/*
181			=> AUX, continue [see note].
182			6. matches {.sh,.bashrc,*.patch,...}
183			=> EXEC, stop.
184			7. matches {metadata/cache/,profiles/,package.,use.mask*,...}
185			=> DATA, stop.
186			8. matches {ChangeLog,metadata.xml,*.desc,...}
187			=> MISC, stop.
188			9. not matched by any other rule
189			=> UNKNOWN, stop.
190
191			The logic behind 5, 6, 7 is ensuring that every item that by it's
192			presence or absense may be dangerous should always be treated strictly.
193			(Consider epatch given a directory of patches ${FILESDIR}/${PV}/, where
194			it blindly includes them, or alternatively, the package.mask file or a
195			profile being altered/missing).
196
197			Note: The AUX entries should only be generated if we are generating a
198			compatible Manifest that supports older versions of Portage. They should
199			be generated along with the new type.
200
201			Backwards Compatibility
202			=======================
203			For generation of existing package Manifests, the AUX entries must
204			continue to be present for the standard Portage deprecation cycle.
205			The new entries may be included already in all Manifest files, as they
206			will be ignored by older Portage versions. Over time, ECLASS, DATA,
207			EXEC, UNKNOWN may replace the existing AUX type.
208
209			The adoption of this proposal does also affect [GLEPxx+1] as part of
210			this GLEP series, however this GLEP was an offset of the research in
211			that GLEP.
212
213			Thanks to
214			=========
215			I'd like to thank the following people for input on this GLEP.
216			- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
217
218			References
219			==========
220			.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
221			http://www.gentoo.org/proj/en/glep/glep-0044.html
222
223			Copyright
224			=========
225			Copyright (c) 2007 by Robin Hugh Johnson. This material may be
226			distributed only subject to the terms and conditions set forth in the
227			Open Publication License, v1.0.
228
229			vim: tw=72 ts=2 expandtab: