/[gentoo]/xml/htdocs/proj/en/glep/glep-0058.txt
Gentoo

Diff of /xml/htdocs/proj/en/glep/glep-0058.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.1 Revision 1.8
1GLEP: 58 1GLEP: 58
2Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest 2Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest
3Version: $Revision: 1.1 $ 3Version: $Revision: 1.8 $
4Last-Modified: $Date: 2008/10/21 23:30:47 $ 4Last-Modified: $Date: 2010/02/07 16:24:17 $
5Author: Robin Hugh Johnson <robbat2@gentoo.org>, 5Author: Robin Hugh Johnson <robbat2@gentoo.org>,
6Status: Draft 6Status: Draft
7Type: Standards Track 7Type: Standards Track
8Content-Type: text/x-rst 8Content-Type: text/x-rst
9Requires: GLEP44, GLEP60 9Requires: 44, 60
10Created: October 2006 10Created: October 2006
11Updated: November 2007, June 2008, July 2008, October 2008 11Updated: November 2007, June 2008, July 2008, October 2008, January 2010
12Post-History: ... 12Post-History: December 2009, January 2010
13 13
14======== 14========
15Abstract 15Abstract
16======== 16========
17MetaManifest provides a means of verifiable distribution from Gentoo 17MetaManifest provides a means of verifiable distribution from Gentoo
… …
73located at the root of a repository. 73located at the root of a repository.
74 74
75--------------------------------------------- 75---------------------------------------------
76Procedure for creating the MetaManifest file: 76Procedure for creating the MetaManifest file:
77--------------------------------------------- 77---------------------------------------------
78Summary:
79========
80The objective of creating the MetaManifest file(s) is to ensure that
81every single file in the tree occurs in at least one Manifest.
82
83Process:
84========
781. Start at the root of the Gentoo Portage tree (gentoo-x86, although 851. Start at the root of the Gentoo Portage tree (gentoo-x86, although
79 this procedure applies to overlays as well). 86 this procedure applies to overlays as well).
80 87
812. Initialize two unordered sets: COVERED, ALL. 882. Initialize two unordered sets: COVERED, ALL.
822.1. 'ALL' will contain every file in the tree. 89
90 1. 'ALL' shall contain every file that exists in the present tree.
832.2. 'COVERED' will contain every file that is mentioned in an existing 91 2. 'COVERED' shall contain EVERY file that is mentioned in an existing
84 Manifest2. 92 Manifest2. If a file is mentioned in a Manifest2, but does not
93 exist, it must still be included. No files should be excluded.
85 94
863. Traverse the tree, depth-first. 953. Traverse the tree, depth-first.
96
873.1. At the top level only, ignore the following directories: distfiles, 97 1. At the top level only, ignore the following directories: distfiles,
88 packages, local 98 packages, local.
893.2. If a directory contains a Manifest file, extract all relevant local 99 2. If a directory contains a Manifest file, extract all relevant local
90 files from it (presently: AUX, MISC, EBUILD; but should follow the 100 files from it (presently: AUX, MISC, EBUILD; but should follow the
91 evolution of Manifest2 entry types per [GLEPxx+5]), and place them 101 evolution of Manifest2 entry types per [#GLEP60]), and place them
92 into the COVERED set. 102 into the COVERED set.
933.3. Recursively add every file in the directory to the ALL set, 103 3. Recursively add every file in the directory to the ALL set,
94 pursusant to the exclusion list as mentioned in [GLEPxx+5]. 104 pursuant to the exclusion list as mentioned in [#GLEP60].
95 105
964. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED). 1064. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED).
97 This is every item that is not covered by another Manifest, or part 107 This is every item that is not covered by another Manifest, or part
98 of an exclusion list. 108 of an exclusion list.
99 109
… …
1057. For unique identification of the MetaManifest, a header line should 1157. For unique identification of the MetaManifest, a header line should
106 be included, using the exact contents of the metadata/timestamp.x 116 be included, using the exact contents of the metadata/timestamp.x
107 file, so that a MetaManifest may be tied back to a tree as 117 file, so that a MetaManifest may be tied back to a tree as
108 distributed by the rsync mirror system. The string of 118 distributed by the rsync mirror system. The string of
109 'metadata/timestamp.x' should be included to identify this revision 119 'metadata/timestamp.x' should be included to identify this revision
110 of MetaManifest generation. Eg: 120 of MetaManifest generation. e.g.:
111 "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC" 121 "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC"
112 The package manager MUST not use the identifying string as a filename. 122 The package manager MUST not use the identifying string as a filename.
113 123
1148. The MetaManifest must ultimately be GnuPG-signed. 1248. The MetaManifest must ultimately be GnuPG-signed.
125
1158.1. For the initial implementation, the same key as used for snapshot 126 1. For the initial implementation, the same key as used for snapshot
116 tarball signing is sufficient. 127 tarball signing is sufficient.
1178.2. For the future, the key used for fully automated signing by infra 128 2. For the future, the key used for fully automated signing by infra
118 should not be on the same keyring as developer keys. See [GLEPxx+3 129 should not be on the same keyring as developer keys. See [#GLEPxx+3
119 for further notes]. 130 for further notes].
120 131
132Notes:
133======
121The above does not conflict the proposal contained in GLEP33, which 134The above does not conflict the proposal contained in GLEP33, which
122restructure eclasses to include subdirectories and Manifest files, as 135restructure eclasses to include subdirectories and Manifest files, as
123the Manifest rules above still provide indirect verification for all 136the Manifest rules above still provide indirect verification for all
124files after the GLEP33 restructuring if it comes to pass. 137files after the GLEP33 restructuring if it comes to pass.
125 138
126If other Manifests are added (such as per-category, or protecting 139Additional levels of Manifests are required, such as per-category, and
127versioned eclases), the size of the MetaManifest will be greatly 140in the eclasses, profiles and metadata directories. This ensures that a
128reduced, and this specification was written with such a possible future 141change to a singular file causes the smallest possible overall change in
129addition in mind. 142the Manifests as propagated. Creation of the additional levels of
143Manifests uses the same process as described above, simply starting at a
144different root point.
130 145
131MetaManifest generation will take place as part of the existing process 146MetaManifest generation will take place as part of the existing process
132by infrastructure that takes the contents of CVS and prepares it for 147by infrastructure that takes the contents of CVS and prepares it for
133distribution via rsync, which includes generating metadata. In-tree 148distribution via rsync, which includes generating metadata. In-tree
134Manifest files are not checked at this point, as they are assumed to be 149Manifest files are not validated at this point, as they are assumed to
135correct. 150be correct.
136 151
137-------------------------------------------------------- 152--------------------------------------------------------
138Verification of one or more items from the MetaManifest: 153Verification of one or more items from the MetaManifest:
139-------------------------------------------------------- 154--------------------------------------------------------
140There are two times that this may happen: firstly, immediately after the 155There are two times that this may happen: firstly, immediately after the
… …
147Procedure for verifying an item in the MetaManifest: 162Procedure for verifying an item in the MetaManifest:
148---------------------------------------------------- 163----------------------------------------------------
149In the following, I've used term 'M2-verify' to note following the hash 164In the following, I've used term 'M2-verify' to note following the hash
150verification procedures as defined by the Manifest2 format - which 165verification procedures as defined by the Manifest2 format - which
151compromise checking the file length, and that the hashes match. Which 166compromise checking the file length, and that the hashes match. Which
152filetypes may be ignored on missing is discussed in [GLEPxx+5]. 167filetypes may be ignored on missing is discussed in [#GLEP60].
153 168
1541. Check the GnuPG signature on the MetaManifest against the keyring of 1691. Check the GnuPG signature on the MetaManifest against the keyring of
155 automated Gentoo keys. See [GLEPxx+3] for full details regarding 170 automated Gentoo keys. See [#GLEPxx+3] for full details regarding
156 verification of GnuPG signatures. 171 verification of GnuPG signatures.
1571.1. Abort if the signature check fails. 172 1. Abort if the signature check fails.
158 173
1592. Check the Timestamp header. If it is significently out of date 1742. Check the Timestamp header. If it is significantly out of date
160 compared to the local clock or a trusted source, halt or require 175 compared to the local clock or a trusted source, halt or require
161 manual intervention from the user. 176 manual intervention from the user.
162 177
1633. For a verification of the tree following an rsync: 1783. For a verification of the tree following an rsync:
179
1643.1. Build a set 'ALL' of every file covered by the rsync. (exclude 180 1. Build a set 'ALL' of every file covered by the rsync. (exclude
165 distfiles/, packages/, local/) 181 distfiles/, packages/, local/)
1663.2. M2-verify every entry in the MetaManifest, descending into inferior 182 2. M2-verify every entry in the MetaManifest, descending into inferior
167 Manifests as needed. Place the relative path of every checked item 183 Manifests as needed. Place the relative path of every checked item
168 into a set 'COVERED'. 184 into a set 'COVERED'.
1693.3. Construct the set 'UNCOVERED' by set-difference between the ALL and 185 3. Construct the set 'UNCOVERED' by set-difference between the ALL and
170 COVERED sets. 186 COVERED sets.
1713.4. For each file in the UNCOVERED set, assign a Manifest2 filetype. 187 4. For each file in the UNCOVERED set, assign a Manifest2 filetype.
1723.5. If the filetype for any file in the UNCOVERED set requires a halt 188 5. If the filetype for any file in the UNCOVERED set requires a halt
173 on error, abort and display a suitable error. 189 on error, abort and display a suitable error.
1743.6. Completed verification 190 6. Completed verification
175 191
1764. If checking at the installation of a package: 1924. If checking at the installation of a package:
193
1774.1. M2-verify the entry in MetaManifest for the Manifest 194 1. M2-verify the entry in MetaManifest for the Manifest
1784.2. M2-verify all relevant metadata/ contents if metadata/ is being 195 2. M2-verify all relevant metadata/ contents if metadata/ is being
179 used in any way (optionally done before dependancy checking). 196 used in any way (optionally done before dependency checking).
1804.3. M2-verifying the contents of the Manifest. 197 3. M2-verifying the contents of the Manifest.
1814.4. Perform M2-verification of all eclasses and profiles used (both 198 4. Perform M2-verification of all eclasses and profiles used (both
182 directly and indirectly) by the ebuild. 199 directly and indirectly) by the ebuild.
183 200
184Notes: 201Notes:
185====== 202======
1861. For initial implementations, it is acceptable to check EVERY item in 2031. For initial implementations, it is acceptable to check EVERY item in
187 the eclass and profiles directory, rather than tracking the exact 204 the eclass and profiles directory, rather than tracking the exact
… …
201the individual Gentoo developers. They will continue to develop and 218the individual Gentoo developers. They will continue to develop and
202commit as they do presently, and the MetaManifest is added by 219commit as they do presently, and the MetaManifest is added by
203Infrastructure during the tree generation process, and distributed to 220Infrastructure during the tree generation process, and distributed to
204users. 221users.
205 222
223Any scripts generating Manifests and the MetaManifest may find it useful
224to generate multiple levels of Manifests in parallel, and this is
225explicitly permitted, provided that every file in the tree is covered by
226at least one Manifest or the MetaManifest file. The uppermost
227Manifest (MetaManifest) is the only item that does not occur in any
228other Manifest file, but is instead GPG-signed to enable it's
229validation.
230
206-------------------------------------------- 231--------------------------------------------
207MetaManifest and the new Manifest2 filetypes 232MetaManifest and the new Manifest2 filetypes
208-------------------------------------------- 233--------------------------------------------
209While [GLEPxx+5] describes the addition of new filetypes, these are NOT 234While [#GLEP60] describes the addition of new filetypes, these are NOT
210needed for implementation of the MetaManifest proposal. Without the new 235needed for implementation of the MetaManifest proposal. Without the new
211filetypes, all entries in the MetaManifest would be of type 'MISC'. 236filetypes, all entries in the MetaManifest would be of type 'MISC'.
212 237
213---------------------------------------------------- 238----------------------------------------------------
214Timestamps & Additional distribution of MetaManifest 239Timestamps & Additional distribution of MetaManifest
… …
222trusted channel. 247trusted channel.
223 248
224On all rsync mirrors directly maintained by the Gentoo infrastructure, 249On all rsync mirrors directly maintained by the Gentoo infrastructure,
225and not on community mirrors, there should be a new module 250and not on community mirrors, there should be a new module
226'gentoo-portage-metamanifests'. Within this module, all MetaManifests 251'gentoo-portage-metamanifests'. Within this module, all MetaManifests
227for a recent time frame (eg one week) should be kept, named as 252for a recent time frame (e.g. one week) should be kept, named as
228"MetaManifest.$TS", where $TS is the timestamp from inside the file. 253"MetaManifest.$TS", where $TS is the timestamp from inside the file.
229The most recent MetaManifest should always be symlinked as 254The most recent MetaManifest should always be symlinked as
230MetaManifest.current. The possibility of serving the recent 255MetaManifest.current. The possibility of serving the recent
231MetaManifests via HTTPS should also be explored to mitigate MitM 256MetaManifests via HTTPS should also be explored to mitigate
232attacks. 257man-in-the-middle attacks.
233 258
234The package manager should obtain MetaManifest.current and use it to 259The package manager should obtain MetaManifest.current and use it to
235decide is the tree is too out of date per operation #2 of the 260decide is the tree is too out of date per operation #2 of the
236verification process. The decision about freshness should be a 261verification process. The decision about freshness should be a
237user-configuration setting, with the ability to override. 262user-configuration setting, with the ability to override.
238 263
239-------------------------------- 264--------------------------------
240MetaManifest size considerations 265MetaManifest size considerations
241-------------------------------- 266--------------------------------
242With only two levels of Manifests (per-package and top-level), every 267With only two levels of Manifests (per-package and top-level), every
243rsync will cause a lot of traffic transfering the modified top-level 268rsync will cause a lot of traffic transferring the modified top-level
244MetaManifest. To reduce this, per-category Manifests are strongly 269MetaManifest. To reduce this, first-level directory Manifests are
245recommended. Alternatively, if the distribution method efficently 270required. Alternatively, if the distribution method efficiently handles
246handles small patch-like changes in an existing file, using an 271small patch-like changes in an existing file, using an uncompressed
247uncompressed MetaManifest may be acceptable (this would primarily be 272MetaManifest may be acceptable (this would primarily be distributed
248distributed version control systems). Other suggestions in reducing this 273version control systems). Other suggestions in reducing this traffic are
249traffic are welcomed. 274welcomed.
250 275
251======================= 276=======================
252Backwards Compatibility 277Backwards Compatibility
253======================= 278=======================
254- There are no backwards compatibility issues, as old versions of 279- There are no backwards compatibility issues, as old versions of
… …
259 284
260====== 285======
261Thanks 286Thanks
262====== 287======
263I'd like to thank the following people for input on this GLEP. 288I'd like to thank the following people for input on this GLEP.
289
264- Patrick Lauer (patrick): Prodding me to get all of the tree-signing 290- Patrick Lauer (patrick): Prodding me to get all of the tree-signing
265 work finished, and helping to edit. 291 work finished, and helping to edit.
266- Ciaran McCreesh (ciaranm): Paludis Manifest2 292- Ciaran McCreesh (ciaranm): Paludis Manifest2
267- Brian Harring (ferringb): pkgcore Manifest2 293- Brian Harring (ferringb): pkgcore Manifest2
268- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2 294- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
… …
276 University of Arizona Technical Report TR08-02. Available online 302 University of Arizona Technical Report TR08-02. Available online
277 from: ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf 303 from: ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf
278[C08b] Cappos, J et al. (2008). "Attacks on Package Managers" 304[C08b] Cappos, J et al. (2008). "Attacks on Package Managers"
279 Available online at: 305 Available online at:
280 http://www.cs.arizona.edu/people/justin/packagemanagersecurity/ 306 http://www.cs.arizona.edu/people/justin/packagemanagersecurity/
307[#GLEPxx+2] Future GLEP on Developer Process security.
308[#GLEPxx+3] Future GLEP on GnuPG Policies and Handling.
281 309
282========= 310=========
283Copyright 311Copyright
284========= 312=========
285Copyright (c) 2006 by Robin Hugh Johnson. This material may be 313Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be
286distributed only subject to the terms and conditions set forth in the 314distributed only subject to the terms and conditions set forth in the
287Open Publication License, v1.0. 315Open Publication License, v1.0.
288 316
289vim: tw=72 ts=2 expandtab: 317vim: tw=72 ts=2 expandtab:
Legend:
Removed from v.1.1  
changed lines
  Added in v.1.8
  ViewVC Help
Powered by ViewVC 1.1.20