Contents of /xml/htdocs/proj/en/glep/glep-0058.txt

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.8 - (hide annotations) (download)
Sun Feb 7 16:24:17 2010 UTC (8 years, 11 months ago) by robbat2
Branch: MAIN
Changes since 1.7: +4 -2 lines
File MIME type: text/plain
Fix up missing references to the unfinished tree-signing gleps that handle the GnuPG side and the developer process side.

1 cardoe 1.1 GLEP: 58
2     Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest
3 robbat2 1.8 Version: $Revision: 1.7 $
4     Last-Modified: $Date: 2010/01/31 07:53:30 $
5 cardoe 1.1 Author: Robin Hugh Johnson <robbat2@gentoo.org>,
6     Status: Draft
7     Type: Standards Track
8     Content-Type: text/x-rst
9 robbat2 1.4 Requires: 44, 60
10 cardoe 1.1 Created: October 2006
11 robbat2 1.5 Updated: November 2007, June 2008, July 2008, October 2008, January 2010
12 robbat2 1.7 Post-History: December 2009, January 2010
13 cardoe 1.1
14     ========
15     Abstract
16     ========
17     MetaManifest provides a means of verifiable distribution from Gentoo
18     Infrastructure to a user system, while data is conveyed over completely
19     untrusted networks and system, by extending the Manifest2 specification,
20     and adding a top-level Manifest file, with support for other nested
21     Manifests.
23     ==========
24     Motivation
25     ==========
26     As part of a comprehensive security plan, we need a way to prove that
27     something originating from Gentoo as an organization (read Gentoo-owned
28     hardware, run by infrastructure), has not been tampered with. This
29     allows the usage of third-party rsync mirrors, without worrying that
30     they have modified something critical (e.g. eclasses, which are still
31     unsigned).
33     Securing the untrusted distribution is one of the easier tasks in the
34     security plan - in short, all that is required is having a hash of every
35     item in the tree, and signing that hash to prove it came from Gentoo.
37     Ironically we have a hashed and signed distribution (it's just not used
38     by most users, due to it's drawbacks): Our tree snapshot tarballs have
39     hashes and signatures.
41     So now we want to add the same verification to our material that is
42     distributed by rsync. We already provide hashes of subsets of the tree -
43     our Manifests protect individual packages. However metadata, eclasses
44     and profiles are not protected at this time. The directories of
45     packages and distfiles are NOT covered by this, as they are not
46     distributed by rsync.
48     This portion of the tree-signing work provides only the following
49     guarantee: A user can prove that the tree from the Gentoo infrastructure
50     has not been tampered with since leaving the Gentoo infrastructure.
51     No other guarantees, either implicit or explicit are made.
53     Additionally, distributing a set of the most recent MetaManifests from a
54     trusted source allows validation of trees that come from community
55     mirrors, and allows detection of all cases of malicious mirrors (either
56     by deliberate delay, replay [C08a, C08b] or alteration).
58     =============
59     Specification
60     =============
61     For lack of a better name, the following solution should be known as the
62     MetaManifest. Those responsible for the name have already been sacked.
64     MetaManifest basically contains hashes of every file in the tree, either
65     directly or indirectly. The direct case applies to ANY file that does
66     not appear in an existing Manifest file (e.g. eclasses, Manifest files
67     themselves). The indirect case is covered by the CONTENTS of existing
68     Manifest files. If the Manifest itself is correct, we know that by
69     tracking the hash of the Manifest, we can be assured that the contents
70     are protected.
72     In the following, the MetaManifest file is a file named 'Manifest',
73     located at the root of a repository.
75     ---------------------------------------------
76     Procedure for creating the MetaManifest file:
77     ---------------------------------------------
78 robbat2 1.7 Summary:
79     ========
80     The objective of creating the MetaManifest file(s) is to ensure that
81     every single file in the tree occurs in at least one Manifest.
83     Process:
84     ========
85 cardoe 1.1 1. Start at the root of the Gentoo Portage tree (gentoo-x86, although
86     this procedure applies to overlays as well).
88     2. Initialize two unordered sets: COVERED, ALL.
89 robbat2 1.2
90 robbat2 1.7 1. 'ALL' shall contain every file that exists in the present tree.
91     2. 'COVERED' shall contain EVERY file that is mentioned in an existing
92     Manifest2. If a file is mentioned in a Manifest2, but does not
93     exist, it must still be included. No files should be excluded.
94 cardoe 1.1
95     3. Traverse the tree, depth-first.
96 robbat2 1.2
97     1. At the top level only, ignore the following directories: distfiles,
98 robbat2 1.7 packages, local.
99 robbat2 1.2 2. If a directory contains a Manifest file, extract all relevant local
100 robbat2 1.3 files from it (presently: AUX, MISC, EBUILD; but should follow the
101 robbat2 1.4 evolution of Manifest2 entry types per [#GLEP60]), and place them
102 robbat2 1.3 into the COVERED set.
103 robbat2 1.2 3. Recursively add every file in the directory to the ALL set,
104 robbat2 1.6 pursuant to the exclusion list as mentioned in [#GLEP60].
105 cardoe 1.1
106     4. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED).
107     This is every item that is not covered by another Manifest, or part
108     of an exclusion list.
110     5. If an existing MetaManifest file is present, remove it.
112     6. For each file in UNCOVERED, assign a Manifest2 type, produce the
113     hashes, and add with the filetype to the MetaManifest file.
115     7. For unique identification of the MetaManifest, a header line should
116     be included, using the exact contents of the metadata/timestamp.x
117     file, so that a MetaManifest may be tied back to a tree as
118     distributed by the rsync mirror system. The string of
119     'metadata/timestamp.x' should be included to identify this revision
120 robbat2 1.6 of MetaManifest generation. e.g.:
121 cardoe 1.1 "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC"
122     The package manager MUST not use the identifying string as a filename.
124     8. The MetaManifest must ultimately be GnuPG-signed.
125 robbat2 1.2
126     1. For the initial implementation, the same key as used for snapshot
127 robbat2 1.3 tarball signing is sufficient.
128 robbat2 1.2 2. For the future, the key used for fully automated signing by infra
129 robbat2 1.4 should not be on the same keyring as developer keys. See [#GLEPxx+3
130 robbat2 1.3 for further notes].
131 cardoe 1.1
132 robbat2 1.7 Notes:
133     ======
134 cardoe 1.1 The above does not conflict the proposal contained in GLEP33, which
135     restructure eclasses to include subdirectories and Manifest files, as
136     the Manifest rules above still provide indirect verification for all
137     files after the GLEP33 restructuring if it comes to pass.
139 robbat2 1.7 Additional levels of Manifests are required, such as per-category, and
140     in the eclasses, profiles and metadata directories. This ensures that a
141     change to a singular file causes the smallest possible overall change in
142     the Manifests as propagated. Creation of the additional levels of
143     Manifests uses the same process as described above, simply starting at a
144     different root point.
145 cardoe 1.1
146     MetaManifest generation will take place as part of the existing process
147     by infrastructure that takes the contents of CVS and prepares it for
148     distribution via rsync, which includes generating metadata. In-tree
149 robbat2 1.7 Manifest files are not validated at this point, as they are assumed to
150     be correct.
151 cardoe 1.1
152     --------------------------------------------------------
153     Verification of one or more items from the MetaManifest:
154     --------------------------------------------------------
155     There are two times that this may happen: firstly, immediately after the
156     rsync has completed - this has the advantage that the kernel file cache
157     is hot, and checking the entire tree can be accomplished quickly.
158     Secondly, the MetaManifest should be checked during installation of a
159     package.
161     ----------------------------------------------------
162     Procedure for verifying an item in the MetaManifest:
163     ----------------------------------------------------
164     In the following, I've used term 'M2-verify' to note following the hash
165     verification procedures as defined by the Manifest2 format - which
166     compromise checking the file length, and that the hashes match. Which
167 robbat2 1.4 filetypes may be ignored on missing is discussed in [#GLEP60].
168 cardoe 1.1
169     1. Check the GnuPG signature on the MetaManifest against the keyring of
170 robbat2 1.4 automated Gentoo keys. See [#GLEPxx+3] for full details regarding
171 cardoe 1.1 verification of GnuPG signatures.
172 robbat2 1.2 1. Abort if the signature check fails.
173 cardoe 1.1
174 robbat2 1.6 2. Check the Timestamp header. If it is significantly out of date
175 cardoe 1.1 compared to the local clock or a trusted source, halt or require
176     manual intervention from the user.
178     3. For a verification of the tree following an rsync:
179 robbat2 1.2
180     1. Build a set 'ALL' of every file covered by the rsync. (exclude
181 robbat2 1.3 distfiles/, packages/, local/)
182 robbat2 1.2 2. M2-verify every entry in the MetaManifest, descending into inferior
183 robbat2 1.3 Manifests as needed. Place the relative path of every checked item
184     into a set 'COVERED'.
185 robbat2 1.2 3. Construct the set 'UNCOVERED' by set-difference between the ALL and
186 robbat2 1.3 COVERED sets.
187 robbat2 1.2 4. For each file in the UNCOVERED set, assign a Manifest2 filetype.
188     5. If the filetype for any file in the UNCOVERED set requires a halt
189 robbat2 1.3 on error, abort and display a suitable error.
190 robbat2 1.2 6. Completed verification
191 cardoe 1.1
192     4. If checking at the installation of a package:
193 robbat2 1.2
194     1. M2-verify the entry in MetaManifest for the Manifest
195     2. M2-verify all relevant metadata/ contents if metadata/ is being
196 robbat2 1.6 used in any way (optionally done before dependency checking).
197 robbat2 1.2 3. M2-verifying the contents of the Manifest.
198     4. Perform M2-verification of all eclasses and profiles used (both
199 robbat2 1.3 directly and indirectly) by the ebuild.
200 cardoe 1.1
201     Notes:
202     ======
203     1. For initial implementations, it is acceptable to check EVERY item in
204     the eclass and profiles directory, rather than tracking the exact
205     files used by every eclass (see note #2). Later implementations
206     should strive to only verify individual eclasses and profiles as
207     needed.
208     2. Tracking of exact files is of specific significance to the libtool
209     eclass, as it stores patches under eclass/ELT-patches, and as such
210     that would not be picked up by any tracing of the inherit function.
211     This may be alleviated by a later eclass and ebuild variable that
212     explicitly declares what files from the tree are used by a package.
214     ====================
215     Implementation Notes
216     ====================
217     For this portion of the tree-signing work, no actions are required of
218     the individual Gentoo developers. They will continue to develop and
219     commit as they do presently, and the MetaManifest is added by
220     Infrastructure during the tree generation process, and distributed to
221     users.
223 robbat2 1.7 Any scripts generating Manifests and the MetaManifest may find it useful
224     to generate multiple levels of Manifests in parallel, and this is
225     explicitly permitted, provided that every file in the tree is covered by
226     at least one Manifest or the MetaManifest file. The uppermost
227     Manifest (MetaManifest) is the only item that does not occur in any
228     other Manifest file, but is instead GPG-signed to enable it's
229     validation.
231 cardoe 1.1 --------------------------------------------
232     MetaManifest and the new Manifest2 filetypes
233     --------------------------------------------
234 robbat2 1.4 While [#GLEP60] describes the addition of new filetypes, these are NOT
235 cardoe 1.1 needed for implementation of the MetaManifest proposal. Without the new
236     filetypes, all entries in the MetaManifest would be of type 'MISC'.
238     ----------------------------------------------------
239     Timestamps & Additional distribution of MetaManifest
240     ----------------------------------------------------
241     As discussed by [C08a,C08b], malicious third-party mirrors may use the
242     principles of exclusion and replay to deny an update to clients, while
243     at the same time recording the identity of clients to attack.
245     This should be guarded against by including a timestamp in the header of
246     the MetaManifest, as well as distributing the latest MetaManifests by a
247     trusted channel.
249     On all rsync mirrors directly maintained by the Gentoo infrastructure,
250     and not on community mirrors, there should be a new module
251     'gentoo-portage-metamanifests'. Within this module, all MetaManifests
252 robbat2 1.6 for a recent time frame (e.g. one week) should be kept, named as
253 cardoe 1.1 "MetaManifest.$TS", where $TS is the timestamp from inside the file.
254     The most recent MetaManifest should always be symlinked as
255     MetaManifest.current. The possibility of serving the recent
256 robbat2 1.6 MetaManifests via HTTPS should also be explored to mitigate
257     man-in-the-middle attacks.
258 cardoe 1.1
259     The package manager should obtain MetaManifest.current and use it to
260     decide is the tree is too out of date per operation #2 of the
261     verification process. The decision about freshness should be a
262     user-configuration setting, with the ability to override.
264     --------------------------------
265     MetaManifest size considerations
266     --------------------------------
267     With only two levels of Manifests (per-package and top-level), every
268 robbat2 1.6 rsync will cause a lot of traffic transferring the modified top-level
269 robbat2 1.5 MetaManifest. To reduce this, first-level directory Manifests are
270 robbat2 1.7 required. Alternatively, if the distribution method efficiently handles
271     small patch-like changes in an existing file, using an uncompressed
272     MetaManifest may be acceptable (this would primarily be distributed
273     version control systems). Other suggestions in reducing this traffic are
274     welcomed.
275 cardoe 1.1
276     =======================
277     Backwards Compatibility
278     =======================
279     - There are no backwards compatibility issues, as old versions of
280     Portage do not look for a Manifest file at the top level of the tree.
281     - Manifest2-aware versions of Portage ignore all entries that they are
282     not certain how to handle. Enabling headers and PGP signing to be
283     conducted easily.
285     ======
286     Thanks
287     ======
288     I'd like to thank the following people for input on this GLEP.
289 robbat2 1.3
290 cardoe 1.1 - Patrick Lauer (patrick): Prodding me to get all of the tree-signing
291     work finished, and helping to edit.
292     - Ciaran McCreesh (ciaranm): Paludis Manifest2
293     - Brian Harring (ferringb): pkgcore Manifest2
294     - Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
295     - Ned Ludd (solar) - Security concept review
297     ==========
298     References
299     ==========
301     [C08a] Cappos, J et al. (2008). "Package Management Security".
302     University of Arizona Technical Report TR08-02. Available online
303     from: ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf
304     [C08b] Cappos, J et al. (2008). "Attacks on Package Managers"
305     Available online at:
306     http://www.cs.arizona.edu/people/justin/packagemanagersecurity/
307 robbat2 1.8 [#GLEPxx+2] Future GLEP on Developer Process security.
308     [#GLEPxx+3] Future GLEP on GnuPG Policies and Handling.
309 cardoe 1.1
310     =========
311     Copyright
312     =========
313 robbat2 1.6 Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be
314 cardoe 1.1 distributed only subject to the terms and conditions set forth in the
315     Open Publication License, v1.0.
317     vim: tw=72 ts=2 expandtab:

  ViewVC Help
Powered by ViewVC 1.1.20