/[gentoo]/xml/htdocs/proj/en/glep/glep-0058.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0058.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.10 - (show annotations) (download)
Wed Apr 7 21:34:24 2010 UTC (4 years, 5 months ago) by robbat2
Branch: MAIN
CVS Tags: HEAD
Changes since 1.9: +19 -13 lines
File MIME type: text/plain
Error occurred while calculating annotation data.
More fixes of markup and reference formatting.

1 GLEP: 58
2 Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest
3 Version: $Revision: 1.9 $
4 Last-Modified: $Date: 2010/04/07 06:35:16 $
5 Author: Robin Hugh Johnson <robbat2@gentoo.org>,
6 Status: Draft
7 Type: Standards Track
8 Content-Type: text/x-rst
9 Requires: 44, 60
10 Created: October 2006
11 Updated: November 2007, June 2008, July 2008, October 2008, January 2010
12 Post-History: December 2009, January 2010
13
14 ========
15 Abstract
16 ========
17 MetaManifest provides a means of verifiable distribution from Gentoo
18 Infrastructure to a user system, while data is conveyed over completely
19 untrusted networks and system, by extending the Manifest2 specification,
20 and adding a top-level Manifest file, with support for other nested
21 Manifests.
22
23 ==========
24 Motivation
25 ==========
26 As part of a comprehensive security plan, we need a way to prove that
27 something originating from Gentoo as an organization (read Gentoo-owned
28 hardware, run by infrastructure), has not been tampered with. This
29 allows the usage of third-party rsync mirrors, without worrying that
30 they have modified something critical (e.g. eclasses, which are still
31 unsigned).
32
33 Securing the untrusted distribution is one of the easier tasks in the
34 security plan - in short, all that is required is having a hash of every
35 item in the tree, and signing that hash to prove it came from Gentoo.
36
37 Ironically we have a hashed and signed distribution (it's just not used
38 by most users, due to it's drawbacks): Our tree snapshot tarballs have
39 hashes and signatures.
40
41 So now we want to add the same verification to our material that is
42 distributed by rsync. We already provide hashes of subsets of the tree -
43 our Manifests protect individual packages. However metadata, eclasses
44 and profiles are not protected at this time. The directories of
45 packages and distfiles are NOT covered by this, as they are not
46 distributed by rsync.
47
48 This portion of the tree-signing work provides only the following
49 guarantee: A user can prove that the tree from the Gentoo infrastructure
50 has not been tampered with since leaving the Gentoo infrastructure.
51 No other guarantees, either implicit or explicit are made.
52
53 Additionally, distributing a set of the most recent MetaManifests from a
54 trusted source allows validation of trees that come from community
55 mirrors, and allows detection of all cases of malicious mirrors (either
56 by deliberate delay, replay [C08a, C08b] or alteration).
57
58 =============
59 Specification
60 =============
61 For lack of a better name, the following solution should be known as the
62 MetaManifest. Those responsible for the name have already been sacked.
63
64 MetaManifest basically contains hashes of every file in the tree, either
65 directly or indirectly. The direct case applies to ANY file that does
66 not appear in an existing Manifest file (e.g. eclasses, Manifest files
67 themselves). The indirect case is covered by the CONTENTS of existing
68 Manifest files. If the Manifest itself is correct, we know that by
69 tracking the hash of the Manifest, we can be assured that the contents
70 are protected.
71
72 In the following, the MetaManifest file is a file named 'Manifest',
73 located at the root of a repository.
74
75 ---------------------------------------------
76 Procedure for creating the MetaManifest file:
77 ---------------------------------------------
78 Summary:
79 ========
80 The objective of creating the MetaManifest file(s) is to ensure that
81 every single file in the tree occurs in at least one Manifest.
82
83 Process:
84 ========
85 1. Start at the root of the Gentoo Portage tree (gentoo-x86, although
86 this procedure applies to overlays as well).
87
88 2. Initialize two unordered sets: COVERED, ALL.
89
90 1. 'ALL' shall contain every file that exists in the present tree.
91 2. 'COVERED' shall contain EVERY file that is mentioned in an existing
92 Manifest2. If a file is mentioned in a Manifest2, but does not
93 exist, it must still be included. No files should be excluded.
94
95 3. Traverse the tree, depth-first.
96
97 1. At the top level only, ignore the following directories: distfiles,
98 packages, local.
99 2. If a directory contains a Manifest file, extract all relevant local
100 files from it (presently: AUX, MISC, EBUILD; but should follow the
101 evolution of Manifest2 entry types per [GLEP60]), and place them
102 into the COVERED set.
103 3. Recursively add every file in the directory to the ALL set,
104 pursuant to the exclusion list as mentioned in [GLEP60].
105
106 4. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED).
107 This is every item that is not covered by another Manifest, or part
108 of an exclusion list.
109
110 5. If an existing MetaManifest file is present, remove it.
111
112 6. For each file in UNCOVERED, assign a Manifest2 type, produce the
113 hashes, and add with the filetype to the MetaManifest file.
114
115 7. For unique identification of the MetaManifest, a header line should
116 be included, using the exact contents of the metadata/timestamp.x
117 file, so that a MetaManifest may be tied back to a tree as
118 distributed by the rsync mirror system. The string of
119 'metadata/timestamp.x' should be included to identify this revision
120 of MetaManifest generation. e.g.:
121 "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC"
122 The package manager MUST not use the identifying string as a filename.
123
124 8. The MetaManifest must ultimately be GnuPG-signed.
125
126 1. For the initial implementation, the same key as used for snapshot
127 tarball signing is sufficient.
128 2. For the future, the key used for fully automated signing by infra
129 should not be on the same keyring as developer keys. See
130 [GLEPxx3] for further notes.
131
132 Notes:
133 ======
134 The above does not conflict the proposal contained in [GLEP33], which
135 restructure eclasses to include subdirectories and Manifest files, as
136 the Manifest rules above still provide indirect verification for all
137 files after the [GLEP33] restructuring if it comes to pass.
138
139 Additional levels of Manifests are required, such as per-category, and
140 in the eclasses, profiles and metadata directories. This ensures that a
141 change to a singular file causes the smallest possible overall change in
142 the Manifests as propagated. Creation of the additional levels of
143 Manifests uses the same process as described above, simply starting at a
144 different root point.
145
146 MetaManifest generation will take place as part of the existing process
147 by infrastructure that takes the contents of CVS and prepares it for
148 distribution via rsync, which includes generating metadata. In-tree
149 Manifest files are not validated at this point, as they are assumed to
150 be correct.
151
152 --------------------------------------------------------
153 Verification of one or more items from the MetaManifest:
154 --------------------------------------------------------
155 There are two times that this may happen: firstly, immediately after the
156 rsync has completed - this has the advantage that the kernel file cache
157 is hot, and checking the entire tree can be accomplished quickly.
158 Secondly, the MetaManifest should be checked during installation of a
159 package.
160
161 ----------------------------------------------------
162 Procedure for verifying an item in the MetaManifest:
163 ----------------------------------------------------
164 In the following, I've used term 'M2-verify' to note following the hash
165 verification procedures as defined by the Manifest2 format - which
166 compromise checking the file length, and that the hashes match. Which
167 filetypes may be ignored on missing is discussed in [GLEP60].
168
169 1. Check the GnuPG signature on the MetaManifest against the keyring of
170 automated Gentoo keys. See [GLEPxx3] for full details regarding
171 verification of GnuPG signatures.
172 1. Abort if the signature check fails.
173
174 2. Check the Timestamp header. If it is significantly out of date
175 compared to the local clock or a trusted source, halt or require
176 manual intervention from the user.
177
178 3. For a verification of the tree following an rsync:
179
180 1. Build a set 'ALL' of every file covered by the rsync. (exclude
181 distfiles/, packages/, local/)
182 2. M2-verify every entry in the MetaManifest, descending into inferior
183 Manifests as needed. Place the relative path of every checked item
184 into a set 'COVERED'.
185 3. Construct the set 'UNCOVERED' by set-difference between the ALL and
186 COVERED sets.
187 4. For each file in the UNCOVERED set, assign a Manifest2 filetype.
188 5. If the filetype for any file in the UNCOVERED set requires a halt
189 on error, abort and display a suitable error.
190 6. Completed verification
191
192 4. If checking at the installation of a package:
193
194 1. M2-verify the entry in MetaManifest for the Manifest
195 2. M2-verify all relevant metadata/ contents if metadata/ is being
196 used in any way (optionally done before dependency checking).
197 3. M2-verifying the contents of the Manifest.
198 4. Perform M2-verification of all eclasses and profiles used (both
199 directly and indirectly) by the ebuild.
200
201 Notes:
202 ======
203 1. For initial implementations, it is acceptable to check EVERY item in
204 the eclass and profiles directory, rather than tracking the exact
205 files used by every eclass (see note #2). Later implementations
206 should strive to only verify individual eclasses and profiles as
207 needed.
208 2. Tracking of exact files is of specific significance to the libtool
209 eclass, as it stores patches under eclass/ELT-patches, and as such
210 that would not be picked up by any tracing of the inherit function.
211 This may be alleviated by a later eclass and ebuild variable that
212 explicitly declares what files from the tree are used by a package.
213
214 ====================
215 Implementation Notes
216 ====================
217 For this portion of the tree-signing work, no actions are required of
218 the individual Gentoo developers. They will continue to develop and
219 commit as they do presently, and the MetaManifest is added by
220 Infrastructure during the tree generation process, and distributed to
221 users.
222
223 Any scripts generating Manifests and the MetaManifest may find it useful
224 to generate multiple levels of Manifests in parallel, and this is
225 explicitly permitted, provided that every file in the tree is covered by
226 at least one Manifest or the MetaManifest file. The uppermost
227 Manifest (MetaManifest) is the only item that does not occur in any
228 other Manifest file, but is instead GPG-signed to enable it's
229 validation.
230
231 --------------------------------------------
232 MetaManifest and the new Manifest2 filetypes
233 --------------------------------------------
234 While [GLEP60] describes the addition of new filetypes, these are NOT
235 needed for implementation of the MetaManifest proposal. Without the new
236 filetypes, all entries in the MetaManifest would be of type 'MISC'.
237
238 ----------------------------------------------------
239 Timestamps & Additional distribution of MetaManifest
240 ----------------------------------------------------
241 As discussed by [C08a,C08b], malicious third-party mirrors may use the
242 principles of exclusion and replay to deny an update to clients, while
243 at the same time recording the identity of clients to attack.
244
245 This should be guarded against by including a timestamp in the header of
246 the MetaManifest, as well as distributing the latest MetaManifests by a
247 trusted channel.
248
249 On all rsync mirrors directly maintained by the Gentoo infrastructure,
250 and not on community mirrors, there should be a new module
251 'gentoo-portage-metamanifests'. Within this module, all MetaManifests
252 for a recent time frame (e.g. one week) should be kept, named as
253 "MetaManifest.$TS", where $TS is the timestamp from inside the file.
254 The most recent MetaManifest should always be symlinked as
255 MetaManifest.current. The possibility of serving the recent
256 MetaManifests via HTTPS should also be explored to mitigate
257 man-in-the-middle attacks.
258
259 The package manager should obtain MetaManifest.current and use it to
260 decide is the tree is too out of date per operation #2 of the
261 verification process. The decision about freshness should be a
262 user-configuration setting, with the ability to override.
263
264 --------------------------------
265 MetaManifest size considerations
266 --------------------------------
267 With only two levels of Manifests (per-package and top-level), every
268 rsync will cause a lot of traffic transferring the modified top-level
269 MetaManifest. To reduce this, first-level directory Manifests are
270 required. Alternatively, if the distribution method efficiently handles
271 small patch-like changes in an existing file, using an uncompressed
272 MetaManifest may be acceptable (this would primarily be distributed
273 version control systems). Other suggestions in reducing this traffic are
274 welcomed.
275
276 =======================
277 Backwards Compatibility
278 =======================
279 - There are no backwards compatibility issues, as old versions of
280 Portage do not look for a Manifest file at the top level of the tree.
281 - Manifest2-aware versions of Portage ignore all entries that they are
282 not certain how to handle. Enabling headers and PGP signing to be
283 conducted easily.
284
285 ======
286 Thanks
287 ======
288 I'd like to thank the following people for input on this GLEP.
289
290 - Patrick Lauer (patrick): Prodding me to get all of the tree-signing
291 work finished, and helping to edit.
292 - Ciaran McCreesh (ciaranm): Paludis Manifest2
293 - Brian Harring (ferringb): pkgcore Manifest2
294 - Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2
295 - Ned Ludd (solar) - Security concept review
296
297 ==========
298 References
299 ==========
300
301 .. [C08a] Cappos, J et al. (2008). "Package Management Security".
302 University of Arizona Technical Report TR08-02. Available online
303 from: ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf
304
305 .. [C08b] Cappos, J et al. (2008). "Attacks on Package Managers"
306 Available online at:
307 http://www.cs.arizona.edu/people/justin/packagemanagersecurity/
308
309 .. [GLEP33] Eclass Restructure/Redesign
310 http://www.gentoo.org/proj/en/glep/glep-0033.html
311
312 .. [GLEP60] Manifest2 filetypes
313 http://www.gentoo.org/proj/en/glep/glep-0044.html
314
315 .. [GLEPxx2] Future GLEP on Developer Process security.
316
317 .. [GLEPxx3] Future GLEP on GnuPG Policies and Handling.
318
319 =========
320 Copyright
321 =========
322 Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be
323 distributed only subject to the terms and conditions set forth in the
324 Open Publication License, v1.0.
325
326 .. vim: tw=72 ts=2 expandtab:

  ViewVC Help
Powered by ViewVC 1.1.20