Contents of /xml/htdocs/proj/en/glep/glep-0058.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.4 - (hide annotations) (download) (as text)
Sun Jan 31 07:53:41 2010 UTC (8 years, 9 months ago) by robbat2
Branch: MAIN
Changes since 1.3: +76 -50 lines
File MIME type: text/html
Revise GLEP58 per Calchan questions: Additional levels of Manifests are no longer optional; Clarifications added to creation process;

1 robbat2 1.1 <?xml version="1.0" encoding="utf-8" ?>
2     <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3     <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
5     <head>
6     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7 robbat2 1.2 <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" />
8 robbat2 1.1 <title>GLEP 58 -- Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest</title>
9     <link rel="stylesheet" href="tools/glep.css" type="text/css" /></head>
10     <body bgcolor="white">
11     <table class="navigation" cellpadding="0" cellspacing="0"
12     width="100%" border="0">
13     <tr><td class="navicon" width="150" height="35">
14     <a href="http://www.gentoo.org/" title="Gentoo Linux Home Page">
15     <img src="http://www.gentoo.org/images/gentoo-new.gif" alt="[Gentoo]"
16     border="0" width="150" height="35" /></a></td>
17     <td class="textlinks" align="left">
18     [<b><a href="http://www.gentoo.org/">Gentoo Linux Home</a></b>]
19     [<b><a href="http://www.gentoo.org/proj/en/glep">GLEP Index</a></b>]
20     [<b><a href="http://www.gentoo.org/proj/en/glep/glep-0058.txt">GLEP Source</a></b>]
21     </td></tr></table>
22     <table class="rfc2822 docutils field-list" frame="void" rules="none">
23     <col class="field-name" />
24     <col class="field-body" />
25     <tbody valign="top">
26     <tr class="field"><th class="field-name">GLEP:</th><td class="field-body">58</td>
27     </tr>
28     <tr class="field"><th class="field-name">Title:</th><td class="field-body">Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest</td>
29     </tr>
30 robbat2 1.4 <tr class="field"><th class="field-name">Version:</th><td class="field-body">1.7</td>
31 robbat2 1.1 </tr>
32 robbat2 1.4 <tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0058.txt?cvsroot=gentoo">2010/01/31 07:53:30</a></td>
33 robbat2 1.1 </tr>
34     <tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;,</td>
35     </tr>
36     <tr class="field"><th class="field-name">Status:</th><td class="field-body">Draft</td>
37     </tr>
38     <tr class="field"><th class="field-name">Type:</th><td class="field-body">Standards Track</td>
39     </tr>
40     <tr class="field"><th class="field-name">Content-Type:</th><td class="field-body"><a class="reference external" href="glep-0002.html">text/x-rst</a></td>
41     </tr>
42     <tr class="field"><th class="field-name">Requires:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0044.html">44</a> <a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0060.html">60</a></td>
43     </tr>
44     <tr class="field"><th class="field-name">Created:</th><td class="field-body">October 2006</td>
45     </tr>
46 robbat2 1.2 <tr class="field"><th class="field-name">Updated:</th><td class="field-body">November 2007, June 2008, July 2008, October 2008, January 2010</td>
47 robbat2 1.1 </tr>
48 robbat2 1.4 <tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009, January 2010</td>
49 robbat2 1.1 </tr>
50     </tbody>
51     </table>
52     <hr />
53     <div class="contents topic" id="contents">
54     <p class="topic-title first">Contents</p>
55     <ul class="simple">
56 robbat2 1.4 <li><a class="reference internal" href="#abstract" id="id2">Abstract</a></li>
57     <li><a class="reference internal" href="#motivation" id="id3">Motivation</a></li>
58     <li><a class="reference internal" href="#specification" id="id4">Specification</a><ul>
59     <li><a class="reference internal" href="#procedure-for-creating-the-metamanifest-file" id="id5">Procedure for creating the MetaManifest file:</a><ul>
60     <li><a class="reference internal" href="#summary" id="id6">Summary:</a></li>
61     <li><a class="reference internal" href="#process" id="id7">Process:</a></li>
62     <li><a class="reference internal" href="#notes" id="id8">Notes:</a></li>
63 robbat2 1.1 </ul>
64     </li>
65 robbat2 1.4 <li><a class="reference internal" href="#verification-of-one-or-more-items-from-the-metamanifest" id="id9">Verification of one or more items from the MetaManifest:</a></li>
66     <li><a class="reference internal" href="#procedure-for-verifying-an-item-in-the-metamanifest" id="id10">Procedure for verifying an item in the MetaManifest:</a><ul>
67     <li><a class="reference internal" href="#id1" id="id11">Notes:</a></li>
68 robbat2 1.1 </ul>
69     </li>
70     </ul>
71     </li>
72 robbat2 1.4 <li><a class="reference internal" href="#implementation-notes" id="id12">Implementation Notes</a><ul>
73     <li><a class="reference internal" href="#metamanifest-and-the-new-manifest2-filetypes" id="id13">MetaManifest and the new Manifest2 filetypes</a></li>
74     <li><a class="reference internal" href="#timestamps-additional-distribution-of-metamanifest" id="id14">Timestamps &amp; Additional distribution of MetaManifest</a></li>
75     <li><a class="reference internal" href="#metamanifest-size-considerations" id="id15">MetaManifest size considerations</a></li>
76     </ul>
77     </li>
78     <li><a class="reference internal" href="#backwards-compatibility" id="id16">Backwards Compatibility</a></li>
79     <li><a class="reference internal" href="#thanks" id="id17">Thanks</a></li>
80     <li><a class="reference internal" href="#references" id="id18">References</a></li>
81     <li><a class="reference internal" href="#copyright" id="id19">Copyright</a></li>
82 robbat2 1.1 </ul>
83     </div>
84     <div class="section" id="abstract">
85 robbat2 1.4 <h1><a class="toc-backref" href="#id2">Abstract</a></h1>
86 robbat2 1.1 <p>MetaManifest provides a means of verifiable distribution from Gentoo
87     Infrastructure to a user system, while data is conveyed over completely
88     untrusted networks and system, by extending the Manifest2 specification,
89     and adding a top-level Manifest file, with support for other nested
90     Manifests.</p>
91     </div>
92     <div class="section" id="motivation">
93 robbat2 1.4 <h1><a class="toc-backref" href="#id3">Motivation</a></h1>
94 robbat2 1.1 <p>As part of a comprehensive security plan, we need a way to prove that
95     something originating from Gentoo as an organization (read Gentoo-owned
96     hardware, run by infrastructure), has not been tampered with. This
97     allows the usage of third-party rsync mirrors, without worrying that
98     they have modified something critical (e.g. eclasses, which are still
99     unsigned).</p>
100     <p>Securing the untrusted distribution is one of the easier tasks in the
101     security plan - in short, all that is required is having a hash of every
102     item in the tree, and signing that hash to prove it came from Gentoo.</p>
103     <p>Ironically we have a hashed and signed distribution (it's just not used
104     by most users, due to it's drawbacks): Our tree snapshot tarballs have
105     hashes and signatures.</p>
106     <p>So now we want to add the same verification to our material that is
107     distributed by rsync. We already provide hashes of subsets of the tree -
108     our Manifests protect individual packages. However metadata, eclasses
109     and profiles are not protected at this time. The directories of
110     packages and distfiles are NOT covered by this, as they are not
111     distributed by rsync.</p>
112     <p>This portion of the tree-signing work provides only the following
113     guarantee: A user can prove that the tree from the Gentoo infrastructure
114     has not been tampered with since leaving the Gentoo infrastructure.
115     No other guarantees, either implicit or explicit are made.</p>
116     <p>Additionally, distributing a set of the most recent MetaManifests from a
117     trusted source allows validation of trees that come from community
118     mirrors, and allows detection of all cases of malicious mirrors (either
119     by deliberate delay, replay [C08a, C08b] or alteration).</p>
120     </div>
121     <div class="section" id="specification">
122 robbat2 1.4 <h1><a class="toc-backref" href="#id4">Specification</a></h1>
123 robbat2 1.1 <p>For lack of a better name, the following solution should be known as the
124     MetaManifest. Those responsible for the name have already been sacked.</p>
125     <p>MetaManifest basically contains hashes of every file in the tree, either
126     directly or indirectly. The direct case applies to ANY file that does
127     not appear in an existing Manifest file (e.g. eclasses, Manifest files
128     themselves). The indirect case is covered by the CONTENTS of existing
129     Manifest files. If the Manifest itself is correct, we know that by
130     tracking the hash of the Manifest, we can be assured that the contents
131     are protected.</p>
132     <p>In the following, the MetaManifest file is a file named 'Manifest',
133     located at the root of a repository.</p>
134     <div class="section" id="procedure-for-creating-the-metamanifest-file">
135 robbat2 1.4 <h2><a class="toc-backref" href="#id5">Procedure for creating the MetaManifest file:</a></h2>
136     <div class="section" id="summary">
137     <h3><a class="toc-backref" href="#id6">Summary:</a></h3>
138     <p>The objective of creating the MetaManifest file(s) is to ensure that
139     every single file in the tree occurs in at least one Manifest.</p>
140     </div>
141     <div class="section" id="process">
142     <h3><a class="toc-backref" href="#id7">Process:</a></h3>
143 robbat2 1.1 <ol class="arabic simple">
144     <li>Start at the root of the Gentoo Portage tree (gentoo-x86, although
145     this procedure applies to overlays as well).</li>
146     <li>Initialize two unordered sets: COVERED, ALL.<ol class="arabic">
147 robbat2 1.4 <li>'ALL' shall contain every file that exists in the present tree.</li>
148     <li>'COVERED' shall contain EVERY file that is mentioned in an existing
149     Manifest2. If a file is mentioned in a Manifest2, but does not
150     exist, it must still be included. No files should be excluded.</li>
151 robbat2 1.1 </ol>
152     </li>
153     <li>Traverse the tree, depth-first.<ol class="arabic">
154     <li>At the top level only, ignore the following directories: distfiles,
155 robbat2 1.4 packages, local.</li>
156 robbat2 1.1 <li>If a directory contains a Manifest file, extract all relevant local
157     files from it (presently: AUX, MISC, EBUILD; but should follow the
158     evolution of Manifest2 entry types per [#GLEP60]), and place them
159     into the COVERED set.</li>
160     <li>Recursively add every file in the directory to the ALL set,
161 robbat2 1.3 pursuant to the exclusion list as mentioned in [#GLEP60].</li>
162 robbat2 1.1 </ol>
163     </li>
164     <li>Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED).
165     This is every item that is not covered by another Manifest, or part
166     of an exclusion list.</li>
167     <li>If an existing MetaManifest file is present, remove it.</li>
168     <li>For each file in UNCOVERED, assign a Manifest2 type, produce the
169     hashes, and add with the filetype to the MetaManifest file.</li>
170     <li>For unique identification of the MetaManifest, a header line should
171     be included, using the exact contents of the metadata/timestamp.x
172     file, so that a MetaManifest may be tied back to a tree as
173     distributed by the rsync mirror system. The string of
174     'metadata/timestamp.x' should be included to identify this revision
175 robbat2 1.3 of MetaManifest generation. e.g.:
176 robbat2 1.1 &quot;Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC&quot;
177     The package manager MUST not use the identifying string as a filename.</li>
178     <li>The MetaManifest must ultimately be GnuPG-signed.<ol class="arabic">
179     <li>For the initial implementation, the same key as used for snapshot
180     tarball signing is sufficient.</li>
181     <li>For the future, the key used for fully automated signing by infra
182     should not be on the same keyring as developer keys. See [#GLEPxx+3
183     for further notes].</li>
184     </ol>
185     </li>
186     </ol>
187 robbat2 1.4 </div>
188     <div class="section" id="notes">
189     <h3><a class="toc-backref" href="#id8">Notes:</a></h3>
190 robbat2 1.1 <p>The above does not conflict the proposal contained in GLEP33, which
191     restructure eclasses to include subdirectories and Manifest files, as
192     the Manifest rules above still provide indirect verification for all
193     files after the GLEP33 restructuring if it comes to pass.</p>
194 robbat2 1.4 <p>Additional levels of Manifests are required, such as per-category, and
195     in the eclasses, profiles and metadata directories. This ensures that a
196     change to a singular file causes the smallest possible overall change in
197     the Manifests as propagated. Creation of the additional levels of
198     Manifests uses the same process as described above, simply starting at a
199     different root point.</p>
200 robbat2 1.1 <p>MetaManifest generation will take place as part of the existing process
201     by infrastructure that takes the contents of CVS and prepares it for
202     distribution via rsync, which includes generating metadata. In-tree
203 robbat2 1.4 Manifest files are not validated at this point, as they are assumed to
204     be correct.</p>
205     </div>
206 robbat2 1.1 </div>
207     <div class="section" id="verification-of-one-or-more-items-from-the-metamanifest">
208 robbat2 1.4 <h2><a class="toc-backref" href="#id9">Verification of one or more items from the MetaManifest:</a></h2>
209 robbat2 1.1 <p>There are two times that this may happen: firstly, immediately after the
210     rsync has completed - this has the advantage that the kernel file cache
211     is hot, and checking the entire tree can be accomplished quickly.
212     Secondly, the MetaManifest should be checked during installation of a
213     package.</p>
214     </div>
215     <div class="section" id="procedure-for-verifying-an-item-in-the-metamanifest">
216 robbat2 1.4 <h2><a class="toc-backref" href="#id10">Procedure for verifying an item in the MetaManifest:</a></h2>
217 robbat2 1.1 <p>In the following, I've used term 'M2-verify' to note following the hash
218     verification procedures as defined by the Manifest2 format - which
219     compromise checking the file length, and that the hashes match. Which
220     filetypes may be ignored on missing is discussed in [#GLEP60].</p>
221     <ol class="arabic simple">
222     <li>Check the GnuPG signature on the MetaManifest against the keyring of
223     automated Gentoo keys. See [#GLEPxx+3] for full details regarding
224     verification of GnuPG signatures.
225     1. Abort if the signature check fails.</li>
226 robbat2 1.3 <li>Check the Timestamp header. If it is significantly out of date
227 robbat2 1.1 compared to the local clock or a trusted source, halt or require
228     manual intervention from the user.</li>
229     <li>For a verification of the tree following an rsync:<ol class="arabic">
230     <li>Build a set 'ALL' of every file covered by the rsync. (exclude
231     distfiles/, packages/, local/)</li>
232     <li>M2-verify every entry in the MetaManifest, descending into inferior
233     Manifests as needed. Place the relative path of every checked item
234     into a set 'COVERED'.</li>
235     <li>Construct the set 'UNCOVERED' by set-difference between the ALL and
236     COVERED sets.</li>
237     <li>For each file in the UNCOVERED set, assign a Manifest2 filetype.</li>
238     <li>If the filetype for any file in the UNCOVERED set requires a halt
239     on error, abort and display a suitable error.</li>
240     <li>Completed verification</li>
241     </ol>
242     </li>
243     <li>If checking at the installation of a package:<ol class="arabic">
244     <li>M2-verify the entry in MetaManifest for the Manifest</li>
245     <li>M2-verify all relevant metadata/ contents if metadata/ is being
246 robbat2 1.3 used in any way (optionally done before dependency checking).</li>
247 robbat2 1.1 <li>M2-verifying the contents of the Manifest.</li>
248     <li>Perform M2-verification of all eclasses and profiles used (both
249     directly and indirectly) by the ebuild.</li>
250     </ol>
251     </li>
252     </ol>
253 robbat2 1.4 <div class="section" id="id1">
254     <h3><a class="toc-backref" href="#id11">Notes:</a></h3>
255 robbat2 1.1 <ol class="arabic simple">
256     <li>For initial implementations, it is acceptable to check EVERY item in
257     the eclass and profiles directory, rather than tracking the exact
258     files used by every eclass (see note #2). Later implementations
259     should strive to only verify individual eclasses and profiles as
260     needed.</li>
261     <li>Tracking of exact files is of specific significance to the libtool
262     eclass, as it stores patches under eclass/ELT-patches, and as such
263     that would not be picked up by any tracing of the inherit function.
264     This may be alleviated by a later eclass and ebuild variable that
265     explicitly declares what files from the tree are used by a package.</li>
266     </ol>
267     </div>
268     </div>
269     </div>
270     <div class="section" id="implementation-notes">
271 robbat2 1.4 <h1><a class="toc-backref" href="#id12">Implementation Notes</a></h1>
272 robbat2 1.1 <p>For this portion of the tree-signing work, no actions are required of
273     the individual Gentoo developers. They will continue to develop and
274     commit as they do presently, and the MetaManifest is added by
275     Infrastructure during the tree generation process, and distributed to
276     users.</p>
277 robbat2 1.4 <p>Any scripts generating Manifests and the MetaManifest may find it useful
278     to generate multiple levels of Manifests in parallel, and this is
279     explicitly permitted, provided that every file in the tree is covered by
280     at least one Manifest or the MetaManifest file. The uppermost
281     Manifest (MetaManifest) is the only item that does not occur in any
282     other Manifest file, but is instead GPG-signed to enable it's
283     validation.</p>
284 robbat2 1.1 <div class="section" id="metamanifest-and-the-new-manifest2-filetypes">
285 robbat2 1.4 <h2><a class="toc-backref" href="#id13">MetaManifest and the new Manifest2 filetypes</a></h2>
286 robbat2 1.1 <p>While [#GLEP60] describes the addition of new filetypes, these are NOT
287     needed for implementation of the MetaManifest proposal. Without the new
288     filetypes, all entries in the MetaManifest would be of type 'MISC'.</p>
289     </div>
290     <div class="section" id="timestamps-additional-distribution-of-metamanifest">
291 robbat2 1.4 <h2><a class="toc-backref" href="#id14">Timestamps &amp; Additional distribution of MetaManifest</a></h2>
292 robbat2 1.1 <p>As discussed by [C08a,C08b], malicious third-party mirrors may use the
293     principles of exclusion and replay to deny an update to clients, while
294     at the same time recording the identity of clients to attack.</p>
295     <p>This should be guarded against by including a timestamp in the header of
296     the MetaManifest, as well as distributing the latest MetaManifests by a
297     trusted channel.</p>
298     <p>On all rsync mirrors directly maintained by the Gentoo infrastructure,
299     and not on community mirrors, there should be a new module
300     'gentoo-portage-metamanifests'. Within this module, all MetaManifests
301 robbat2 1.3 for a recent time frame (e.g. one week) should be kept, named as
302 robbat2 1.1 &quot;MetaManifest.$TS&quot;, where $TS is the timestamp from inside the file.
303     The most recent MetaManifest should always be symlinked as
304     MetaManifest.current. The possibility of serving the recent
305 robbat2 1.3 MetaManifests via HTTPS should also be explored to mitigate
306     man-in-the-middle attacks.</p>
307 robbat2 1.1 <p>The package manager should obtain MetaManifest.current and use it to
308     decide is the tree is too out of date per operation #2 of the
309     verification process. The decision about freshness should be a
310     user-configuration setting, with the ability to override.</p>
311     </div>
312     <div class="section" id="metamanifest-size-considerations">
313 robbat2 1.4 <h2><a class="toc-backref" href="#id15">MetaManifest size considerations</a></h2>
314 robbat2 1.1 <p>With only two levels of Manifests (per-package and top-level), every
315 robbat2 1.3 rsync will cause a lot of traffic transferring the modified top-level
316 robbat2 1.2 MetaManifest. To reduce this, first-level directory Manifests are
317 robbat2 1.4 required. Alternatively, if the distribution method efficiently handles
318     small patch-like changes in an existing file, using an uncompressed
319     MetaManifest may be acceptable (this would primarily be distributed
320     version control systems). Other suggestions in reducing this traffic are
321     welcomed.</p>
322 robbat2 1.1 </div>
323     </div>
324     <div class="section" id="backwards-compatibility">
325 robbat2 1.4 <h1><a class="toc-backref" href="#id16">Backwards Compatibility</a></h1>
326 robbat2 1.1 <ul class="simple">
327     <li>There are no backwards compatibility issues, as old versions of
328     Portage do not look for a Manifest file at the top level of the tree.</li>
329     <li>Manifest2-aware versions of Portage ignore all entries that they are
330     not certain how to handle. Enabling headers and PGP signing to be
331     conducted easily.</li>
332     </ul>
333     </div>
334     <div class="section" id="thanks">
335 robbat2 1.4 <h1><a class="toc-backref" href="#id17">Thanks</a></h1>
336 robbat2 1.1 <p>I'd like to thank the following people for input on this GLEP.</p>
337     <ul class="simple">
338     <li>Patrick Lauer (patrick): Prodding me to get all of the tree-signing
339     work finished, and helping to edit.</li>
340     <li>Ciaran McCreesh (ciaranm): Paludis Manifest2</li>
341     <li>Brian Harring (ferringb): pkgcore Manifest2</li>
342     <li>Marius Mauch (genone) &amp; Zac Medico (zmedico): Portage Manifest2</li>
343     <li>Ned Ludd (solar) - Security concept review</li>
344     </ul>
345     </div>
346     <div class="section" id="references">
347 robbat2 1.4 <h1><a class="toc-backref" href="#id18">References</a></h1>
348 robbat2 1.1 <dl class="docutils">
349     <dt>[C08a] Cappos, J et al. (2008). &quot;Package Management Security&quot;.</dt>
350     <dd>University of Arizona Technical Report TR08-02. Available online
351     from: <a class="reference external" href="ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf">ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf</a></dd>
352     <dt>[C08b] Cappos, J et al. (2008). &quot;Attacks on Package Managers&quot;</dt>
353     <dd>Available online at:
354     <a class="reference external" href="http://www.cs.arizona.edu/people/justin/packagemanagersecurity/">http://www.cs.arizona.edu/people/justin/packagemanagersecurity/</a></dd>
355     </dl>
356     </div>
357     <div class="section" id="copyright">
358 robbat2 1.4 <h1><a class="toc-backref" href="#id19">Copyright</a></h1>
359 robbat2 1.3 <p>Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be
360 robbat2 1.1 distributed only subject to the terms and conditions set forth in the
361     Open Publication License, v1.0.</p>
362     <p>vim: tw=72 ts=2 expandtab:</p>
363     </div>
365     </div>
366     <div class="footer">
367     <hr class="footer" />
368     <a class="reference external" href="glep-0058.txt">View document source</a>.
369 robbat2 1.4 Generated on: 2010-01-31 07:53 UTC.
370 robbat2 1.1 Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.
372     </div>
373     </body>
374     </html>

  ViewVC Help
Powered by ViewVC 1.1.20