| 1 |
GLEP: 58 |
| 2 |
Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest |
| 3 |
Version: $Revision: 1.7 $ |
| 4 |
Last-Modified: $Date: 2010/01/31 07:53:30 $ |
| 5 |
Author: Robin Hugh Johnson <robbat2@gentoo.org>, |
| 6 |
Status: Draft |
| 7 |
Type: Standards Track |
| 8 |
Content-Type: text/x-rst |
| 9 |
Requires: 44, 60 |
| 10 |
Created: October 2006 |
| 11 |
Updated: November 2007, June 2008, July 2008, October 2008, January 2010 |
| 12 |
Post-History: December 2009, January 2010 |
| 13 |
|
| 14 |
======== |
| 15 |
Abstract |
| 16 |
======== |
| 17 |
MetaManifest provides a means of verifiable distribution from Gentoo |
| 18 |
Infrastructure to a user system, while data is conveyed over completely |
| 19 |
untrusted networks and system, by extending the Manifest2 specification, |
| 20 |
and adding a top-level Manifest file, with support for other nested |
| 21 |
Manifests. |
| 22 |
|
| 23 |
========== |
| 24 |
Motivation |
| 25 |
========== |
| 26 |
As part of a comprehensive security plan, we need a way to prove that |
| 27 |
something originating from Gentoo as an organization (read Gentoo-owned |
| 28 |
hardware, run by infrastructure), has not been tampered with. This |
| 29 |
allows the usage of third-party rsync mirrors, without worrying that |
| 30 |
they have modified something critical (e.g. eclasses, which are still |
| 31 |
unsigned). |
| 32 |
|
| 33 |
Securing the untrusted distribution is one of the easier tasks in the |
| 34 |
security plan - in short, all that is required is having a hash of every |
| 35 |
item in the tree, and signing that hash to prove it came from Gentoo. |
| 36 |
|
| 37 |
Ironically we have a hashed and signed distribution (it's just not used |
| 38 |
by most users, due to it's drawbacks): Our tree snapshot tarballs have |
| 39 |
hashes and signatures. |
| 40 |
|
| 41 |
So now we want to add the same verification to our material that is |
| 42 |
distributed by rsync. We already provide hashes of subsets of the tree - |
| 43 |
our Manifests protect individual packages. However metadata, eclasses |
| 44 |
and profiles are not protected at this time. The directories of |
| 45 |
packages and distfiles are NOT covered by this, as they are not |
| 46 |
distributed by rsync. |
| 47 |
|
| 48 |
This portion of the tree-signing work provides only the following |
| 49 |
guarantee: A user can prove that the tree from the Gentoo infrastructure |
| 50 |
has not been tampered with since leaving the Gentoo infrastructure. |
| 51 |
No other guarantees, either implicit or explicit are made. |
| 52 |
|
| 53 |
Additionally, distributing a set of the most recent MetaManifests from a |
| 54 |
trusted source allows validation of trees that come from community |
| 55 |
mirrors, and allows detection of all cases of malicious mirrors (either |
| 56 |
by deliberate delay, replay [C08a, C08b] or alteration). |
| 57 |
|
| 58 |
============= |
| 59 |
Specification |
| 60 |
============= |
| 61 |
For lack of a better name, the following solution should be known as the |
| 62 |
MetaManifest. Those responsible for the name have already been sacked. |
| 63 |
|
| 64 |
MetaManifest basically contains hashes of every file in the tree, either |
| 65 |
directly or indirectly. The direct case applies to ANY file that does |
| 66 |
not appear in an existing Manifest file (e.g. eclasses, Manifest files |
| 67 |
themselves). The indirect case is covered by the CONTENTS of existing |
| 68 |
Manifest files. If the Manifest itself is correct, we know that by |
| 69 |
tracking the hash of the Manifest, we can be assured that the contents |
| 70 |
are protected. |
| 71 |
|
| 72 |
In the following, the MetaManifest file is a file named 'Manifest', |
| 73 |
located at the root of a repository. |
| 74 |
|
| 75 |
--------------------------------------------- |
| 76 |
Procedure for creating the MetaManifest file: |
| 77 |
--------------------------------------------- |
| 78 |
Summary: |
| 79 |
======== |
| 80 |
The objective of creating the MetaManifest file(s) is to ensure that |
| 81 |
every single file in the tree occurs in at least one Manifest. |
| 82 |
|
| 83 |
Process: |
| 84 |
======== |
| 85 |
1. Start at the root of the Gentoo Portage tree (gentoo-x86, although |
| 86 |
this procedure applies to overlays as well). |
| 87 |
|
| 88 |
2. Initialize two unordered sets: COVERED, ALL. |
| 89 |
|
| 90 |
1. 'ALL' shall contain every file that exists in the present tree. |
| 91 |
2. 'COVERED' shall contain EVERY file that is mentioned in an existing |
| 92 |
Manifest2. If a file is mentioned in a Manifest2, but does not |
| 93 |
exist, it must still be included. No files should be excluded. |
| 94 |
|
| 95 |
3. Traverse the tree, depth-first. |
| 96 |
|
| 97 |
1. At the top level only, ignore the following directories: distfiles, |
| 98 |
packages, local. |
| 99 |
2. If a directory contains a Manifest file, extract all relevant local |
| 100 |
files from it (presently: AUX, MISC, EBUILD; but should follow the |
| 101 |
evolution of Manifest2 entry types per [#GLEP60]), and place them |
| 102 |
into the COVERED set. |
| 103 |
3. Recursively add every file in the directory to the ALL set, |
| 104 |
pursuant to the exclusion list as mentioned in [#GLEP60]. |
| 105 |
|
| 106 |
4. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED). |
| 107 |
This is every item that is not covered by another Manifest, or part |
| 108 |
of an exclusion list. |
| 109 |
|
| 110 |
5. If an existing MetaManifest file is present, remove it. |
| 111 |
|
| 112 |
6. For each file in UNCOVERED, assign a Manifest2 type, produce the |
| 113 |
hashes, and add with the filetype to the MetaManifest file. |
| 114 |
|
| 115 |
7. For unique identification of the MetaManifest, a header line should |
| 116 |
be included, using the exact contents of the metadata/timestamp.x |
| 117 |
file, so that a MetaManifest may be tied back to a tree as |
| 118 |
distributed by the rsync mirror system. The string of |
| 119 |
'metadata/timestamp.x' should be included to identify this revision |
| 120 |
of MetaManifest generation. e.g.: |
| 121 |
"Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC" |
| 122 |
The package manager MUST not use the identifying string as a filename. |
| 123 |
|
| 124 |
8. The MetaManifest must ultimately be GnuPG-signed. |
| 125 |
|
| 126 |
1. For the initial implementation, the same key as used for snapshot |
| 127 |
tarball signing is sufficient. |
| 128 |
2. For the future, the key used for fully automated signing by infra |
| 129 |
should not be on the same keyring as developer keys. See [#GLEPxx+3 |
| 130 |
for further notes]. |
| 131 |
|
| 132 |
Notes: |
| 133 |
====== |
| 134 |
The above does not conflict the proposal contained in GLEP33, which |
| 135 |
restructure eclasses to include subdirectories and Manifest files, as |
| 136 |
the Manifest rules above still provide indirect verification for all |
| 137 |
files after the GLEP33 restructuring if it comes to pass. |
| 138 |
|
| 139 |
Additional levels of Manifests are required, such as per-category, and |
| 140 |
in the eclasses, profiles and metadata directories. This ensures that a |
| 141 |
change to a singular file causes the smallest possible overall change in |
| 142 |
the Manifests as propagated. Creation of the additional levels of |
| 143 |
Manifests uses the same process as described above, simply starting at a |
| 144 |
different root point. |
| 145 |
|
| 146 |
MetaManifest generation will take place as part of the existing process |
| 147 |
by infrastructure that takes the contents of CVS and prepares it for |
| 148 |
distribution via rsync, which includes generating metadata. In-tree |
| 149 |
Manifest files are not validated at this point, as they are assumed to |
| 150 |
be correct. |
| 151 |
|
| 152 |
-------------------------------------------------------- |
| 153 |
Verification of one or more items from the MetaManifest: |
| 154 |
-------------------------------------------------------- |
| 155 |
There are two times that this may happen: firstly, immediately after the |
| 156 |
rsync has completed - this has the advantage that the kernel file cache |
| 157 |
is hot, and checking the entire tree can be accomplished quickly. |
| 158 |
Secondly, the MetaManifest should be checked during installation of a |
| 159 |
package. |
| 160 |
|
| 161 |
---------------------------------------------------- |
| 162 |
Procedure for verifying an item in the MetaManifest: |
| 163 |
---------------------------------------------------- |
| 164 |
In the following, I've used term 'M2-verify' to note following the hash |
| 165 |
verification procedures as defined by the Manifest2 format - which |
| 166 |
compromise checking the file length, and that the hashes match. Which |
| 167 |
filetypes may be ignored on missing is discussed in [#GLEP60]. |
| 168 |
|
| 169 |
1. Check the GnuPG signature on the MetaManifest against the keyring of |
| 170 |
automated Gentoo keys. See [#GLEPxx+3] for full details regarding |
| 171 |
verification of GnuPG signatures. |
| 172 |
1. Abort if the signature check fails. |
| 173 |
|
| 174 |
2. Check the Timestamp header. If it is significantly out of date |
| 175 |
compared to the local clock or a trusted source, halt or require |
| 176 |
manual intervention from the user. |
| 177 |
|
| 178 |
3. For a verification of the tree following an rsync: |
| 179 |
|
| 180 |
1. Build a set 'ALL' of every file covered by the rsync. (exclude |
| 181 |
distfiles/, packages/, local/) |
| 182 |
2. M2-verify every entry in the MetaManifest, descending into inferior |
| 183 |
Manifests as needed. Place the relative path of every checked item |
| 184 |
into a set 'COVERED'. |
| 185 |
3. Construct the set 'UNCOVERED' by set-difference between the ALL and |
| 186 |
COVERED sets. |
| 187 |
4. For each file in the UNCOVERED set, assign a Manifest2 filetype. |
| 188 |
5. If the filetype for any file in the UNCOVERED set requires a halt |
| 189 |
on error, abort and display a suitable error. |
| 190 |
6. Completed verification |
| 191 |
|
| 192 |
4. If checking at the installation of a package: |
| 193 |
|
| 194 |
1. M2-verify the entry in MetaManifest for the Manifest |
| 195 |
2. M2-verify all relevant metadata/ contents if metadata/ is being |
| 196 |
used in any way (optionally done before dependency checking). |
| 197 |
3. M2-verifying the contents of the Manifest. |
| 198 |
4. Perform M2-verification of all eclasses and profiles used (both |
| 199 |
directly and indirectly) by the ebuild. |
| 200 |
|
| 201 |
Notes: |
| 202 |
====== |
| 203 |
1. For initial implementations, it is acceptable to check EVERY item in |
| 204 |
the eclass and profiles directory, rather than tracking the exact |
| 205 |
files used by every eclass (see note #2). Later implementations |
| 206 |
should strive to only verify individual eclasses and profiles as |
| 207 |
needed. |
| 208 |
2. Tracking of exact files is of specific significance to the libtool |
| 209 |
eclass, as it stores patches under eclass/ELT-patches, and as such |
| 210 |
that would not be picked up by any tracing of the inherit function. |
| 211 |
This may be alleviated by a later eclass and ebuild variable that |
| 212 |
explicitly declares what files from the tree are used by a package. |
| 213 |
|
| 214 |
==================== |
| 215 |
Implementation Notes |
| 216 |
==================== |
| 217 |
For this portion of the tree-signing work, no actions are required of |
| 218 |
the individual Gentoo developers. They will continue to develop and |
| 219 |
commit as they do presently, and the MetaManifest is added by |
| 220 |
Infrastructure during the tree generation process, and distributed to |
| 221 |
users. |
| 222 |
|
| 223 |
Any scripts generating Manifests and the MetaManifest may find it useful |
| 224 |
to generate multiple levels of Manifests in parallel, and this is |
| 225 |
explicitly permitted, provided that every file in the tree is covered by |
| 226 |
at least one Manifest or the MetaManifest file. The uppermost |
| 227 |
Manifest (MetaManifest) is the only item that does not occur in any |
| 228 |
other Manifest file, but is instead GPG-signed to enable it's |
| 229 |
validation. |
| 230 |
|
| 231 |
-------------------------------------------- |
| 232 |
MetaManifest and the new Manifest2 filetypes |
| 233 |
-------------------------------------------- |
| 234 |
While [#GLEP60] describes the addition of new filetypes, these are NOT |
| 235 |
needed for implementation of the MetaManifest proposal. Without the new |
| 236 |
filetypes, all entries in the MetaManifest would be of type 'MISC'. |
| 237 |
|
| 238 |
---------------------------------------------------- |
| 239 |
Timestamps & Additional distribution of MetaManifest |
| 240 |
---------------------------------------------------- |
| 241 |
As discussed by [C08a,C08b], malicious third-party mirrors may use the |
| 242 |
principles of exclusion and replay to deny an update to clients, while |
| 243 |
at the same time recording the identity of clients to attack. |
| 244 |
|
| 245 |
This should be guarded against by including a timestamp in the header of |
| 246 |
the MetaManifest, as well as distributing the latest MetaManifests by a |
| 247 |
trusted channel. |
| 248 |
|
| 249 |
On all rsync mirrors directly maintained by the Gentoo infrastructure, |
| 250 |
and not on community mirrors, there should be a new module |
| 251 |
'gentoo-portage-metamanifests'. Within this module, all MetaManifests |
| 252 |
for a recent time frame (e.g. one week) should be kept, named as |
| 253 |
"MetaManifest.$TS", where $TS is the timestamp from inside the file. |
| 254 |
The most recent MetaManifest should always be symlinked as |
| 255 |
MetaManifest.current. The possibility of serving the recent |
| 256 |
MetaManifests via HTTPS should also be explored to mitigate |
| 257 |
man-in-the-middle attacks. |
| 258 |
|
| 259 |
The package manager should obtain MetaManifest.current and use it to |
| 260 |
decide is the tree is too out of date per operation #2 of the |
| 261 |
verification process. The decision about freshness should be a |
| 262 |
user-configuration setting, with the ability to override. |
| 263 |
|
| 264 |
-------------------------------- |
| 265 |
MetaManifest size considerations |
| 266 |
-------------------------------- |
| 267 |
With only two levels of Manifests (per-package and top-level), every |
| 268 |
rsync will cause a lot of traffic transferring the modified top-level |
| 269 |
MetaManifest. To reduce this, first-level directory Manifests are |
| 270 |
required. Alternatively, if the distribution method efficiently handles |
| 271 |
small patch-like changes in an existing file, using an uncompressed |
| 272 |
MetaManifest may be acceptable (this would primarily be distributed |
| 273 |
version control systems). Other suggestions in reducing this traffic are |
| 274 |
welcomed. |
| 275 |
|
| 276 |
======================= |
| 277 |
Backwards Compatibility |
| 278 |
======================= |
| 279 |
- There are no backwards compatibility issues, as old versions of |
| 280 |
Portage do not look for a Manifest file at the top level of the tree. |
| 281 |
- Manifest2-aware versions of Portage ignore all entries that they are |
| 282 |
not certain how to handle. Enabling headers and PGP signing to be |
| 283 |
conducted easily. |
| 284 |
|
| 285 |
====== |
| 286 |
Thanks |
| 287 |
====== |
| 288 |
I'd like to thank the following people for input on this GLEP. |
| 289 |
|
| 290 |
- Patrick Lauer (patrick): Prodding me to get all of the tree-signing |
| 291 |
work finished, and helping to edit. |
| 292 |
- Ciaran McCreesh (ciaranm): Paludis Manifest2 |
| 293 |
- Brian Harring (ferringb): pkgcore Manifest2 |
| 294 |
- Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2 |
| 295 |
- Ned Ludd (solar) - Security concept review |
| 296 |
|
| 297 |
========== |
| 298 |
References |
| 299 |
========== |
| 300 |
|
| 301 |
[C08a] Cappos, J et al. (2008). "Package Management Security". |
| 302 |
University of Arizona Technical Report TR08-02. Available online |
| 303 |
from: ftp://ftp.cs.arizona.edu/reports/2008/TR08-02.pdf |
| 304 |
[C08b] Cappos, J et al. (2008). "Attacks on Package Managers" |
| 305 |
Available online at: |
| 306 |
http://www.cs.arizona.edu/people/justin/packagemanagersecurity/ |
| 307 |
[#GLEPxx+2] Future GLEP on Developer Process security. |
| 308 |
[#GLEPxx+3] Future GLEP on GnuPG Policies and Handling. |
| 309 |
|
| 310 |
========= |
| 311 |
Copyright |
| 312 |
========= |
| 313 |
Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be |
| 314 |
distributed only subject to the terms and conditions set forth in the |
| 315 |
Open Publication License, v1.0. |
| 316 |
|
| 317 |
vim: tw=72 ts=2 expandtab: |