--- xml/htdocs/proj/en/glep/glep-0058.html 2008/10/28 07:47:52 1.1 +++ xml/htdocs/proj/en/glep/glep-0058.html 2010/01/31 07:53:41 1.4 @@ -4,7 +4,7 @@ - + GLEP 58 -- Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest @@ -27,9 +27,9 @@ Title:Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest -Version:1.4 +Version:1.7 -Last-Modified:2008/10/28 07:45:27 +Last-Modified:2010/01/31 07:53:30 Author:Robin Hugh Johnson <robbat2 at gentoo.org>, @@ -43,9 +43,9 @@ Created:October 2006 -Updated:November 2007, June 2008, July 2008, October 2008 +Updated:November 2007, June 2008, July 2008, October 2008, January 2010 -Post-History: +Post-History:December 2009, January 2010 @@ -53,31 +53,36 @@

Contents

-

Abstract

+

Abstract

MetaManifest provides a means of verifiable distribution from Gentoo Infrastructure to a user system, while data is conveyed over completely untrusted networks and system, by extending the Manifest2 specification, @@ -85,7 +90,7 @@ Manifests.

-

Motivation

+

Motivation

As part of a comprehensive security plan, we need a way to prove that something originating from Gentoo as an organization (read Gentoo-owned hardware, run by infrastructure), has not been tampered with. This @@ -114,7 +119,7 @@ by deliberate delay, replay [C08a, C08b] or alteration).

-

Specification

+

Specification

For lack of a better name, the following solution should be known as the MetaManifest. Those responsible for the name have already been sacked.

MetaManifest basically contains hashes of every file in the tree, either @@ -127,25 +132,33 @@

In the following, the MetaManifest file is a file named 'Manifest', located at the root of a repository.

-

Procedure for creating the MetaManifest file:

+

Procedure for creating the MetaManifest file:

+
+

Summary:

+

The objective of creating the MetaManifest file(s) is to ensure that +every single file in the tree occurs in at least one Manifest.

+
+
+

Process:

  1. Start at the root of the Gentoo Portage tree (gentoo-x86, although this procedure applies to overlays as well).
  2. Initialize two unordered sets: COVERED, ALL.
      -
    1. 'ALL' will contain every file in the tree.
    2. -
    3. 'COVERED' will contain every file that is mentioned in an existing -Manifest2.
    4. +
    5. 'ALL' shall contain every file that exists in the present tree.
    6. +
    7. 'COVERED' shall contain EVERY file that is mentioned in an existing +Manifest2. If a file is mentioned in a Manifest2, but does not +exist, it must still be included. No files should be excluded.
  3. Traverse the tree, depth-first.
    1. At the top level only, ignore the following directories: distfiles, -packages, local
    2. +packages, local.
    3. If a directory contains a Manifest file, extract all relevant local files from it (presently: AUX, MISC, EBUILD; but should follow the evolution of Manifest2 entry types per [#GLEP60]), and place them into the COVERED set.
    4. Recursively add every file in the directory to the ALL set, -pursusant to the exclusion list as mentioned in [#GLEP60].
    5. +pursuant to the exclusion list as mentioned in [#GLEP60].
  4. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED). @@ -159,7 +172,7 @@ file, so that a MetaManifest may be tied back to a tree as distributed by the rsync mirror system. The string of 'metadata/timestamp.x' should be included to identify this revision -of MetaManifest generation. Eg: +of MetaManifest generation. e.g.: "Timestamp: metadata/timestamp.x: 1215722461 Thu Jul 10 20:41:01 2008 UTC" The package manager MUST not use the identifying string as a filename.
  5. The MetaManifest must ultimately be GnuPG-signed.
      @@ -171,22 +184,28 @@
+
+
+

Notes:

The above does not conflict the proposal contained in GLEP33, which restructure eclasses to include subdirectories and Manifest files, as the Manifest rules above still provide indirect verification for all files after the GLEP33 restructuring if it comes to pass.

-

If other Manifests are added (such as per-category, or protecting -versioned eclases), the size of the MetaManifest will be greatly -reduced, and this specification was written with such a possible future -addition in mind.

+

Additional levels of Manifests are required, such as per-category, and +in the eclasses, profiles and metadata directories. This ensures that a +change to a singular file causes the smallest possible overall change in +the Manifests as propagated. Creation of the additional levels of +Manifests uses the same process as described above, simply starting at a +different root point.

MetaManifest generation will take place as part of the existing process by infrastructure that takes the contents of CVS and prepares it for distribution via rsync, which includes generating metadata. In-tree -Manifest files are not checked at this point, as they are assumed to be -correct.

+Manifest files are not validated at this point, as they are assumed to +be correct.

+
-

Verification of one or more items from the MetaManifest:

+

Verification of one or more items from the MetaManifest:

There are two times that this may happen: firstly, immediately after the rsync has completed - this has the advantage that the kernel file cache is hot, and checking the entire tree can be accomplished quickly. @@ -194,7 +213,7 @@ package.

-

Procedure for verifying an item in the MetaManifest:

+

Procedure for verifying an item in the MetaManifest:

In the following, I've used term 'M2-verify' to note following the hash verification procedures as defined by the Manifest2 format - which compromise checking the file length, and that the hashes match. Which @@ -204,7 +223,7 @@ automated Gentoo keys. See [#GLEPxx+3] for full details regarding verification of GnuPG signatures. 1. Abort if the signature check fails. -

  • Check the Timestamp header. If it is significently out of date +
  • Check the Timestamp header. If it is significantly out of date compared to the local clock or a trusted source, halt or require manual intervention from the user.
  • For a verification of the tree following an rsync:
      @@ -224,15 +243,15 @@
    1. If checking at the installation of a package:
      1. M2-verify the entry in MetaManifest for the Manifest
      2. M2-verify all relevant metadata/ contents if metadata/ is being -used in any way (optionally done before dependancy checking).
      3. +used in any way (optionally done before dependency checking).
      4. M2-verifying the contents of the Manifest.
      5. Perform M2-verification of all eclasses and profiles used (both directly and indirectly) by the ebuild.
    -
    -

    Notes:

    +
    +

    Notes:

    1. For initial implementations, it is acceptable to check EVERY item in the eclass and profiles directory, rather than tracking the exact @@ -249,20 +268,27 @@
    -

    Implementation Notes

    +

    Implementation Notes

    For this portion of the tree-signing work, no actions are required of the individual Gentoo developers. They will continue to develop and commit as they do presently, and the MetaManifest is added by Infrastructure during the tree generation process, and distributed to users.

    +

    Any scripts generating Manifests and the MetaManifest may find it useful +to generate multiple levels of Manifests in parallel, and this is +explicitly permitted, provided that every file in the tree is covered by +at least one Manifest or the MetaManifest file. The uppermost +Manifest (MetaManifest) is the only item that does not occur in any +other Manifest file, but is instead GPG-signed to enable it's +validation.

    -

    MetaManifest and the new Manifest2 filetypes

    +

    MetaManifest and the new Manifest2 filetypes

    While [#GLEP60] describes the addition of new filetypes, these are NOT needed for implementation of the MetaManifest proposal. Without the new filetypes, all entries in the MetaManifest would be of type 'MISC'.

    -

    Timestamps & Additional distribution of MetaManifest

    +

    Timestamps & Additional distribution of MetaManifest

    As discussed by [C08a,C08b], malicious third-party mirrors may use the principles of exclusion and replay to deny an update to clients, while at the same time recording the identity of clients to attack.

    @@ -272,31 +298,31 @@

    On all rsync mirrors directly maintained by the Gentoo infrastructure, and not on community mirrors, there should be a new module 'gentoo-portage-metamanifests'. Within this module, all MetaManifests -for a recent time frame (eg one week) should be kept, named as +for a recent time frame (e.g. one week) should be kept, named as "MetaManifest.$TS", where $TS is the timestamp from inside the file. The most recent MetaManifest should always be symlinked as MetaManifest.current. The possibility of serving the recent -MetaManifests via HTTPS should also be explored to mitigate MitM -attacks.

    +MetaManifests via HTTPS should also be explored to mitigate +man-in-the-middle attacks.

    The package manager should obtain MetaManifest.current and use it to decide is the tree is too out of date per operation #2 of the verification process. The decision about freshness should be a user-configuration setting, with the ability to override.

    -

    MetaManifest size considerations

    +

    MetaManifest size considerations

    With only two levels of Manifests (per-package and top-level), every -rsync will cause a lot of traffic transfering the modified top-level -MetaManifest. To reduce this, per-category Manifests are strongly -recommended. Alternatively, if the distribution method efficently -handles small patch-like changes in an existing file, using an -uncompressed MetaManifest may be acceptable (this would primarily be -distributed version control systems). Other suggestions in reducing this -traffic are welcomed.

    +rsync will cause a lot of traffic transferring the modified top-level +MetaManifest. To reduce this, first-level directory Manifests are +required. Alternatively, if the distribution method efficiently handles +small patch-like changes in an existing file, using an uncompressed +MetaManifest may be acceptable (this would primarily be distributed +version control systems). Other suggestions in reducing this traffic are +welcomed.

    -

    Backwards Compatibility

    +

    Backwards Compatibility

    • There are no backwards compatibility issues, as old versions of Portage do not look for a Manifest file at the top level of the tree.
    • @@ -306,7 +332,7 @@
    -

    Thanks

    +

    Thanks

    I'd like to thank the following people for input on this GLEP.

    • Patrick Lauer (patrick): Prodding me to get all of the tree-signing @@ -318,7 +344,7 @@
    -

    References

    +

    References

    [C08a] Cappos, J et al. (2008). "Package Management Security".
    University of Arizona Technical Report TR08-02. Available online @@ -329,8 +355,8 @@