--- xml/htdocs/proj/en/glep/glep-0044.html 2007/10/14 17:00:15 1.10 +++ xml/htdocs/proj/en/glep/glep-0044.html 2009/01/11 19:40:56 1.11 @@ -4,10 +4,9 @@ - + GLEP 44 -- Manifest2 format - - + @@ -30,15 +29,15 @@ - + - + - + @@ -47,45 +46,45 @@
-
-

Contents

+
+

Contents

-
-

Abstract

+
+

Abstract

This GLEP proposes a new format for the Portage Manifest and digest file system by unifying both filetypes into one to improve functional and non-functional aspects of the Portage Tree.

-
-

Motivation

-

Please see [1] for a general overview. +

+

Motivation

+

Please see [1] for a general overview. The main long term goals of this proposal are to:

  • Remove the tiny digest files from the tree. They are a major annoyance as on a @@ -98,8 +97,8 @@
  • Difference between filetypes for a more flexible verification system
-
-

Specification

+
+

Specification

The new Manifest format would change the existing format in the following ways:

  • Addition of a filetype specifier, currently planned are

    @@ -126,9 +125,9 @@ <filetype> <filename> <filesize> <chksumtype1> <chksum1> ... <chksumtypen> <chksumn>

    However theses entries will be stored in the existing Manifest files.

    -

    An actual example [6] for a (pure) Manifest2 file..

    -
    -

    Compability Entries

    +

    An actual example [6] for a (pure) Manifest2 file..

    +
    +

    Compability Entries

    To maintain compability with existing portage versions a transition period after is the introduction of the Manifest2 format is required during which portage will not only have to be capable of using existing Manifest and digest files but @@ -138,8 +137,8 @@ the new style entries. For the digest files there are no new entries to care about.

    -
    -

    Scope

    +
    +

    Scope

    It is important to note that this proposal only deals with a change of the format of the digest and Manifest system.

    It does not expand the scope of it to cover eclasses, profiles or anything @@ -147,10 +146,10 @@ the Manifest signing efforts in any way (though the implementations of both might be coupled).

    Also while multiple hash functions will become standard with the proposed -implementation they are not a specific feature of this format [2].

    +implementation they are not a specific feature of this format [2].

    -
    -

    Number of hashes

    +
    +

    Number of hashes

    While using multiple hashes for each file is a major feature of this proposal we have to make sure that the number of hashes listed is limited to avoid an explosion of the Manifest size that would revert the main benefit of this proposal @@ -162,13 +161,13 @@ have any benefit in terms of compability).

    -
    -

    Rationale

    -

    The main goals of the proposal have been listed in the Motivation, here now +

    +

    Rationale

    +

    The main goals of the proposal have been listed in the Motivation, here now the explanation why they are improvements and how the proposed format will accomplish them.

    -
    -

    Removal of digest files

    +
    +

    Removal of digest files

    Normal users that don't use a "tuned" filesystem for the portage tree are wasting several dozen to a few hundred megabytes of disk space with the current system, largely caused by the digest files. @@ -185,17 +184,17 @@ (in the long run), reducing the apparent tree size by about 20%, benefitting both users and the Gentoo infrastructure.

    -
    -

    Reducing redundancy

    +
    +

    Reducing redundancy

    When multiple hashes are used with the current system both the filename and filesize are repeated for every checksum type used as each checksum is standalone. However this doesn't add any functionality and is therefore useless, so the new format removes this redundancy. This is a theoretical improvement at this moment as only one hash function is in -use, but expected to change soon (see [2]).

    +use, but expected to change soon (see [2]).

    -
    -

    Removal of checksum collisions

    +
    +

    Removal of checksum collisions

    The current system theoretically allows for a DIST type file to be recorded in multiple digest files with different sizes and/or checksums. In such a case one version of a package would report a checksum violation while another one @@ -204,8 +203,8 @@ existing system. As the new format lists each file exactly once this would be no longer possible.

    -
    -

    Flexible verification system

    +
    +

    Flexible verification system

    Right now portage verifies the checksum of every file listed in the Manifest before using any file of the package and all DIST files of an ebuild before using that ebuild. This is unnecessary in many cases:

    @@ -226,8 +225,8 @@
-
-

Backwards Compatibility

+
+

Backwards Compatibility

Switching the Manifest system is a task that will need a long transition period like most changes affecting both portage and the tree. In this case the implementation will be rolled out in several phases:

@@ -269,10 +268,10 @@ to be completed due to compability issues mentioned above some benefits of the system can selectively be used as soon as step 2) is completed.

-
-

Other problems

-
-

Impacts on infrastructure

+
+

Other problems

+
+

Impacts on infrastructure

While one long term goal of this proposal is to reduce the size of the tree and therefore make life for the Gentoo Infrastructure easier this will only take effect once the implementation is rolled out completely. In the meantime @@ -282,20 +281,20 @@ propagation of Manifest2 capable portage versions among devs or the update rate of the tree. It has been suggested that Manifest files that are not gpg signed could be mass converted in one step, this could certainly help but only -to some degree (according to a recent research [3] about 40% of +to some degree (according to a recent research [3] about 40% of all Manifests in the tree are signed, but this number hasn't been verified).

-
-

Reference Implementation

+
+

Reference Implementation

A patch for a prototype implementation of Manifest2 verification and partial -generation has been posted at [4], it will be reworked before +generation has been posted at [4], it will be reworked before being considered for inclusion in portage. However it shows that adding support for verification is quite simple, but generation is a bit tricky and will therefore be implemented later.

-
-

Options

+
+

Options

Some things have been considered for this GLEP but aren't part of the proposal yet for various reasons:

    @@ -317,8 +316,8 @@ should be kept.
-
-

Credits

+
+

Credits

Thanks to the following persons for their input on or related to this GLEP (even though they might not have known it): Ned Ludd (solar), Brian Harring (ferringb), Jason Stubbs (jstubbs), @@ -327,59 +326,58 @@ resistent enough to be able to handle this change without too many transition problems.

-