|author||Michał Górny <email@example.com>||2017-09-14 23:14:39 +0200|
|committer||Ulrich Müller <firstname.lastname@example.org>||2017-10-09 12:08:51 +0200|
|parent||glep-0045: Mark Final since GLEP 1 now uses ISO 8601 dates (diff)|
Rename all GLEPs to .rst
Diffstat (limited to 'glep-0044.rst')
1 files changed, 327 insertions, 0 deletions
diff --git a/glep-0044.rst b/glep-0044.rst
new file mode 100644
@@ -0,0 +1,327 @@
+Title: Manifest2 format
+Author: Marius Mauch <email@example.com>,
+Type: Standards Track
+Post-History: 06-Dec-2005, 23-Jan-2006, 3-Sep-2006
+This GLEP proposes a new format for the Portage Manifest and digest file system
+by unifying both filetypes into one to improve functional and non-functional
+aspects of the Portage Tree.
+Please see [#reorg-thread]_ for a general overview.
+The main long term goals of this proposal are to:
+- Remove the tiny digest files from the tree. They are a major annoyance as on a
+ typical configuration they waste a lot of disk space and the simple transmission
+ of the names for all digest files during a ``emerge --sync`` needs a substantial
+ amount of bandwidth.
+- Reduce redundancy when multiple hash functions are used
+- Remove potential for checksum collisions if a file is recorded in more than one
+ digest file
+- Difference between filetypes for a more flexible verification system
+The new Manifest format would change the existing format in the following ways:
+- Addition of a filetype specifier, currently planned are
+ * ``AUX`` for files directly used by ebuilds (e.g. patches or initscripts),
+ located in the ``files/`` subdirectory
+ * ``EBUILD`` for all ebuilds
+ * ``MISC`` for files not directly used by ebuilds like ``ChangeLog`` or
+ ``metadata.xml`` files
+ * ``DIST`` for release tarballs recorded in the ``SRC_URI`` variable of an ebuild,
+ these were previously recorded in the digest files
+ Future portage improvements might extend this list (for example with types
+ relevant for eclasses or profiles)
+- Only have one line per file listing all information instead of one line per
+ file and checksum type
+- Remove the separated digest-* files in the ``files/`` subdirectory
+Each line in the new format has the following format:
+ <filetype> <filename> <filesize> <chksumtype1> <chksum1> ... <chksumtypen> <chksumn>
+However theses entries will be stored in the existing Manifest files.
+An `actual example`__ for a (pure) Manifest2 file..
+.. __: glep-0044-extras/manifest2-example.txt
+To maintain compatibility with existing portage versions a transition period after
+is the introduction of the Manifest2 format is required during which portage
+will not only have to be capable of using existing Manifest and digest files but
+also generate them in addition to the new entries.
+Fortunately this can be accomplished by simply mixing old and new style entries
+in one file for the Manifest files, existing portage versions will simply ignore
+the new style entries. For the digest files there are no new entries to care
+It is important to note that this proposal only deals with a change of the
+format of the digest and Manifest system.
+It does not expand the scope of it to cover eclasses, profiles or anything
+else not already covered by the Manifest system, it also doesn't affect
+the Manifest signing efforts in any way (though the implementations of both
+might be coupled).
+Also while multiple hash functions will become standard with the proposed
+implementation they are not a specific feature of this format [#multi-hash-thread]_.
+Number of hashes
+While using multiple hashes for each file is a major feature of this proposal
+we have to make sure that the number of hashes listed is limited to avoid
+an explosion of the Manifest size that would revert the main benefit of this proposal
+(reducing tree size). Therefore the number of hashes that will be generated
+will be limited to three different hash functions. For compatibility though we
+have to rely on at least one hash function to always be present, this proposal
+suggest to use SHA1 for this purpose (as it is supposed to be more secure than MD5
+and currently only SHA1 and MD5 are directly available in python, also MD5 doesn't
+have any benefit in terms of compatibility).
+The main goals of the proposal have been listed in the `Motivation`_, here now
+the explanation why they are improvements and how the proposed format will
+Removal of digest files
+Normal users that don't use a "tuned" filesystem for the portage tree are
+wasting several dozen to a few hundred megabytes of disk space with the current
+system, largely caused by the digest files.
+This is due to the filesystem overhead present in most filesystems that
+have a standard blocksize of four kilobytes while most digest files are under
+one kilobyte in size, so this results in approximately a waste of three kilobytes
+per digest file (likely even more). At the time of this writing the tree contains
+roughly 22.000 digest files, so the overall waste caused by digest files is
+estimated at about 70-100 megabytes.
+Furthermore it is assumed that this will also reduce the disk space wasted by
+the Manifest files as they now contain more content, but this hasn't been
+By unifying the digest files with the Manifest these tiny files are eliminated
+(in the long run), reducing the apparent tree size by about 20%, benefiting
+both users and the Gentoo infrastructure.
+When multiple hashes are used with the current system
+both the filename and filesize are repeated for every checksum type used as each
+checksum is standalone. However this doesn't add any functionality and is
+therefore useless, so the new format removes this redundancy.
+This is a theoretical improvement at this moment as only one hash function is in
+use, but expected to change soon (see [#multi-hash-thread]_).
+Removal of checksum collisions
+The current system theoretically allows for a ``DIST`` type file to be recorded
+in multiple digest files with different sizes and/or checksums. In such a case
+one version of a package would report a checksum violation while another one
+would not. This could create confusion and uncertainty among users.
+So far this case hasn't been observed, but it can't be ruled out with the
+As the new format lists each file exactly once this would be no longer possible.
+Flexible verification system
+Right now portage verifies the checksum of every file listed in the Manifest
+before using any file of the package and all ``DIST`` files of an ebuild
+before using that ebuild. This is unnecessary in many cases:
+- During the "depend" phase (when the ebuild metadata is generated) only
+ files of type ``EBUILD`` are used, so verifying the other types isn't
+ necessary. Theoretically it is possible for an ebuild to include other
+ files like those of type ``AUX`` at this phase, but that would be a
+ major QA violation and should never occur, so it can be ignored here.
+ It is also not a security concern as the ebuild is verified before parsing
+ it, so each manipulation would show up.
+- Generally files of type ``MISC`` don't need to be verified as they are
+ only used in very specific situations, aren't executed (just parsed at most)
+ and don't affect the package build process.
+- Files of type ``DIST`` only need to be verified directly after fetching and
+ before unpacking them (which often will be one step), not every time their
+ associated ebuild is used.
+Switching the Manifest system is a task that will need a long transition period
+like most changes affecting both portage and the tree. In this case the
+implementation will be rolled out in several phases:
+1. Add support for verification of Manifest2 entries in portage
+2. Enable generation of Manifest2 entries in addition to the current system
+3. Ignore digests during ``emerge --sync`` to get the size-benefit clientside.
+ This step may be omitted if the following steps are expected to follow soon.
+4. Disable generation of entries for the current system
+5. Remove all traces of the current system from the tree (serverside)
+Each step has its own issues. While 1) and 2) can be implemented without any
+compatibility problems all later steps have a major impact:
+- Step 3) can only be implemented when the whole tree is Manifest2 ready
+ (ideally speaking, practically the requirement will be more like 95% coverage
+ with the expectation that for the remaining 5% either bugs will be filed after
+ step 3) is completed or they'll be updated at step 5).
+- Steps 4) and 5) will render all portage versions without Manifest2 support
+ basically useless (users would have to regenerate the digest and Manifest
+ for each package before being able to merge it), so this requires a almost
+ 100% coverage of the userbase with Manifest2 capable portage versions
+ (with step 1) completely implemented).
+Another problem is that some steps affect different targets:
+- Steps 1) and 3) target portage versions used by users
+- Steps 2) and 4) target portage versions used by devs
+- Step 5) targets the portage tree on the cvs server
+While it is relatively easy to get all devs to use a new portage version this is
+practically impossible with users as some don't update their systems regularly.
+While six months are probably sufficient to reach a 95% coverage one year is
+estimated to reach an almost-complete coverage. All times are relative to the
+stable-marking of a compatible portage version.
+No timeframe for implementation is presented here as it is highly dependent
+on the completion of each step.
+In summary it can be said that while a full conversion will take over a year
+to be completed due to compatibility issues mentioned above some benefits of the
+system can selectively be used as soon as step 2) is completed.
+Impacts on infrastructure
+While one long term goal of this proposal is to reduce the size of the tree
+and therefore make life for the Gentoo Infrastructure easier this will only
+take effect once the implementation is rolled out completely. In the meantime
+however it will increase the tree size due to keeping checksums in both formats.
+It's not possible to give a usable estimate on the degree of the increase as
+it depends on many variables such as the exact implementation timeframe,
+propagation of Manifest2 capable portage versions among devs or the update
+rate of the tree. It has been suggested that Manifest files that are not gpg
+signed could be mass converted in one step, this could certainly help but only
+to some degree (according to a recent research [#gpg-numbers]_ about 40% of
+all Manifests in the tree are signed, but this number hasn't been verified).
+A patch for a prototype implementation of Manifest2 verification and partial
+generation has been posted at [#manifest2-patch]_, it will be reworked before
+being considered for inclusion in portage. However it shows that adding support
+for verification is quite simple, but generation is a bit tricky and will
+therefore be implemented later.
+Some things have been considered for this GLEP but aren't part of the proposal
+yet for various reasons:
+- timestamp field: the author has considered adding a timestamp field for
+ each entry to list the time the entry was created. However so far no practical
+ use for such a feature has been found.
+- convert size field into checksum: Another idea was to treat the size field
+ like any other checksum. But so far no real benefit (other than a slightly
+ more modular implementation) for this has been seen while it has several
+ drawbacks: For once, unlike checksums, the size field is definitely required
+ for all ``DIST`` files, also it would slightly increase the length of
+ each entry by adding a ``SIZE`` keyword.
+- removal of the ``MISC`` type: It has been suggested to completely drop
+ entries of type ``MISC``. This would result in a minor space reduction
+ (its rather unlikely to free any blocks) but completely remove the ability
+ to check these files for integrity. While they don't influence portage
+ or packages directly they can contain viable information for users, so
+ the author has the opinion that at least the option for integrity checks
+ should be kept.
+Thanks to the following persons for their input on or related to this GLEP
+(even though they might not have known it):
+Ned Ludd (solar), Brian Harring (ferringb), Jason Stubbs (jstubbs),
+Robin H. Johnson (robbat2), Aron Griffis (agriffis)
+Also thanks to Nicholas Jones (carpaski) to make the current Manifest system
+resistent enough to be able to handle this change without too many transition
+.. [#reorg-thread] http://thread.gmane.org/gmane.linux.gentoo.devel/21920
+.. [#multi-hash-thread] http://thread.gmane.org/gmane.linux.gentoo.devel/33434
+.. [#gpg-numbers] gentoo-core mailing list, topic "Gentoo key signing practices
+ and official Gentoo keyring", Message-ID <20051117075838.GB15734@curie-int.vc.shawcable.net>
+.. [#manifest2-patch] http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1374
+.. [#manifest2-example] http://www.gentoo.org/proj/en/glep/glep-0044-extras/manifest2-example
+This work is licensed under the Creative Commons Attribution-ShareAlike 3.0
+Unported License. To view a copy of this license, visit