Title:Manifest2 compression
Last-Modified:2008/10/28 07:45:56
Author:Robin Hugh Johnson <robbat2 at>
Type:Standards Track
Created:July 2008
Updated:October 2008



Deals with compression of large Manifest2 files.


With the introduction of MetaManifest, and full-tree Manifest coverage, we are faced with the possibility of having very large Manifests.

Preliminary experiments with MetaManifest, show that with just the existing per-package Manifests, the full MetaManifest, for a tree including metadata/, exceeds 8MiB in size. Applying common compression can achieve a 50-60% reduction in this size.


When searching for a Manifest2 file, if the basename form does not exist, the package manager should search in the same location using common compressed suffixes, and use the compressed file in place of the Manifest2.

gzip, bzip2, lzma should all be supported if available on the given platform. In the case that multiple versions exist, the package manager should simply pick one - they should be identical, differing only in compression.

The Manifest generation process is required to ensure that inconsistent compressed versions do not exist.

Backwards Compatibility

The package Manifests should also be maintained as ONLY uncompressed in CVS.

For processing of all existing per-package Manifests, if compression is used, it should be done in parallel to the existing Manifests, to provide for a changeover period. Newer versions of Portage may later choose to exclude all non-compressed Manifests during emerge --sync if compressed versions are guaranteed to exist on the servers.

MetaManifests may come into existence as compressed from the start, as do not have an backwards compatibility issues.

As a side note, this breaks all manual interaction with Manifests such as grep, and so should only be applied to large Manifest2 files, such as the MetaManifest. 384KiB is suggested as a arbitary cut-off point to start generating compressed Manifest2 files.


[1]Mauch, M. (2005) GLEP44 - Manifest2 format.