en/glep/glep-0061.txt

GLEP: 61
Title: Manifest2 compression
Version: $Revision: 1.3 $
Last-Modified: $Date: 2010/01/13 01:14:35 $
Author: Robin Hugh Johnson <robbat2@gentoo.org> 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 44
Created: July 2008
Updated: October 2008, January 2010
Updates: 44
Post-History: December 2009

Abstract
========
Deals with compression of large Manifest2 files.

Motivation
==========
With the introduction of MetaManifest, and full-tree Manifest coverage,
we are faced with the possibility of having very large Manifests.

Preliminary experiments with MetaManifest, show that with just the
existing per-package Manifests, the full MetaManifest, for a tree
including metadata/, exceeds 8MiB in size. Applying common compression
can achieve a 50-60% reduction in this size.

Specification
=============
When searching for a Manifest2 file, if the basename form does not
exist, the package manager should search in the same location using
common compressed suffixes, and use the compressed file in place of the
Manifest2.

gzip, bzip2, lzma, xz should all be supported if available on the given
platform. In the case that multiple versions exist, the package manager
should simply pick one - they should be identical, differing only in
compression.

The Manifest generation process is required to ensure that inconsistent
compressed versions do not exist.

Backwards Compatibility
=======================
The package Manifests should also be maintained as ONLY uncompressed in
CVS.

For processing of all existing per-package Manifests, if compression is
used, it should be done in parallel to the existing Manifests, to
provide for a changeover period. Newer versions of Portage may later
choose to exclude all non-compressed Manifests during emerge --sync if
compressed versions are guaranteed to exist on the servers.

MetaManifests may come into existence as compressed from the start, as
do not have an backwards compatibility issues.

As a side note, this breaks all manual interaction with Manifests
such as grep, and so should only be applied to large Manifest2 files,
such as the MetaManifest. 384KiB is suggested as a arbitrary cut-off
point to start generating compressed Manifest2 files.

References
==========
.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
  http://www.gentoo.org/proj/en/glep/glep-0044.html     

Copyright
=========
Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0.

vim: tw=72 ts=2 expandtab:
1	cardoe	1.1	GLEP: 61
2			Title: Manifest2 compression
3	robbat2	1.4	Version: $Revision: 1.3 $
4			Last-Modified: $Date: 2010/01/13 01:14:35 $
5	cardoe	1.1	Author: Robin Hugh Johnson <robbat2@gentoo.org>
6			Status: Draft
7			Type: Standards Track
8			Content-Type: text/x-rst
9			Requires: 44
10			Created: July 2008
11	robbat2	1.3	Updated: October 2008, January 2010
12	cardoe	1.1	Updates: 44
13	robbat2	1.3	Post-History: December 2009
14	cardoe	1.1
15			Abstract
16			========
17			Deals with compression of large Manifest2 files.
18
19			Motivation
20			==========
21			With the introduction of MetaManifest, and full-tree Manifest coverage,
22			we are faced with the possibility of having very large Manifests.
23
24			Preliminary experiments with MetaManifest, show that with just the
25			existing per-package Manifests, the full MetaManifest, for a tree
26			including metadata/, exceeds 8MiB in size. Applying common compression
27			can achieve a 50-60% reduction in this size.
28
29			Specification
30			=============
31			When searching for a Manifest2 file, if the basename form does not
32			exist, the package manager should search in the same location using
33			common compressed suffixes, and use the compressed file in place of the
34			Manifest2.
35
36	robbat2	1.3	gzip, bzip2, lzma, xz should all be supported if available on the given
37	cardoe	1.1	platform. In the case that multiple versions exist, the package manager
38			should simply pick one - they should be identical, differing only in
39			compression.
40
41			The Manifest generation process is required to ensure that inconsistent
42			compressed versions do not exist.
43
44			Backwards Compatibility
45			=======================
46			The package Manifests should also be maintained as ONLY uncompressed in
47			CVS.
48
49			For processing of all existing per-package Manifests, if compression is
50			used, it should be done in parallel to the existing Manifests, to
51			provide for a changeover period. Newer versions of Portage may later
52			choose to exclude all non-compressed Manifests during emerge --sync if
53			compressed versions are guaranteed to exist on the servers.
54
55			MetaManifests may come into existence as compressed from the start, as
56			do not have an backwards compatibility issues.
57
58			As a side note, this breaks all manual interaction with Manifests
59			such as grep, and so should only be applied to large Manifest2 files,
60	robbat2	1.4	such as the MetaManifest. 384KiB is suggested as a arbitrary cut-off
61	cardoe	1.1	point to start generating compressed Manifest2 files.
62
63			References
64			==========
65			.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
66			http://www.gentoo.org/proj/en/glep/glep-0044.html
67
68			Copyright
69			=========
70	robbat2	1.4	Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
71	cardoe	1.1	distributed only subject to the terms and conditions set forth in the
72			Open Publication License, v1.0.
73
74			vim: tw=72 ts=2 expandtab: