en/glep/glep-0061.txt

GLEP: 61
Title: Manifest2 compression
Version: $Revision: 1.2 $
Last-Modified: $Date: 2008/10/28 07:45:56 $
Author: Robin Hugh Johnson <robbat2@gentoo.org> 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 44
Created: July 2008
Updated: October 2008, January 2010
Updates: 44
Post-History: December 2009

Abstract
========
Deals with compression of large Manifest2 files.

Motivation
==========
With the introduction of MetaManifest, and full-tree Manifest coverage,
we are faced with the possibility of having very large Manifests.

Preliminary experiments with MetaManifest, show that with just the
existing per-package Manifests, the full MetaManifest, for a tree
including metadata/, exceeds 8MiB in size. Applying common compression
can achieve a 50-60% reduction in this size.

Specification
=============
When searching for a Manifest2 file, if the basename form does not
exist, the package manager should search in the same location using
common compressed suffixes, and use the compressed file in place of the
Manifest2.

gzip, bzip2, lzma, xz should all be supported if available on the given
platform. In the case that multiple versions exist, the package manager
should simply pick one - they should be identical, differing only in
compression.

The Manifest generation process is required to ensure that inconsistent
compressed versions do not exist.

Backwards Compatibility
=======================
The package Manifests should also be maintained as ONLY uncompressed in
CVS.

For processing of all existing per-package Manifests, if compression is
used, it should be done in parallel to the existing Manifests, to
provide for a changeover period. Newer versions of Portage may later
choose to exclude all non-compressed Manifests during emerge --sync if
compressed versions are guaranteed to exist on the servers.

MetaManifests may come into existence as compressed from the start, as
do not have an backwards compatibility issues.

As a side note, this breaks all manual interaction with Manifests
such as grep, and so should only be applied to large Manifest2 files,
such as the MetaManifest. 384KiB is suggested as a arbitary cut-off
point to start generating compressed Manifest2 files.

References
==========
.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
  http://www.gentoo.org/proj/en/glep/glep-0044.html     

Copyright
=========
Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0.

vim: tw=72 ts=2 expandtab:
1	GLEP: 61
2	Title: Manifest2 compression
3	Version: $Revision: 1.2 $
4	Last-Modified: $Date: 2008/10/28 07:45:56 $
5	Author: Robin Hugh Johnson <robbat2@gentoo.org>
6	Status: Draft
7	Type: Standards Track
8	Content-Type: text/x-rst
9	Requires: 44
10	Created: July 2008
11	Updated: October 2008, January 2010
12	Updates: 44
13	Post-History: December 2009
14
15	Abstract
16	========
17	Deals with compression of large Manifest2 files.
18
19	Motivation
20	==========
21	With the introduction of MetaManifest, and full-tree Manifest coverage,
22	we are faced with the possibility of having very large Manifests.
23
24	Preliminary experiments with MetaManifest, show that with just the
25	existing per-package Manifests, the full MetaManifest, for a tree
26	including metadata/, exceeds 8MiB in size. Applying common compression
27	can achieve a 50-60% reduction in this size.
28
29	Specification
30	=============
31	When searching for a Manifest2 file, if the basename form does not
32	exist, the package manager should search in the same location using
33	common compressed suffixes, and use the compressed file in place of the
34	Manifest2.
35
36	gzip, bzip2, lzma, xz should all be supported if available on the given
37	platform. In the case that multiple versions exist, the package manager
38	should simply pick one - they should be identical, differing only in
39	compression.
40
41	The Manifest generation process is required to ensure that inconsistent
42	compressed versions do not exist.
43
44	Backwards Compatibility
45	=======================
46	The package Manifests should also be maintained as ONLY uncompressed in
47	CVS.
48
49	For processing of all existing per-package Manifests, if compression is
50	used, it should be done in parallel to the existing Manifests, to
51	provide for a changeover period. Newer versions of Portage may later
52	choose to exclude all non-compressed Manifests during emerge --sync if
53	compressed versions are guaranteed to exist on the servers.
54
55	MetaManifests may come into existence as compressed from the start, as
56	do not have an backwards compatibility issues.
57
58	As a side note, this breaks all manual interaction with Manifests
59	such as grep, and so should only be applied to large Manifest2 files,
60	such as the MetaManifest. 384KiB is suggested as a arbitary cut-off
61	point to start generating compressed Manifest2 files.
62
63	References
64	==========
65	.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
66	http://www.gentoo.org/proj/en/glep/glep-0044.html
67
68	Copyright
69	=========
70	Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be
71	distributed only subject to the terms and conditions set forth in the
72	Open Publication License, v1.0.
73
74	vim: tw=72 ts=2 expandtab: