/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.txt
Gentoo

Diff of /xml/htdocs/proj/en/glep/glep-0061.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.3 Revision 1.6
1GLEP: 61 1GLEP: 61
2Title: Manifest2 compression 2Title: Manifest2 compression
3Version: $Revision: 1.3 $ 3Version: $Revision: 1.6 $
4Last-Modified: $Date: 2010/01/13 01:14:35 $ 4Last-Modified: $Date: 2010/01/31 09:55:43 $
5Author: Robin Hugh Johnson <robbat2@gentoo.org> 5Author: Robin Hugh Johnson <robbat2@gentoo.org>
6Status: Draft 6Status: Draft
7Type: Standards Track 7Type: Standards Track
8Content-Type: text/x-rst 8Content-Type: text/x-rst
9Requires: 44 9Requires: 44
10Created: July 2008 10Created: July 2008
11Updated: October 2008, January 2010 11Updated: October 2008, January 2010
12Updates: 44 12Updates: 44
13Post-History: December 2009 13Post-History: December 2009, January 2010
14 14
15Abstract 15Abstract
16======== 16========
17Deals with compression of large Manifest2 files. 17Deals with compression of large Manifest2 files.
18 18
… …
20========== 20==========
21With the introduction of MetaManifest, and full-tree Manifest coverage, 21With the introduction of MetaManifest, and full-tree Manifest coverage,
22we are faced with the possibility of having very large Manifests. 22we are faced with the possibility of having very large Manifests.
23 23
24Preliminary experiments with MetaManifest, show that with just the 24Preliminary experiments with MetaManifest, show that with just the
25existing per-package Manifests, the full MetaManifest, for a tree 25existing per-package Manifests, the full MetaManifest (top-level only,
26including metadata/, exceeds 8MiB in size. Applying common compression 26no first-level sub directories), for a tree including metadata/, exceeds
27can achieve a 50-60% reduction in this size. 278MiB in size. Applying common compression can achieve a 50-60% reduction
28in this size.
29
30Additionally, some of the larger already-existing Manifests in the tree
31can also be reduced.
32
33This GLEP is not mandatory for the tree-signing specification, but
34instead helps to cut down the size impact of large Manifest2 files, some
35of which are already present in the tree. As such, it is also able to
36stand on it's own.
28 37
29Specification 38Specification
30============= 39=============
40Creation of compressed Manifests:
41---------------------------------
4232KiB is suggested as a arbitrary cut-off point to start generating
43compressed Manifest2 files.
44
45The compression must only applied during the creation of a tree intended
46for end users. No Manifests stored in a VCS should be compressed in the
47VCS. For the main gentoo-portage tree, this means that the compressed
48Manifests should be generated using the CVS to Rsync process.
49
50The Manifest compression process is required to ensure that inconsistent
51compressed versions do not exist.
52
53Validation of Manifests:
54------------------------
31When searching for a Manifest2 file, if the basename form does not 55When searching for a Manifest2 file, if the basename form does not
32exist, the package manager should search in the same location using 56exist, the package manager should search in the same location using
33common compressed suffixes, and use the compressed file in place of the 57common compressed suffixes, and use the compressed file in place of the
34Manifest2. 58Manifest2.
35 59
36gzip, bzip2, lzma, xz should all be supported if available on the given 60gzip, bzip2, lzma, xz should all be supported if available on the given
37platform. In the case that multiple versions exist, the package manager 61platform. In the case that multiple versions exist, the package manager
38should simply pick one - they should be identical, differing only in 62should simply pick one - they should be identical, differing only in
39compression. 63compression.
40 64
41The Manifest generation process is required to ensure that inconsistent 65Example Results with a 32KiB cut-off, gzip algorithm
42compressed versions do not exist. 66====================================================
67As of 2010/01/30, the suggested cut-off would impact the following 21 existing
68Manifests, for a saving of nearly 900KiB::
69
70 Size Path
71 65788 app-doc/linux-gazette/Manifest
72 75739 app-office/openoffice-bin/Manifest
73 40534 app-text/texlive-core/Manifest
74 41710 dev-texlive/texlive-bibtexextra/Manifest
75 38197 dev-texlive/texlive-documentation-english/Manifest
76 129610 dev-texlive/texlive-fontsextra/Manifest
77 36022 dev-texlive/texlive-humanities/Manifest
78 686118 dev-texlive/texlive-latexextra/Manifest
79 43392 dev-texlive/texlive-latexrecommended/Manifest
80 33375 dev-texlive/texlive-mathextra/Manifest
81 39781 dev-texlive/texlive-pictures/Manifest
82 69567 dev-texlive/texlive-pstricks/Manifest
83 75460 dev-texlive/texlive-publishers/Manifest
84 50879 dev-texlive/texlive-science/Manifest
85 36711 kde-base/kde-l10n/Manifest
86 36539 media-gfx/bootsplash-themes/Manifest
87 33058 net-fs/autofs/Manifest
88 39781 www-client/firefox-bin/Manifest
89 48983 www-client/icecat/Manifest
90 60213 www-client/mozilla-firefox/Manifest
91 39065 x11-themes/gkrellm-themes/Manifest
92
93
94Additionally, with the MetaManifest proposal, the following new manifests would
95also be compressed, for a saving of nearly 4MiB::
96
97 Size Path
98 33442 app-admin/Manifest
99 71073 app-dicts/Manifest
100 35923 app-emacs/Manifest
101 45808 app-misc/Manifest
102 50169 app-text/Manifest
103 112786 dev-java/Manifest
104 65581 dev-libs/Manifest
105 42619 dev-lisp/Manifest
106 182163 dev-perl/Manifest
107 96198 dev-python/Manifest
108 58963 dev-ruby/Manifest
109 59736 dev-util/Manifest
110 58338 eclass/Manifest
111 55749 kde-base/Manifest
112 110064 licenses/Manifest
113 35262 media-gfx/Manifest
114 53995 media-libs/Manifest
115 55607 media-plugins/Manifest
116 71911 media-sound/Manifest
117 34835 media-video/Manifest
118 5747849 metadata/Manifest
119 47452 net-analyzer/Manifest
120 65989 net-misc/Manifest
121 316787 profiles/Manifest
122 67784 sys-apps/Manifest
123 48971 x11-misc/Manifest
124 41475 x11-plugins/Manifest
125
43 126
44Backwards Compatibility 127Backwards Compatibility
45======================= 128=======================
46The package Manifests should also be maintained as ONLY uncompressed in 129The package Manifests should also be maintained as ONLY uncompressed in
47CVS. 130CVS.
… …
55MetaManifests may come into existence as compressed from the start, as 138MetaManifests may come into existence as compressed from the start, as
56do not have an backwards compatibility issues. 139do not have an backwards compatibility issues.
57 140
58As a side note, this breaks all manual interaction with Manifests 141As a side note, this breaks all manual interaction with Manifests
59such as grep, and so should only be applied to large Manifest2 files, 142such as grep, and so should only be applied to large Manifest2 files,
60such as the MetaManifest. 384KiB is suggested as a arbitary cut-off 143such as the MetaManifest.
61point to start generating compressed Manifest2 files.
62 144
63References 145References
64========== 146==========
65.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format. 147.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
66 http://www.gentoo.org/proj/en/glep/glep-0044.html 148 http://www.gentoo.org/proj/en/glep/glep-0044.html
67 149
68Copyright 150Copyright
69========= 151=========
70Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be 152Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
71distributed only subject to the terms and conditions set forth in the 153distributed only subject to the terms and conditions set forth in the
72Open Publication License, v1.0. 154Open Publication License, v1.0.
73 155
74vim: tw=72 ts=2 expandtab: 156vim: tw=72 ts=2 expandtab:
Legend:
Removed from v.1.3  
changed lines
  Added in v.1.6
  ViewVC Help
Powered by ViewVC 1.1.20