/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0061.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (show annotations) (download)
Sun Jan 31 09:55:43 2010 UTC (4 years, 6 months ago) by robbat2
Branch: MAIN
Changes since 1.5: +7 -2 lines
File MIME type: text/plain
Note that GLEP59-61 are independantly useful, even without GLEP58.

1 GLEP: 61
2 Title: Manifest2 compression
3 Version: $Revision: 1.5 $
4 Last-Modified: $Date: 2010/01/31 08:27:05 $
5 Author: Robin Hugh Johnson <robbat2@gentoo.org>
6 Status: Draft
7 Type: Standards Track
8 Content-Type: text/x-rst
9 Requires: 44
10 Created: July 2008
11 Updated: October 2008, January 2010
12 Updates: 44
13 Post-History: December 2009, January 2010
14
15 Abstract
16 ========
17 Deals with compression of large Manifest2 files.
18
19 Motivation
20 ==========
21 With the introduction of MetaManifest, and full-tree Manifest coverage,
22 we are faced with the possibility of having very large Manifests.
23
24 Preliminary experiments with MetaManifest, show that with just the
25 existing per-package Manifests, the full MetaManifest (top-level only,
26 no first-level sub directories), for a tree including metadata/, exceeds
27 8MiB in size. Applying common compression can achieve a 50-60% reduction
28 in this size.
29
30 Additionally, some of the larger already-existing Manifests in the tree
31 can also be reduced.
32
33 This GLEP is not mandatory for the tree-signing specification, but
34 instead helps to cut down the size impact of large Manifest2 files, some
35 of which are already present in the tree. As such, it is also able to
36 stand on it's own.
37
38 Specification
39 =============
40 Creation of compressed Manifests:
41 ---------------------------------
42 32KiB is suggested as a arbitrary cut-off point to start generating
43 compressed Manifest2 files.
44
45 The compression must only applied during the creation of a tree intended
46 for end users. No Manifests stored in a VCS should be compressed in the
47 VCS. For the main gentoo-portage tree, this means that the compressed
48 Manifests should be generated using the CVS to Rsync process.
49
50 The Manifest compression process is required to ensure that inconsistent
51 compressed versions do not exist.
52
53 Validation of Manifests:
54 ------------------------
55 When searching for a Manifest2 file, if the basename form does not
56 exist, the package manager should search in the same location using
57 common compressed suffixes, and use the compressed file in place of the
58 Manifest2.
59
60 gzip, bzip2, lzma, xz should all be supported if available on the given
61 platform. In the case that multiple versions exist, the package manager
62 should simply pick one - they should be identical, differing only in
63 compression.
64
65 Example Results with a 32KiB cut-off, gzip algorithm
66 ====================================================
67 As of 2010/01/30, the suggested cut-off would impact the following 21 existing
68 Manifests, for a saving of nearly 900KiB::
69
70 Size Path
71 65788 app-doc/linux-gazette/Manifest
72 75739 app-office/openoffice-bin/Manifest
73 40534 app-text/texlive-core/Manifest
74 41710 dev-texlive/texlive-bibtexextra/Manifest
75 38197 dev-texlive/texlive-documentation-english/Manifest
76 129610 dev-texlive/texlive-fontsextra/Manifest
77 36022 dev-texlive/texlive-humanities/Manifest
78 686118 dev-texlive/texlive-latexextra/Manifest
79 43392 dev-texlive/texlive-latexrecommended/Manifest
80 33375 dev-texlive/texlive-mathextra/Manifest
81 39781 dev-texlive/texlive-pictures/Manifest
82 69567 dev-texlive/texlive-pstricks/Manifest
83 75460 dev-texlive/texlive-publishers/Manifest
84 50879 dev-texlive/texlive-science/Manifest
85 36711 kde-base/kde-l10n/Manifest
86 36539 media-gfx/bootsplash-themes/Manifest
87 33058 net-fs/autofs/Manifest
88 39781 www-client/firefox-bin/Manifest
89 48983 www-client/icecat/Manifest
90 60213 www-client/mozilla-firefox/Manifest
91 39065 x11-themes/gkrellm-themes/Manifest
92
93
94 Additionally, with the MetaManifest proposal, the following new manifests would
95 also be compressed, for a saving of nearly 4MiB::
96
97 Size Path
98 33442 app-admin/Manifest
99 71073 app-dicts/Manifest
100 35923 app-emacs/Manifest
101 45808 app-misc/Manifest
102 50169 app-text/Manifest
103 112786 dev-java/Manifest
104 65581 dev-libs/Manifest
105 42619 dev-lisp/Manifest
106 182163 dev-perl/Manifest
107 96198 dev-python/Manifest
108 58963 dev-ruby/Manifest
109 59736 dev-util/Manifest
110 58338 eclass/Manifest
111 55749 kde-base/Manifest
112 110064 licenses/Manifest
113 35262 media-gfx/Manifest
114 53995 media-libs/Manifest
115 55607 media-plugins/Manifest
116 71911 media-sound/Manifest
117 34835 media-video/Manifest
118 5747849 metadata/Manifest
119 47452 net-analyzer/Manifest
120 65989 net-misc/Manifest
121 316787 profiles/Manifest
122 67784 sys-apps/Manifest
123 48971 x11-misc/Manifest
124 41475 x11-plugins/Manifest
125
126
127 Backwards Compatibility
128 =======================
129 The package Manifests should also be maintained as ONLY uncompressed in
130 CVS.
131
132 For processing of all existing per-package Manifests, if compression is
133 used, it should be done in parallel to the existing Manifests, to
134 provide for a changeover period. Newer versions of Portage may later
135 choose to exclude all non-compressed Manifests during emerge --sync if
136 compressed versions are guaranteed to exist on the servers.
137
138 MetaManifests may come into existence as compressed from the start, as
139 do not have an backwards compatibility issues.
140
141 As a side note, this breaks all manual interaction with Manifests
142 such as grep, and so should only be applied to large Manifest2 files,
143 such as the MetaManifest.
144
145 References
146 ==========
147 .. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
148 http://www.gentoo.org/proj/en/glep/glep-0044.html
149
150 Copyright
151 =========
152 Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
153 distributed only subject to the terms and conditions set forth in the
154 Open Publication License, v1.0.
155
156 vim: tw=72 ts=2 expandtab:

  ViewVC Help
Powered by ViewVC 1.1.20