/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0061.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (hide annotations) (download)
Sun Jan 31 09:55:43 2010 UTC (4 years, 2 months ago) by robbat2
Branch: MAIN
Changes since 1.5: +7 -2 lines
File MIME type: text/plain
Note that GLEP59-61 are independantly useful, even without GLEP58.

1 cardoe 1.1 GLEP: 61
2     Title: Manifest2 compression
3 robbat2 1.6 Version: $Revision: 1.5 $
4     Last-Modified: $Date: 2010/01/31 08:27:05 $
5 cardoe 1.1 Author: Robin Hugh Johnson <robbat2@gentoo.org>
6     Status: Draft
7     Type: Standards Track
8     Content-Type: text/x-rst
9     Requires: 44
10     Created: July 2008
11 robbat2 1.3 Updated: October 2008, January 2010
12 cardoe 1.1 Updates: 44
13 robbat2 1.5 Post-History: December 2009, January 2010
14 cardoe 1.1
15     Abstract
16     ========
17     Deals with compression of large Manifest2 files.
18    
19     Motivation
20     ==========
21     With the introduction of MetaManifest, and full-tree Manifest coverage,
22     we are faced with the possibility of having very large Manifests.
23    
24     Preliminary experiments with MetaManifest, show that with just the
25 robbat2 1.5 existing per-package Manifests, the full MetaManifest (top-level only,
26     no first-level sub directories), for a tree including metadata/, exceeds
27     8MiB in size. Applying common compression can achieve a 50-60% reduction
28     in this size.
29    
30     Additionally, some of the larger already-existing Manifests in the tree
31     can also be reduced.
32 cardoe 1.1
33 robbat2 1.6 This GLEP is not mandatory for the tree-signing specification, but
34     instead helps to cut down the size impact of large Manifest2 files, some
35     of which are already present in the tree. As such, it is also able to
36     stand on it's own.
37    
38 cardoe 1.1 Specification
39     =============
40 robbat2 1.5 Creation of compressed Manifests:
41     ---------------------------------
42     32KiB is suggested as a arbitrary cut-off point to start generating
43     compressed Manifest2 files.
44    
45     The compression must only applied during the creation of a tree intended
46     for end users. No Manifests stored in a VCS should be compressed in the
47     VCS. For the main gentoo-portage tree, this means that the compressed
48     Manifests should be generated using the CVS to Rsync process.
49    
50     The Manifest compression process is required to ensure that inconsistent
51     compressed versions do not exist.
52    
53     Validation of Manifests:
54     ------------------------
55 cardoe 1.1 When searching for a Manifest2 file, if the basename form does not
56     exist, the package manager should search in the same location using
57     common compressed suffixes, and use the compressed file in place of the
58     Manifest2.
59    
60 robbat2 1.3 gzip, bzip2, lzma, xz should all be supported if available on the given
61 cardoe 1.1 platform. In the case that multiple versions exist, the package manager
62     should simply pick one - they should be identical, differing only in
63     compression.
64    
65 robbat2 1.5 Example Results with a 32KiB cut-off, gzip algorithm
66     ====================================================
67     As of 2010/01/30, the suggested cut-off would impact the following 21 existing
68     Manifests, for a saving of nearly 900KiB::
69    
70     Size Path
71     65788 app-doc/linux-gazette/Manifest
72     75739 app-office/openoffice-bin/Manifest
73     40534 app-text/texlive-core/Manifest
74     41710 dev-texlive/texlive-bibtexextra/Manifest
75     38197 dev-texlive/texlive-documentation-english/Manifest
76     129610 dev-texlive/texlive-fontsextra/Manifest
77     36022 dev-texlive/texlive-humanities/Manifest
78     686118 dev-texlive/texlive-latexextra/Manifest
79     43392 dev-texlive/texlive-latexrecommended/Manifest
80     33375 dev-texlive/texlive-mathextra/Manifest
81     39781 dev-texlive/texlive-pictures/Manifest
82     69567 dev-texlive/texlive-pstricks/Manifest
83     75460 dev-texlive/texlive-publishers/Manifest
84     50879 dev-texlive/texlive-science/Manifest
85     36711 kde-base/kde-l10n/Manifest
86     36539 media-gfx/bootsplash-themes/Manifest
87     33058 net-fs/autofs/Manifest
88     39781 www-client/firefox-bin/Manifest
89     48983 www-client/icecat/Manifest
90     60213 www-client/mozilla-firefox/Manifest
91     39065 x11-themes/gkrellm-themes/Manifest
92    
93    
94     Additionally, with the MetaManifest proposal, the following new manifests would
95     also be compressed, for a saving of nearly 4MiB::
96    
97     Size Path
98     33442 app-admin/Manifest
99     71073 app-dicts/Manifest
100     35923 app-emacs/Manifest
101     45808 app-misc/Manifest
102     50169 app-text/Manifest
103     112786 dev-java/Manifest
104     65581 dev-libs/Manifest
105     42619 dev-lisp/Manifest
106     182163 dev-perl/Manifest
107     96198 dev-python/Manifest
108     58963 dev-ruby/Manifest
109     59736 dev-util/Manifest
110     58338 eclass/Manifest
111     55749 kde-base/Manifest
112     110064 licenses/Manifest
113     35262 media-gfx/Manifest
114     53995 media-libs/Manifest
115     55607 media-plugins/Manifest
116     71911 media-sound/Manifest
117     34835 media-video/Manifest
118     5747849 metadata/Manifest
119     47452 net-analyzer/Manifest
120     65989 net-misc/Manifest
121     316787 profiles/Manifest
122     67784 sys-apps/Manifest
123     48971 x11-misc/Manifest
124     41475 x11-plugins/Manifest
125    
126 cardoe 1.1
127     Backwards Compatibility
128     =======================
129     The package Manifests should also be maintained as ONLY uncompressed in
130     CVS.
131    
132     For processing of all existing per-package Manifests, if compression is
133     used, it should be done in parallel to the existing Manifests, to
134     provide for a changeover period. Newer versions of Portage may later
135     choose to exclude all non-compressed Manifests during emerge --sync if
136     compressed versions are guaranteed to exist on the servers.
137    
138     MetaManifests may come into existence as compressed from the start, as
139     do not have an backwards compatibility issues.
140    
141     As a side note, this breaks all manual interaction with Manifests
142     such as grep, and so should only be applied to large Manifest2 files,
143 robbat2 1.5 such as the MetaManifest.
144 cardoe 1.1
145     References
146     ==========
147     .. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
148     http://www.gentoo.org/proj/en/glep/glep-0044.html
149    
150     Copyright
151     =========
152 robbat2 1.4 Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
153 cardoe 1.1 distributed only subject to the terms and conditions set forth in the
154     Open Publication License, v1.0.
155    
156     vim: tw=72 ts=2 expandtab:

  ViewVC Help
Powered by ViewVC 1.1.20