/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.txt
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0061.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.5 - (hide annotations) (download)
Sun Jan 31 08:27:05 2010 UTC (4 years, 7 months ago) by robbat2
Branch: MAIN
Changes since 1.4: +87 -10 lines
File MIME type: text/plain
Revise GLEP61 per Calchan questions: Cutoff point moved down to 32KiB, and example size results included; Split specification into generation and validation phases.

1 cardoe 1.1 GLEP: 61
2     Title: Manifest2 compression
3 robbat2 1.5 Version: $Revision: 1.4 $
4     Last-Modified: $Date: 2010/01/13 03:26:53 $
5 cardoe 1.1 Author: Robin Hugh Johnson <robbat2@gentoo.org>
6     Status: Draft
7     Type: Standards Track
8     Content-Type: text/x-rst
9     Requires: 44
10     Created: July 2008
11 robbat2 1.3 Updated: October 2008, January 2010
12 cardoe 1.1 Updates: 44
13 robbat2 1.5 Post-History: December 2009, January 2010
14 cardoe 1.1
15     Abstract
16     ========
17     Deals with compression of large Manifest2 files.
18    
19     Motivation
20     ==========
21     With the introduction of MetaManifest, and full-tree Manifest coverage,
22     we are faced with the possibility of having very large Manifests.
23    
24     Preliminary experiments with MetaManifest, show that with just the
25 robbat2 1.5 existing per-package Manifests, the full MetaManifest (top-level only,
26     no first-level sub directories), for a tree including metadata/, exceeds
27     8MiB in size. Applying common compression can achieve a 50-60% reduction
28     in this size.
29    
30     Additionally, some of the larger already-existing Manifests in the tree
31     can also be reduced.
32 cardoe 1.1
33     Specification
34     =============
35 robbat2 1.5 Creation of compressed Manifests:
36     ---------------------------------
37     32KiB is suggested as a arbitrary cut-off point to start generating
38     compressed Manifest2 files.
39    
40     The compression must only applied during the creation of a tree intended
41     for end users. No Manifests stored in a VCS should be compressed in the
42     VCS. For the main gentoo-portage tree, this means that the compressed
43     Manifests should be generated using the CVS to Rsync process.
44    
45     The Manifest compression process is required to ensure that inconsistent
46     compressed versions do not exist.
47    
48     Validation of Manifests:
49     ------------------------
50 cardoe 1.1 When searching for a Manifest2 file, if the basename form does not
51     exist, the package manager should search in the same location using
52     common compressed suffixes, and use the compressed file in place of the
53     Manifest2.
54    
55 robbat2 1.3 gzip, bzip2, lzma, xz should all be supported if available on the given
56 cardoe 1.1 platform. In the case that multiple versions exist, the package manager
57     should simply pick one - they should be identical, differing only in
58     compression.
59    
60 robbat2 1.5 Example Results with a 32KiB cut-off, gzip algorithm
61     ====================================================
62     As of 2010/01/30, the suggested cut-off would impact the following 21 existing
63     Manifests, for a saving of nearly 900KiB::
64    
65     Size Path
66     65788 app-doc/linux-gazette/Manifest
67     75739 app-office/openoffice-bin/Manifest
68     40534 app-text/texlive-core/Manifest
69     41710 dev-texlive/texlive-bibtexextra/Manifest
70     38197 dev-texlive/texlive-documentation-english/Manifest
71     129610 dev-texlive/texlive-fontsextra/Manifest
72     36022 dev-texlive/texlive-humanities/Manifest
73     686118 dev-texlive/texlive-latexextra/Manifest
74     43392 dev-texlive/texlive-latexrecommended/Manifest
75     33375 dev-texlive/texlive-mathextra/Manifest
76     39781 dev-texlive/texlive-pictures/Manifest
77     69567 dev-texlive/texlive-pstricks/Manifest
78     75460 dev-texlive/texlive-publishers/Manifest
79     50879 dev-texlive/texlive-science/Manifest
80     36711 kde-base/kde-l10n/Manifest
81     36539 media-gfx/bootsplash-themes/Manifest
82     33058 net-fs/autofs/Manifest
83     39781 www-client/firefox-bin/Manifest
84     48983 www-client/icecat/Manifest
85     60213 www-client/mozilla-firefox/Manifest
86     39065 x11-themes/gkrellm-themes/Manifest
87    
88    
89     Additionally, with the MetaManifest proposal, the following new manifests would
90     also be compressed, for a saving of nearly 4MiB::
91    
92     Size Path
93     33442 app-admin/Manifest
94     71073 app-dicts/Manifest
95     35923 app-emacs/Manifest
96     45808 app-misc/Manifest
97     50169 app-text/Manifest
98     112786 dev-java/Manifest
99     65581 dev-libs/Manifest
100     42619 dev-lisp/Manifest
101     182163 dev-perl/Manifest
102     96198 dev-python/Manifest
103     58963 dev-ruby/Manifest
104     59736 dev-util/Manifest
105     58338 eclass/Manifest
106     55749 kde-base/Manifest
107     110064 licenses/Manifest
108     35262 media-gfx/Manifest
109     53995 media-libs/Manifest
110     55607 media-plugins/Manifest
111     71911 media-sound/Manifest
112     34835 media-video/Manifest
113     5747849 metadata/Manifest
114     47452 net-analyzer/Manifest
115     65989 net-misc/Manifest
116     316787 profiles/Manifest
117     67784 sys-apps/Manifest
118     48971 x11-misc/Manifest
119     41475 x11-plugins/Manifest
120    
121 cardoe 1.1
122     Backwards Compatibility
123     =======================
124     The package Manifests should also be maintained as ONLY uncompressed in
125     CVS.
126    
127     For processing of all existing per-package Manifests, if compression is
128     used, it should be done in parallel to the existing Manifests, to
129     provide for a changeover period. Newer versions of Portage may later
130     choose to exclude all non-compressed Manifests during emerge --sync if
131     compressed versions are guaranteed to exist on the servers.
132    
133     MetaManifests may come into existence as compressed from the start, as
134     do not have an backwards compatibility issues.
135    
136     As a side note, this breaks all manual interaction with Manifests
137     such as grep, and so should only be applied to large Manifest2 files,
138 robbat2 1.5 such as the MetaManifest.
139 cardoe 1.1
140     References
141     ==========
142     .. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
143     http://www.gentoo.org/proj/en/glep/glep-0044.html
144    
145     Copyright
146     =========
147 robbat2 1.4 Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
148 cardoe 1.1 distributed only subject to the terms and conditions set forth in the
149     Open Publication License, v1.0.
150    
151     vim: tw=72 ts=2 expandtab:

  ViewVC Help
Powered by ViewVC 1.1.20