/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.html
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0061.html

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.6 - (hide annotations) (download) (as text)
Sun Jan 31 09:56:08 2010 UTC (4 years, 5 months ago) by robbat2
Branch: MAIN
Changes since 1.5: +7 -3 lines
File MIME type: text/html
Update HTML.

1 cardoe 1.1 <?xml version="1.0" encoding="utf-8" ?>
2     <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3     <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4    
5     <head>
6     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7 robbat2 1.4 <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" />
8 cardoe 1.1 <title>GLEP 61 -- Manifest2 compression</title>
9     <link rel="stylesheet" href="tools/glep.css" type="text/css" /></head>
10     <body bgcolor="white">
11     <table class="navigation" cellpadding="0" cellspacing="0"
12     width="100%" border="0">
13     <tr><td class="navicon" width="150" height="35">
14     <a href="http://www.gentoo.org/" title="Gentoo Linux Home Page">
15     <img src="http://www.gentoo.org/images/gentoo-new.gif" alt="[Gentoo]"
16     border="0" width="150" height="35" /></a></td>
17     <td class="textlinks" align="left">
18     [<b><a href="http://www.gentoo.org/">Gentoo Linux Home</a></b>]
19     [<b><a href="http://www.gentoo.org/proj/en/glep">GLEP Index</a></b>]
20     [<b><a href="http://www.gentoo.org/proj/en/glep/glep-0061.txt">GLEP Source</a></b>]
21     </td></tr></table>
22     <table class="rfc2822 docutils field-list" frame="void" rules="none">
23     <col class="field-name" />
24     <col class="field-body" />
25     <tbody valign="top">
26     <tr class="field"><th class="field-name">GLEP:</th><td class="field-body">61</td>
27     </tr>
28     <tr class="field"><th class="field-name">Title:</th><td class="field-body">Manifest2 compression</td>
29     </tr>
30 robbat2 1.6 <tr class="field"><th class="field-name">Version:</th><td class="field-body">1.6</td>
31 cardoe 1.1 </tr>
32 robbat2 1.6 <tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0061.txt?cvsroot=gentoo">2010/01/31 09:55:43</a></td>
33 cardoe 1.1 </tr>
34     <tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;</td>
35     </tr>
36     <tr class="field"><th class="field-name">Status:</th><td class="field-body">Draft</td>
37     </tr>
38     <tr class="field"><th class="field-name">Type:</th><td class="field-body">Standards Track</td>
39     </tr>
40     <tr class="field"><th class="field-name">Content-Type:</th><td class="field-body"><a class="reference external" href="glep-0002.html">text/x-rst</a></td>
41     </tr>
42     <tr class="field"><th class="field-name">Requires:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0044.html">44</a></td>
43     </tr>
44     <tr class="field"><th class="field-name">Created:</th><td class="field-body">July 2008</td>
45     </tr>
46 robbat2 1.4 <tr class="field"><th class="field-name">Updated:</th><td class="field-body">October 2008, January 2010</td>
47 robbat2 1.3 </tr>
48 cardoe 1.1 <tr class="field"><th class="field-name">Updates:</th><td class="field-body">44</td>
49     </tr>
50 robbat2 1.5 <tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009, January 2010</td>
51 robbat2 1.3 </tr>
52 cardoe 1.1 </tbody>
53     </table>
54     <hr />
55     <div class="contents topic" id="contents">
56     <p class="topic-title first">Contents</p>
57     <ul class="simple">
58     <li><a class="reference internal" href="#abstract" id="id1">Abstract</a></li>
59     <li><a class="reference internal" href="#motivation" id="id2">Motivation</a></li>
60 robbat2 1.5 <li><a class="reference internal" href="#specification" id="id3">Specification</a><ul>
61     <li><a class="reference internal" href="#creation-of-compressed-manifests" id="id4">Creation of compressed Manifests:</a></li>
62     <li><a class="reference internal" href="#validation-of-manifests" id="id5">Validation of Manifests:</a></li>
63     </ul>
64     </li>
65     <li><a class="reference internal" href="#example-results-with-a-32kib-cut-off-gzip-algorithm" id="id6">Example Results with a 32KiB cut-off, gzip algorithm</a></li>
66     <li><a class="reference internal" href="#backwards-compatibility" id="id7">Backwards Compatibility</a></li>
67     <li><a class="reference internal" href="#references" id="id8">References</a></li>
68     <li><a class="reference internal" href="#copyright" id="id9">Copyright</a></li>
69 cardoe 1.1 </ul>
70     </div>
71     <div class="section" id="abstract">
72     <h1><a class="toc-backref" href="#id1">Abstract</a></h1>
73     <p>Deals with compression of large Manifest2 files.</p>
74     </div>
75     <div class="section" id="motivation">
76     <h1><a class="toc-backref" href="#id2">Motivation</a></h1>
77     <p>With the introduction of MetaManifest, and full-tree Manifest coverage,
78     we are faced with the possibility of having very large Manifests.</p>
79     <p>Preliminary experiments with MetaManifest, show that with just the
80 robbat2 1.5 existing per-package Manifests, the full MetaManifest (top-level only,
81     no first-level sub directories), for a tree including metadata/, exceeds
82     8MiB in size. Applying common compression can achieve a 50-60% reduction
83     in this size.</p>
84     <p>Additionally, some of the larger already-existing Manifests in the tree
85     can also be reduced.</p>
86 robbat2 1.6 <p>This GLEP is not mandatory for the tree-signing specification, but
87     instead helps to cut down the size impact of large Manifest2 files, some
88     of which are already present in the tree. As such, it is also able to
89     stand on it's own.</p>
90 cardoe 1.1 </div>
91     <div class="section" id="specification">
92     <h1><a class="toc-backref" href="#id3">Specification</a></h1>
93 robbat2 1.5 <div class="section" id="creation-of-compressed-manifests">
94     <h2><a class="toc-backref" href="#id4">Creation of compressed Manifests:</a></h2>
95     <p>32KiB is suggested as a arbitrary cut-off point to start generating
96     compressed Manifest2 files.</p>
97     <p>The compression must only applied during the creation of a tree intended
98     for end users. No Manifests stored in a VCS should be compressed in the
99     VCS. For the main gentoo-portage tree, this means that the compressed
100     Manifests should be generated using the CVS to Rsync process.</p>
101     <p>The Manifest compression process is required to ensure that inconsistent
102     compressed versions do not exist.</p>
103     </div>
104     <div class="section" id="validation-of-manifests">
105     <h2><a class="toc-backref" href="#id5">Validation of Manifests:</a></h2>
106 cardoe 1.1 <p>When searching for a Manifest2 file, if the basename form does not
107     exist, the package manager should search in the same location using
108     common compressed suffixes, and use the compressed file in place of the
109     Manifest2.</p>
110 robbat2 1.4 <p>gzip, bzip2, lzma, xz should all be supported if available on the given
111 cardoe 1.1 platform. In the case that multiple versions exist, the package manager
112     should simply pick one - they should be identical, differing only in
113     compression.</p>
114 robbat2 1.5 </div>
115     </div>
116     <div class="section" id="example-results-with-a-32kib-cut-off-gzip-algorithm">
117     <h1><a class="toc-backref" href="#id6">Example Results with a 32KiB cut-off, gzip algorithm</a></h1>
118     <p>As of 2010/01/30, the suggested cut-off would impact the following 21 existing
119     Manifests, for a saving of nearly 900KiB:</p>
120     <pre class="literal-block">
121     Size Path
122     65788 app-doc/linux-gazette/Manifest
123     75739 app-office/openoffice-bin/Manifest
124     40534 app-text/texlive-core/Manifest
125     41710 dev-texlive/texlive-bibtexextra/Manifest
126     38197 dev-texlive/texlive-documentation-english/Manifest
127     129610 dev-texlive/texlive-fontsextra/Manifest
128     36022 dev-texlive/texlive-humanities/Manifest
129     686118 dev-texlive/texlive-latexextra/Manifest
130     43392 dev-texlive/texlive-latexrecommended/Manifest
131     33375 dev-texlive/texlive-mathextra/Manifest
132     39781 dev-texlive/texlive-pictures/Manifest
133     69567 dev-texlive/texlive-pstricks/Manifest
134     75460 dev-texlive/texlive-publishers/Manifest
135     50879 dev-texlive/texlive-science/Manifest
136     36711 kde-base/kde-l10n/Manifest
137     36539 media-gfx/bootsplash-themes/Manifest
138     33058 net-fs/autofs/Manifest
139     39781 www-client/firefox-bin/Manifest
140     48983 www-client/icecat/Manifest
141     60213 www-client/mozilla-firefox/Manifest
142     39065 x11-themes/gkrellm-themes/Manifest
143     </pre>
144     <p>Additionally, with the MetaManifest proposal, the following new manifests would
145     also be compressed, for a saving of nearly 4MiB:</p>
146     <pre class="literal-block">
147     Size Path
148     33442 app-admin/Manifest
149     71073 app-dicts/Manifest
150     35923 app-emacs/Manifest
151     45808 app-misc/Manifest
152     50169 app-text/Manifest
153     112786 dev-java/Manifest
154     65581 dev-libs/Manifest
155     42619 dev-lisp/Manifest
156     182163 dev-perl/Manifest
157     96198 dev-python/Manifest
158     58963 dev-ruby/Manifest
159     59736 dev-util/Manifest
160     58338 eclass/Manifest
161     55749 kde-base/Manifest
162     110064 licenses/Manifest
163     35262 media-gfx/Manifest
164     53995 media-libs/Manifest
165     55607 media-plugins/Manifest
166     71911 media-sound/Manifest
167     34835 media-video/Manifest
168     5747849 metadata/Manifest
169     47452 net-analyzer/Manifest
170     65989 net-misc/Manifest
171     316787 profiles/Manifest
172     67784 sys-apps/Manifest
173     48971 x11-misc/Manifest
174     41475 x11-plugins/Manifest
175     </pre>
176 cardoe 1.1 </div>
177     <div class="section" id="backwards-compatibility">
178 robbat2 1.5 <h1><a class="toc-backref" href="#id7">Backwards Compatibility</a></h1>
179 cardoe 1.1 <p>The package Manifests should also be maintained as ONLY uncompressed in
180     CVS.</p>
181     <p>For processing of all existing per-package Manifests, if compression is
182     used, it should be done in parallel to the existing Manifests, to
183     provide for a changeover period. Newer versions of Portage may later
184     choose to exclude all non-compressed Manifests during emerge --sync if
185     compressed versions are guaranteed to exist on the servers.</p>
186     <p>MetaManifests may come into existence as compressed from the start, as
187     do not have an backwards compatibility issues.</p>
188     <p>As a side note, this breaks all manual interaction with Manifests
189     such as grep, and so should only be applied to large Manifest2 files,
190 robbat2 1.5 such as the MetaManifest.</p>
191 cardoe 1.1 </div>
192     <div class="section" id="references">
193 robbat2 1.5 <h1><a class="toc-backref" href="#id8">References</a></h1>
194 cardoe 1.1 <table class="docutils footnote" frame="void" id="glep44" rules="none">
195     <colgroup><col class="label" /><col /></colgroup>
196     <tbody valign="top">
197     <tr><td class="label">[1]</td><td>Mauch, M. (2005) GLEP44 - Manifest2 format.
198     <a class="reference external" href="http://www.gentoo.org/proj/en/glep/glep-0044.html">http://www.gentoo.org/proj/en/glep/glep-0044.html</a></td></tr>
199     </tbody>
200     </table>
201     </div>
202     <div class="section" id="copyright">
203 robbat2 1.5 <h1><a class="toc-backref" href="#id9">Copyright</a></h1>
204 robbat2 1.4 <p>Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
205 cardoe 1.1 distributed only subject to the terms and conditions set forth in the
206     Open Publication License, v1.0.</p>
207     <p>vim: tw=72 ts=2 expandtab:</p>
208     </div>
209    
210     </div>
211     <div class="footer">
212     <hr class="footer" />
213     <a class="reference external" href="glep-0061.txt">View document source</a>.
214 robbat2 1.6 Generated on: 2010-01-31 09:56 UTC.
215 cardoe 1.1 Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.
216    
217     </div>
218     </body>
219     </html>

  ViewVC Help
Powered by ViewVC 1.1.20