/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.html
Gentoo

Diff of /xml/htdocs/proj/en/glep/glep-0061.html

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.3 Revision 1.6
… …
2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 3<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 4
5<head> 5<head>
6 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 6 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7 <meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" /> 7 <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" />
8 <title>GLEP 61 -- Manifest2 compression</title> 8 <title>GLEP 61 -- Manifest2 compression</title>
9 <link rel="stylesheet" href="tools/glep.css" type="text/css" /></head> 9 <link rel="stylesheet" href="tools/glep.css" type="text/css" /></head>
10<body bgcolor="white"> 10<body bgcolor="white">
11<table class="navigation" cellpadding="0" cellspacing="0" 11<table class="navigation" cellpadding="0" cellspacing="0"
12 width="100%" border="0"> 12 width="100%" border="0">
… …
25<tbody valign="top"> 25<tbody valign="top">
26<tr class="field"><th class="field-name">GLEP:</th><td class="field-body">61</td> 26<tr class="field"><th class="field-name">GLEP:</th><td class="field-body">61</td>
27</tr> 27</tr>
28<tr class="field"><th class="field-name">Title:</th><td class="field-body">Manifest2 compression</td> 28<tr class="field"><th class="field-name">Title:</th><td class="field-body">Manifest2 compression</td>
29</tr> 29</tr>
30<tr class="field"><th class="field-name">Version:</th><td class="field-body">1.2</td> 30<tr class="field"><th class="field-name">Version:</th><td class="field-body">1.6</td>
31</tr> 31</tr>
32<tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0061.txt?cvsroot=gentoo">2008/10/28 07:45:56</a></td> 32<tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0061.txt?cvsroot=gentoo">2010/01/31 09:55:43</a></td>
33</tr> 33</tr>
34<tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;</td> 34<tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;</td>
35</tr> 35</tr>
36<tr class="field"><th class="field-name">Status:</th><td class="field-body">Draft</td> 36<tr class="field"><th class="field-name">Status:</th><td class="field-body">Draft</td>
37</tr> 37</tr>
… …
41</tr> 41</tr>
42<tr class="field"><th class="field-name">Requires:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0044.html">44</a></td> 42<tr class="field"><th class="field-name">Requires:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0044.html">44</a></td>
43</tr> 43</tr>
44<tr class="field"><th class="field-name">Created:</th><td class="field-body">July 2008</td> 44<tr class="field"><th class="field-name">Created:</th><td class="field-body">July 2008</td>
45</tr> 45</tr>
46<tr class="field"><th class="field-name">Updated:</th><td class="field-body">October 2008</td> 46<tr class="field"><th class="field-name">Updated:</th><td class="field-body">October 2008, January 2010</td>
47</tr> 47</tr>
48<tr class="field"><th class="field-name">Updates:</th><td class="field-body">44</td> 48<tr class="field"><th class="field-name">Updates:</th><td class="field-body">44</td>
49</tr> 49</tr>
50<tr class="field"><th class="field-name">Post-History:</th><td class="field-body"></td> 50<tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009, January 2010</td>
51</tr> 51</tr>
52</tbody> 52</tbody>
53</table> 53</table>
54<hr /> 54<hr />
55<div class="contents topic" id="contents"> 55<div class="contents topic" id="contents">
56<p class="topic-title first">Contents</p> 56<p class="topic-title first">Contents</p>
57<ul class="simple"> 57<ul class="simple">
58<li><a class="reference internal" href="#abstract" id="id1">Abstract</a></li> 58<li><a class="reference internal" href="#abstract" id="id1">Abstract</a></li>
59<li><a class="reference internal" href="#motivation" id="id2">Motivation</a></li> 59<li><a class="reference internal" href="#motivation" id="id2">Motivation</a></li>
60<li><a class="reference internal" href="#specification" id="id3">Specification</a></li> 60<li><a class="reference internal" href="#specification" id="id3">Specification</a><ul>
61<li><a class="reference internal" href="#creation-of-compressed-manifests" id="id4">Creation of compressed Manifests:</a></li>
62<li><a class="reference internal" href="#validation-of-manifests" id="id5">Validation of Manifests:</a></li>
63</ul>
64</li>
65<li><a class="reference internal" href="#example-results-with-a-32kib-cut-off-gzip-algorithm" id="id6">Example Results with a 32KiB cut-off, gzip algorithm</a></li>
61<li><a class="reference internal" href="#backwards-compatibility" id="id4">Backwards Compatibility</a></li> 66<li><a class="reference internal" href="#backwards-compatibility" id="id7">Backwards Compatibility</a></li>
62<li><a class="reference internal" href="#references" id="id5">References</a></li> 67<li><a class="reference internal" href="#references" id="id8">References</a></li>
63<li><a class="reference internal" href="#copyright" id="id6">Copyright</a></li> 68<li><a class="reference internal" href="#copyright" id="id9">Copyright</a></li>
64</ul> 69</ul>
65</div> 70</div>
66<div class="section" id="abstract"> 71<div class="section" id="abstract">
67<h1><a class="toc-backref" href="#id1">Abstract</a></h1> 72<h1><a class="toc-backref" href="#id1">Abstract</a></h1>
68<p>Deals with compression of large Manifest2 files.</p> 73<p>Deals with compression of large Manifest2 files.</p>
… …
70<div class="section" id="motivation"> 75<div class="section" id="motivation">
71<h1><a class="toc-backref" href="#id2">Motivation</a></h1> 76<h1><a class="toc-backref" href="#id2">Motivation</a></h1>
72<p>With the introduction of MetaManifest, and full-tree Manifest coverage, 77<p>With the introduction of MetaManifest, and full-tree Manifest coverage,
73we are faced with the possibility of having very large Manifests.</p> 78we are faced with the possibility of having very large Manifests.</p>
74<p>Preliminary experiments with MetaManifest, show that with just the 79<p>Preliminary experiments with MetaManifest, show that with just the
75existing per-package Manifests, the full MetaManifest, for a tree 80existing per-package Manifests, the full MetaManifest (top-level only,
76including metadata/, exceeds 8MiB in size. Applying common compression 81no first-level sub directories), for a tree including metadata/, exceeds
77can achieve a 50-60% reduction in this size.</p> 828MiB in size. Applying common compression can achieve a 50-60% reduction
83in this size.</p>
84<p>Additionally, some of the larger already-existing Manifests in the tree
85can also be reduced.</p>
86<p>This GLEP is not mandatory for the tree-signing specification, but
87instead helps to cut down the size impact of large Manifest2 files, some
88of which are already present in the tree. As such, it is also able to
89stand on it's own.</p>
78</div> 90</div>
79<div class="section" id="specification"> 91<div class="section" id="specification">
80<h1><a class="toc-backref" href="#id3">Specification</a></h1> 92<h1><a class="toc-backref" href="#id3">Specification</a></h1>
93<div class="section" id="creation-of-compressed-manifests">
94<h2><a class="toc-backref" href="#id4">Creation of compressed Manifests:</a></h2>
95<p>32KiB is suggested as a arbitrary cut-off point to start generating
96compressed Manifest2 files.</p>
97<p>The compression must only applied during the creation of a tree intended
98for end users. No Manifests stored in a VCS should be compressed in the
99VCS. For the main gentoo-portage tree, this means that the compressed
100Manifests should be generated using the CVS to Rsync process.</p>
101<p>The Manifest compression process is required to ensure that inconsistent
102compressed versions do not exist.</p>
103</div>
104<div class="section" id="validation-of-manifests">
105<h2><a class="toc-backref" href="#id5">Validation of Manifests:</a></h2>
81<p>When searching for a Manifest2 file, if the basename form does not 106<p>When searching for a Manifest2 file, if the basename form does not
82exist, the package manager should search in the same location using 107exist, the package manager should search in the same location using
83common compressed suffixes, and use the compressed file in place of the 108common compressed suffixes, and use the compressed file in place of the
84Manifest2.</p> 109Manifest2.</p>
85<p>gzip, bzip2, lzma should all be supported if available on the given 110<p>gzip, bzip2, lzma, xz should all be supported if available on the given
86platform. In the case that multiple versions exist, the package manager 111platform. In the case that multiple versions exist, the package manager
87should simply pick one - they should be identical, differing only in 112should simply pick one - they should be identical, differing only in
88compression.</p> 113compression.</p>
89<p>The Manifest generation process is required to ensure that inconsistent 114</div>
90compressed versions do not exist.</p> 115</div>
116<div class="section" id="example-results-with-a-32kib-cut-off-gzip-algorithm">
117<h1><a class="toc-backref" href="#id6">Example Results with a 32KiB cut-off, gzip algorithm</a></h1>
118<p>As of 2010/01/30, the suggested cut-off would impact the following 21 existing
119Manifests, for a saving of nearly 900KiB:</p>
120<pre class="literal-block">
121Size Path
122 65788 app-doc/linux-gazette/Manifest
123 75739 app-office/openoffice-bin/Manifest
124 40534 app-text/texlive-core/Manifest
125 41710 dev-texlive/texlive-bibtexextra/Manifest
126 38197 dev-texlive/texlive-documentation-english/Manifest
127129610 dev-texlive/texlive-fontsextra/Manifest
128 36022 dev-texlive/texlive-humanities/Manifest
129686118 dev-texlive/texlive-latexextra/Manifest
130 43392 dev-texlive/texlive-latexrecommended/Manifest
131 33375 dev-texlive/texlive-mathextra/Manifest
132 39781 dev-texlive/texlive-pictures/Manifest
133 69567 dev-texlive/texlive-pstricks/Manifest
134 75460 dev-texlive/texlive-publishers/Manifest
135 50879 dev-texlive/texlive-science/Manifest
136 36711 kde-base/kde-l10n/Manifest
137 36539 media-gfx/bootsplash-themes/Manifest
138 33058 net-fs/autofs/Manifest
139 39781 www-client/firefox-bin/Manifest
140 48983 www-client/icecat/Manifest
141 60213 www-client/mozilla-firefox/Manifest
142 39065 x11-themes/gkrellm-themes/Manifest
143</pre>
144<p>Additionally, with the MetaManifest proposal, the following new manifests would
145also be compressed, for a saving of nearly 4MiB:</p>
146<pre class="literal-block">
147Size Path
148 33442 app-admin/Manifest
149 71073 app-dicts/Manifest
150 35923 app-emacs/Manifest
151 45808 app-misc/Manifest
152 50169 app-text/Manifest
153 112786 dev-java/Manifest
154 65581 dev-libs/Manifest
155 42619 dev-lisp/Manifest
156 182163 dev-perl/Manifest
157 96198 dev-python/Manifest
158 58963 dev-ruby/Manifest
159 59736 dev-util/Manifest
160 58338 eclass/Manifest
161 55749 kde-base/Manifest
162 110064 licenses/Manifest
163 35262 media-gfx/Manifest
164 53995 media-libs/Manifest
165 55607 media-plugins/Manifest
166 71911 media-sound/Manifest
167 34835 media-video/Manifest
1685747849 metadata/Manifest
169 47452 net-analyzer/Manifest
170 65989 net-misc/Manifest
171 316787 profiles/Manifest
172 67784 sys-apps/Manifest
173 48971 x11-misc/Manifest
174 41475 x11-plugins/Manifest
175</pre>
91</div> 176</div>
92<div class="section" id="backwards-compatibility"> 177<div class="section" id="backwards-compatibility">
93<h1><a class="toc-backref" href="#id4">Backwards Compatibility</a></h1> 178<h1><a class="toc-backref" href="#id7">Backwards Compatibility</a></h1>
94<p>The package Manifests should also be maintained as ONLY uncompressed in 179<p>The package Manifests should also be maintained as ONLY uncompressed in
95CVS.</p> 180CVS.</p>
96<p>For processing of all existing per-package Manifests, if compression is 181<p>For processing of all existing per-package Manifests, if compression is
97used, it should be done in parallel to the existing Manifests, to 182used, it should be done in parallel to the existing Manifests, to
98provide for a changeover period. Newer versions of Portage may later 183provide for a changeover period. Newer versions of Portage may later
… …
100compressed versions are guaranteed to exist on the servers.</p> 185compressed versions are guaranteed to exist on the servers.</p>
101<p>MetaManifests may come into existence as compressed from the start, as 186<p>MetaManifests may come into existence as compressed from the start, as
102do not have an backwards compatibility issues.</p> 187do not have an backwards compatibility issues.</p>
103<p>As a side note, this breaks all manual interaction with Manifests 188<p>As a side note, this breaks all manual interaction with Manifests
104such as grep, and so should only be applied to large Manifest2 files, 189such as grep, and so should only be applied to large Manifest2 files,
105such as the MetaManifest. 384KiB is suggested as a arbitary cut-off 190such as the MetaManifest.</p>
106point to start generating compressed Manifest2 files.</p>
107</div> 191</div>
108<div class="section" id="references"> 192<div class="section" id="references">
109<h1><a class="toc-backref" href="#id5">References</a></h1> 193<h1><a class="toc-backref" href="#id8">References</a></h1>
110<table class="docutils footnote" frame="void" id="glep44" rules="none"> 194<table class="docutils footnote" frame="void" id="glep44" rules="none">
111<colgroup><col class="label" /><col /></colgroup> 195<colgroup><col class="label" /><col /></colgroup>
112<tbody valign="top"> 196<tbody valign="top">
113<tr><td class="label">[1]</td><td>Mauch, M. (2005) GLEP44 - Manifest2 format. 197<tr><td class="label">[1]</td><td>Mauch, M. (2005) GLEP44 - Manifest2 format.
114<a class="reference external" href="http://www.gentoo.org/proj/en/glep/glep-0044.html">http://www.gentoo.org/proj/en/glep/glep-0044.html</a></td></tr> 198<a class="reference external" href="http://www.gentoo.org/proj/en/glep/glep-0044.html">http://www.gentoo.org/proj/en/glep/glep-0044.html</a></td></tr>
115</tbody> 199</tbody>
116</table> 200</table>
117</div> 201</div>
118<div class="section" id="copyright"> 202<div class="section" id="copyright">
119<h1><a class="toc-backref" href="#id6">Copyright</a></h1> 203<h1><a class="toc-backref" href="#id9">Copyright</a></h1>
120<p>Copyright (c) 2007 by Robin Hugh Johnson. This material may be 204<p>Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
121distributed only subject to the terms and conditions set forth in the 205distributed only subject to the terms and conditions set forth in the
122Open Publication License, v1.0.</p> 206Open Publication License, v1.0.</p>
123<p>vim: tw=72 ts=2 expandtab:</p> 207<p>vim: tw=72 ts=2 expandtab:</p>
124</div> 208</div>
125 209
126</div> 210</div>
127<div class="footer"> 211<div class="footer">
128<hr class="footer" /> 212<hr class="footer" />
129<a class="reference external" href="glep-0061.txt">View document source</a>. 213<a class="reference external" href="glep-0061.txt">View document source</a>.
130Generated on: 2008-10-28 07:47 UTC. 214Generated on: 2010-01-31 09:56 UTC.
131Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source. 215Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.
132 216
133</div> 217</div>
134</body> 218</body>
135</html> 219</html>
Legend:
Removed from v.1.3  
changed lines
  Added in v.1.6
  ViewVC Help
Powered by ViewVC 1.1.13