/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.html
Gentoo

Contents of /xml/htdocs/proj/en/glep/glep-0061.html

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.5 - (show annotations) (download) (as text)
Sun Jan 31 08:50:52 2010 UTC (4 years, 10 months ago) by robbat2
Branch: MAIN
Changes since 1.4: +98 -18 lines
File MIME type: text/html
Resync HTML for treesigning GLEPs.

1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4
5 <head>
6 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7 <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/" />
8 <title>GLEP 61 -- Manifest2 compression</title>
9 <link rel="stylesheet" href="tools/glep.css" type="text/css" /></head>
10 <body bgcolor="white">
11 <table class="navigation" cellpadding="0" cellspacing="0"
12 width="100%" border="0">
13 <tr><td class="navicon" width="150" height="35">
14 <a href="http://www.gentoo.org/" title="Gentoo Linux Home Page">
15 <img src="http://www.gentoo.org/images/gentoo-new.gif" alt="[Gentoo]"
16 border="0" width="150" height="35" /></a></td>
17 <td class="textlinks" align="left">
18 [<b><a href="http://www.gentoo.org/">Gentoo Linux Home</a></b>]
19 [<b><a href="http://www.gentoo.org/proj/en/glep">GLEP Index</a></b>]
20 [<b><a href="http://www.gentoo.org/proj/en/glep/glep-0061.txt">GLEP Source</a></b>]
21 </td></tr></table>
22 <table class="rfc2822 docutils field-list" frame="void" rules="none">
23 <col class="field-name" />
24 <col class="field-body" />
25 <tbody valign="top">
26 <tr class="field"><th class="field-name">GLEP:</th><td class="field-body">61</td>
27 </tr>
28 <tr class="field"><th class="field-name">Title:</th><td class="field-body">Manifest2 compression</td>
29 </tr>
30 <tr class="field"><th class="field-name">Version:</th><td class="field-body">1.5</td>
31 </tr>
32 <tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0061.txt?cvsroot=gentoo">2010/01/31 08:27:05</a></td>
33 </tr>
34 <tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;</td>
35 </tr>
36 <tr class="field"><th class="field-name">Status:</th><td class="field-body">Draft</td>
37 </tr>
38 <tr class="field"><th class="field-name">Type:</th><td class="field-body">Standards Track</td>
39 </tr>
40 <tr class="field"><th class="field-name">Content-Type:</th><td class="field-body"><a class="reference external" href="glep-0002.html">text/x-rst</a></td>
41 </tr>
42 <tr class="field"><th class="field-name">Requires:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/proj/en/glepglep-0044.html">44</a></td>
43 </tr>
44 <tr class="field"><th class="field-name">Created:</th><td class="field-body">July 2008</td>
45 </tr>
46 <tr class="field"><th class="field-name">Updated:</th><td class="field-body">October 2008, January 2010</td>
47 </tr>
48 <tr class="field"><th class="field-name">Updates:</th><td class="field-body">44</td>
49 </tr>
50 <tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009, January 2010</td>
51 </tr>
52 </tbody>
53 </table>
54 <hr />
55 <div class="contents topic" id="contents">
56 <p class="topic-title first">Contents</p>
57 <ul class="simple">
58 <li><a class="reference internal" href="#abstract" id="id1">Abstract</a></li>
59 <li><a class="reference internal" href="#motivation" id="id2">Motivation</a></li>
60 <li><a class="reference internal" href="#specification" id="id3">Specification</a><ul>
61 <li><a class="reference internal" href="#creation-of-compressed-manifests" id="id4">Creation of compressed Manifests:</a></li>
62 <li><a class="reference internal" href="#validation-of-manifests" id="id5">Validation of Manifests:</a></li>
63 </ul>
64 </li>
65 <li><a class="reference internal" href="#example-results-with-a-32kib-cut-off-gzip-algorithm" id="id6">Example Results with a 32KiB cut-off, gzip algorithm</a></li>
66 <li><a class="reference internal" href="#backwards-compatibility" id="id7">Backwards Compatibility</a></li>
67 <li><a class="reference internal" href="#references" id="id8">References</a></li>
68 <li><a class="reference internal" href="#copyright" id="id9">Copyright</a></li>
69 </ul>
70 </div>
71 <div class="section" id="abstract">
72 <h1><a class="toc-backref" href="#id1">Abstract</a></h1>
73 <p>Deals with compression of large Manifest2 files.</p>
74 </div>
75 <div class="section" id="motivation">
76 <h1><a class="toc-backref" href="#id2">Motivation</a></h1>
77 <p>With the introduction of MetaManifest, and full-tree Manifest coverage,
78 we are faced with the possibility of having very large Manifests.</p>
79 <p>Preliminary experiments with MetaManifest, show that with just the
80 existing per-package Manifests, the full MetaManifest (top-level only,
81 no first-level sub directories), for a tree including metadata/, exceeds
82 8MiB in size. Applying common compression can achieve a 50-60% reduction
83 in this size.</p>
84 <p>Additionally, some of the larger already-existing Manifests in the tree
85 can also be reduced.</p>
86 </div>
87 <div class="section" id="specification">
88 <h1><a class="toc-backref" href="#id3">Specification</a></h1>
89 <div class="section" id="creation-of-compressed-manifests">
90 <h2><a class="toc-backref" href="#id4">Creation of compressed Manifests:</a></h2>
91 <p>32KiB is suggested as a arbitrary cut-off point to start generating
92 compressed Manifest2 files.</p>
93 <p>The compression must only applied during the creation of a tree intended
94 for end users. No Manifests stored in a VCS should be compressed in the
95 VCS. For the main gentoo-portage tree, this means that the compressed
96 Manifests should be generated using the CVS to Rsync process.</p>
97 <p>The Manifest compression process is required to ensure that inconsistent
98 compressed versions do not exist.</p>
99 </div>
100 <div class="section" id="validation-of-manifests">
101 <h2><a class="toc-backref" href="#id5">Validation of Manifests:</a></h2>
102 <p>When searching for a Manifest2 file, if the basename form does not
103 exist, the package manager should search in the same location using
104 common compressed suffixes, and use the compressed file in place of the
105 Manifest2.</p>
106 <p>gzip, bzip2, lzma, xz should all be supported if available on the given
107 platform. In the case that multiple versions exist, the package manager
108 should simply pick one - they should be identical, differing only in
109 compression.</p>
110 </div>
111 </div>
112 <div class="section" id="example-results-with-a-32kib-cut-off-gzip-algorithm">
113 <h1><a class="toc-backref" href="#id6">Example Results with a 32KiB cut-off, gzip algorithm</a></h1>
114 <p>As of 2010/01/30, the suggested cut-off would impact the following 21 existing
115 Manifests, for a saving of nearly 900KiB:</p>
116 <pre class="literal-block">
117 Size Path
118 65788 app-doc/linux-gazette/Manifest
119 75739 app-office/openoffice-bin/Manifest
120 40534 app-text/texlive-core/Manifest
121 41710 dev-texlive/texlive-bibtexextra/Manifest
122 38197 dev-texlive/texlive-documentation-english/Manifest
123 129610 dev-texlive/texlive-fontsextra/Manifest
124 36022 dev-texlive/texlive-humanities/Manifest
125 686118 dev-texlive/texlive-latexextra/Manifest
126 43392 dev-texlive/texlive-latexrecommended/Manifest
127 33375 dev-texlive/texlive-mathextra/Manifest
128 39781 dev-texlive/texlive-pictures/Manifest
129 69567 dev-texlive/texlive-pstricks/Manifest
130 75460 dev-texlive/texlive-publishers/Manifest
131 50879 dev-texlive/texlive-science/Manifest
132 36711 kde-base/kde-l10n/Manifest
133 36539 media-gfx/bootsplash-themes/Manifest
134 33058 net-fs/autofs/Manifest
135 39781 www-client/firefox-bin/Manifest
136 48983 www-client/icecat/Manifest
137 60213 www-client/mozilla-firefox/Manifest
138 39065 x11-themes/gkrellm-themes/Manifest
139 </pre>
140 <p>Additionally, with the MetaManifest proposal, the following new manifests would
141 also be compressed, for a saving of nearly 4MiB:</p>
142 <pre class="literal-block">
143 Size Path
144 33442 app-admin/Manifest
145 71073 app-dicts/Manifest
146 35923 app-emacs/Manifest
147 45808 app-misc/Manifest
148 50169 app-text/Manifest
149 112786 dev-java/Manifest
150 65581 dev-libs/Manifest
151 42619 dev-lisp/Manifest
152 182163 dev-perl/Manifest
153 96198 dev-python/Manifest
154 58963 dev-ruby/Manifest
155 59736 dev-util/Manifest
156 58338 eclass/Manifest
157 55749 kde-base/Manifest
158 110064 licenses/Manifest
159 35262 media-gfx/Manifest
160 53995 media-libs/Manifest
161 55607 media-plugins/Manifest
162 71911 media-sound/Manifest
163 34835 media-video/Manifest
164 5747849 metadata/Manifest
165 47452 net-analyzer/Manifest
166 65989 net-misc/Manifest
167 316787 profiles/Manifest
168 67784 sys-apps/Manifest
169 48971 x11-misc/Manifest
170 41475 x11-plugins/Manifest
171 </pre>
172 </div>
173 <div class="section" id="backwards-compatibility">
174 <h1><a class="toc-backref" href="#id7">Backwards Compatibility</a></h1>
175 <p>The package Manifests should also be maintained as ONLY uncompressed in
176 CVS.</p>
177 <p>For processing of all existing per-package Manifests, if compression is
178 used, it should be done in parallel to the existing Manifests, to
179 provide for a changeover period. Newer versions of Portage may later
180 choose to exclude all non-compressed Manifests during emerge --sync if
181 compressed versions are guaranteed to exist on the servers.</p>
182 <p>MetaManifests may come into existence as compressed from the start, as
183 do not have an backwards compatibility issues.</p>
184 <p>As a side note, this breaks all manual interaction with Manifests
185 such as grep, and so should only be applied to large Manifest2 files,
186 such as the MetaManifest.</p>
187 </div>
188 <div class="section" id="references">
189 <h1><a class="toc-backref" href="#id8">References</a></h1>
190 <table class="docutils footnote" frame="void" id="glep44" rules="none">
191 <colgroup><col class="label" /><col /></colgroup>
192 <tbody valign="top">
193 <tr><td class="label">[1]</td><td>Mauch, M. (2005) GLEP44 - Manifest2 format.
194 <a class="reference external" href="http://www.gentoo.org/proj/en/glep/glep-0044.html">http://www.gentoo.org/proj/en/glep/glep-0044.html</a></td></tr>
195 </tbody>
196 </table>
197 </div>
198 <div class="section" id="copyright">
199 <h1><a class="toc-backref" href="#id9">Copyright</a></h1>
200 <p>Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
201 distributed only subject to the terms and conditions set forth in the
202 Open Publication License, v1.0.</p>
203 <p>vim: tw=72 ts=2 expandtab:</p>
204 </div>
205
206 </div>
207 <div class="footer">
208 <hr class="footer" />
209 <a class="reference external" href="glep-0061.txt">View document source</a>.
210 Generated on: 2010-01-31 08:27 UTC.
211 Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.
212
213 </div>
214 </body>
215 </html>

  ViewVC Help
Powered by ViewVC 1.1.20