--- xml/htdocs/proj/en/glep/glep-0061.html 2010/01/13 03:28:33 1.4 +++ xml/htdocs/proj/en/glep/glep-0061.html 2010/01/31 08:50:52 1.5 @@ -27,9 +27,9 @@ Title:Manifest2 compression -Version:1.4 +Version:1.5 -Last-Modified:2010/01/13 03:26:53 +Last-Modified:2010/01/31 08:27:05 Author:Robin Hugh Johnson <robbat2 at gentoo.org> @@ -47,7 +47,7 @@ Updates:44 -Post-History:December 2009 +Post-History:December 2009, January 2010 @@ -57,10 +57,15 @@
@@ -72,12 +77,28 @@

With the introduction of MetaManifest, and full-tree Manifest coverage, we are faced with the possibility of having very large Manifests.

Preliminary experiments with MetaManifest, show that with just the -existing per-package Manifests, the full MetaManifest, for a tree -including metadata/, exceeds 8MiB in size. Applying common compression -can achieve a 50-60% reduction in this size.

+existing per-package Manifests, the full MetaManifest (top-level only, +no first-level sub directories), for a tree including metadata/, exceeds +8MiB in size. Applying common compression can achieve a 50-60% reduction +in this size.

+

Additionally, some of the larger already-existing Manifests in the tree +can also be reduced.

Specification

+
+

Creation of compressed Manifests:

+

32KiB is suggested as a arbitrary cut-off point to start generating +compressed Manifest2 files.

+

The compression must only applied during the creation of a tree intended +for end users. No Manifests stored in a VCS should be compressed in the +VCS. For the main gentoo-portage tree, this means that the compressed +Manifests should be generated using the CVS to Rsync process.

+

The Manifest compression process is required to ensure that inconsistent +compressed versions do not exist.

+
+
+

Validation of Manifests:

When searching for a Manifest2 file, if the basename form does not exist, the package manager should search in the same location using common compressed suffixes, and use the compressed file in place of the @@ -86,11 +107,71 @@ platform. In the case that multiple versions exist, the package manager should simply pick one - they should be identical, differing only in compression.

-

The Manifest generation process is required to ensure that inconsistent -compressed versions do not exist.

+
+
+
+

Example Results with a 32KiB cut-off, gzip algorithm

+

As of 2010/01/30, the suggested cut-off would impact the following 21 existing +Manifests, for a saving of nearly 900KiB:

+
+Size   Path
+ 65788 app-doc/linux-gazette/Manifest
+ 75739 app-office/openoffice-bin/Manifest
+ 40534 app-text/texlive-core/Manifest
+ 41710 dev-texlive/texlive-bibtexextra/Manifest
+ 38197 dev-texlive/texlive-documentation-english/Manifest
+129610 dev-texlive/texlive-fontsextra/Manifest
+ 36022 dev-texlive/texlive-humanities/Manifest
+686118 dev-texlive/texlive-latexextra/Manifest
+ 43392 dev-texlive/texlive-latexrecommended/Manifest
+ 33375 dev-texlive/texlive-mathextra/Manifest
+ 39781 dev-texlive/texlive-pictures/Manifest
+ 69567 dev-texlive/texlive-pstricks/Manifest
+ 75460 dev-texlive/texlive-publishers/Manifest
+ 50879 dev-texlive/texlive-science/Manifest
+ 36711 kde-base/kde-l10n/Manifest
+ 36539 media-gfx/bootsplash-themes/Manifest
+ 33058 net-fs/autofs/Manifest
+ 39781 www-client/firefox-bin/Manifest
+ 48983 www-client/icecat/Manifest
+ 60213 www-client/mozilla-firefox/Manifest
+ 39065 x11-themes/gkrellm-themes/Manifest
+
+

Additionally, with the MetaManifest proposal, the following new manifests would +also be compressed, for a saving of nearly 4MiB:

+
+Size    Path
+  33442 app-admin/Manifest
+  71073 app-dicts/Manifest
+  35923 app-emacs/Manifest
+  45808 app-misc/Manifest
+  50169 app-text/Manifest
+ 112786 dev-java/Manifest
+  65581 dev-libs/Manifest
+  42619 dev-lisp/Manifest
+ 182163 dev-perl/Manifest
+  96198 dev-python/Manifest
+  58963 dev-ruby/Manifest
+  59736 dev-util/Manifest
+  58338 eclass/Manifest
+  55749 kde-base/Manifest
+ 110064 licenses/Manifest
+  35262 media-gfx/Manifest
+  53995 media-libs/Manifest
+  55607 media-plugins/Manifest
+  71911 media-sound/Manifest
+  34835 media-video/Manifest
+5747849 metadata/Manifest
+  47452 net-analyzer/Manifest
+  65989 net-misc/Manifest
+ 316787 profiles/Manifest
+  67784 sys-apps/Manifest
+  48971 x11-misc/Manifest
+  41475 x11-plugins/Manifest
+
-

Backwards Compatibility

+

Backwards Compatibility

The package Manifests should also be maintained as ONLY uncompressed in CVS.

For processing of all existing per-package Manifests, if compression is @@ -102,11 +183,10 @@ do not have an backwards compatibility issues.

As a side note, this breaks all manual interaction with Manifests such as grep, and so should only be applied to large Manifest2 files, -such as the MetaManifest. 384KiB is suggested as a arbitrary cut-off -point to start generating compressed Manifest2 files.

+such as the MetaManifest.

-

References

+

References

@@ -116,7 +196,7 @@