/[gentoo]/xml/htdocs/proj/en/glep/glep-0061.txt
Gentoo

Diff of /xml/htdocs/proj/en/glep/glep-0061.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.3 Revision 1.5
1GLEP: 61 1GLEP: 61
2Title: Manifest2 compression 2Title: Manifest2 compression
3Version: $Revision: 1.3 $ 3Version: $Revision: 1.5 $
4Last-Modified: $Date: 2010/01/13 01:14:35 $ 4Last-Modified: $Date: 2010/01/31 08:27:05 $
5Author: Robin Hugh Johnson <robbat2@gentoo.org> 5Author: Robin Hugh Johnson <robbat2@gentoo.org>
6Status: Draft 6Status: Draft
7Type: Standards Track 7Type: Standards Track
8Content-Type: text/x-rst 8Content-Type: text/x-rst
9Requires: 44 9Requires: 44
10Created: July 2008 10Created: July 2008
11Updated: October 2008, January 2010 11Updated: October 2008, January 2010
12Updates: 44 12Updates: 44
13Post-History: December 2009 13Post-History: December 2009, January 2010
14 14
15Abstract 15Abstract
16======== 16========
17Deals with compression of large Manifest2 files. 17Deals with compression of large Manifest2 files.
18 18
20========== 20==========
21With the introduction of MetaManifest, and full-tree Manifest coverage, 21With the introduction of MetaManifest, and full-tree Manifest coverage,
22we are faced with the possibility of having very large Manifests. 22we are faced with the possibility of having very large Manifests.
23 23
24Preliminary experiments with MetaManifest, show that with just the 24Preliminary experiments with MetaManifest, show that with just the
25existing per-package Manifests, the full MetaManifest, for a tree 25existing per-package Manifests, the full MetaManifest (top-level only,
26including metadata/, exceeds 8MiB in size. Applying common compression 26no first-level sub directories), for a tree including metadata/, exceeds
27can achieve a 50-60% reduction in this size. 278MiB in size. Applying common compression can achieve a 50-60% reduction
28in this size.
29
30Additionally, some of the larger already-existing Manifests in the tree
31can also be reduced.
28 32
29Specification 33Specification
30============= 34=============
35Creation of compressed Manifests:
36---------------------------------
3732KiB is suggested as a arbitrary cut-off point to start generating
38compressed Manifest2 files.
39
40The compression must only applied during the creation of a tree intended
41for end users. No Manifests stored in a VCS should be compressed in the
42VCS. For the main gentoo-portage tree, this means that the compressed
43Manifests should be generated using the CVS to Rsync process.
44
45The Manifest compression process is required to ensure that inconsistent
46compressed versions do not exist.
47
48Validation of Manifests:
49------------------------
31When searching for a Manifest2 file, if the basename form does not 50When searching for a Manifest2 file, if the basename form does not
32exist, the package manager should search in the same location using 51exist, the package manager should search in the same location using
33common compressed suffixes, and use the compressed file in place of the 52common compressed suffixes, and use the compressed file in place of the
34Manifest2. 53Manifest2.
35 54
36gzip, bzip2, lzma, xz should all be supported if available on the given 55gzip, bzip2, lzma, xz should all be supported if available on the given
37platform. In the case that multiple versions exist, the package manager 56platform. In the case that multiple versions exist, the package manager
38should simply pick one - they should be identical, differing only in 57should simply pick one - they should be identical, differing only in
39compression. 58compression.
40 59
41The Manifest generation process is required to ensure that inconsistent 60Example Results with a 32KiB cut-off, gzip algorithm
42compressed versions do not exist. 61====================================================
62As of 2010/01/30, the suggested cut-off would impact the following 21 existing
63Manifests, for a saving of nearly 900KiB::
64
65 Size Path
66 65788 app-doc/linux-gazette/Manifest
67 75739 app-office/openoffice-bin/Manifest
68 40534 app-text/texlive-core/Manifest
69 41710 dev-texlive/texlive-bibtexextra/Manifest
70 38197 dev-texlive/texlive-documentation-english/Manifest
71 129610 dev-texlive/texlive-fontsextra/Manifest
72 36022 dev-texlive/texlive-humanities/Manifest
73 686118 dev-texlive/texlive-latexextra/Manifest
74 43392 dev-texlive/texlive-latexrecommended/Manifest
75 33375 dev-texlive/texlive-mathextra/Manifest
76 39781 dev-texlive/texlive-pictures/Manifest
77 69567 dev-texlive/texlive-pstricks/Manifest
78 75460 dev-texlive/texlive-publishers/Manifest
79 50879 dev-texlive/texlive-science/Manifest
80 36711 kde-base/kde-l10n/Manifest
81 36539 media-gfx/bootsplash-themes/Manifest
82 33058 net-fs/autofs/Manifest
83 39781 www-client/firefox-bin/Manifest
84 48983 www-client/icecat/Manifest
85 60213 www-client/mozilla-firefox/Manifest
86 39065 x11-themes/gkrellm-themes/Manifest
87
88
89Additionally, with the MetaManifest proposal, the following new manifests would
90also be compressed, for a saving of nearly 4MiB::
91
92 Size Path
93 33442 app-admin/Manifest
94 71073 app-dicts/Manifest
95 35923 app-emacs/Manifest
96 45808 app-misc/Manifest
97 50169 app-text/Manifest
98 112786 dev-java/Manifest
99 65581 dev-libs/Manifest
100 42619 dev-lisp/Manifest
101 182163 dev-perl/Manifest
102 96198 dev-python/Manifest
103 58963 dev-ruby/Manifest
104 59736 dev-util/Manifest
105 58338 eclass/Manifest
106 55749 kde-base/Manifest
107 110064 licenses/Manifest
108 35262 media-gfx/Manifest
109 53995 media-libs/Manifest
110 55607 media-plugins/Manifest
111 71911 media-sound/Manifest
112 34835 media-video/Manifest
113 5747849 metadata/Manifest
114 47452 net-analyzer/Manifest
115 65989 net-misc/Manifest
116 316787 profiles/Manifest
117 67784 sys-apps/Manifest
118 48971 x11-misc/Manifest
119 41475 x11-plugins/Manifest
120
43 121
44Backwards Compatibility 122Backwards Compatibility
45======================= 123=======================
46The package Manifests should also be maintained as ONLY uncompressed in 124The package Manifests should also be maintained as ONLY uncompressed in
47CVS. 125CVS.
55MetaManifests may come into existence as compressed from the start, as 133MetaManifests may come into existence as compressed from the start, as
56do not have an backwards compatibility issues. 134do not have an backwards compatibility issues.
57 135
58As a side note, this breaks all manual interaction with Manifests 136As a side note, this breaks all manual interaction with Manifests
59such as grep, and so should only be applied to large Manifest2 files, 137such as grep, and so should only be applied to large Manifest2 files,
60such as the MetaManifest. 384KiB is suggested as a arbitary cut-off 138such as the MetaManifest.
61point to start generating compressed Manifest2 files.
62 139
63References 140References
64========== 141==========
65.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format. 142.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
66 http://www.gentoo.org/proj/en/glep/glep-0044.html 143 http://www.gentoo.org/proj/en/glep/glep-0044.html
67 144
68Copyright 145Copyright
69========= 146=========
70Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be 147Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
71distributed only subject to the terms and conditions set forth in the 148distributed only subject to the terms and conditions set forth in the
72Open Publication License, v1.0. 149Open Publication License, v1.0.
73 150
74vim: tw=72 ts=2 expandtab: 151vim: tw=72 ts=2 expandtab:

Legend:
Removed from v.1.3  
changed lines
  Added in v.1.5

  ViewVC Help
Powered by ViewVC 1.1.20