| 1 |
g2boojum |
1.1 |
GLEP: 25 |
| 2 |
|
|
Title: Distfile Patching Support |
| 3 |
|
|
Version: $Revision: 1.0 $ |
| 4 |
|
|
Last-Modified: $Date: 2004/04/01 01:07:40 $ |
| 5 |
|
|
Author: Brian Harring <ferringb@gentoo.org> |
| 6 |
|
|
Status: Draft |
| 7 |
|
|
Type: Standards Track |
| 8 |
|
|
Content-Type: text/x-rst |
| 9 |
|
|
Created: 6-Mar-2004 |
| 10 |
|
|
Post-History: 4-Apr-2004 |
| 11 |
|
|
|
| 12 |
|
|
Abstract |
| 13 |
|
|
======== |
| 14 |
|
|
|
| 15 |
|
|
The intention of this GLEP is to propose the creation of patching support for |
| 16 |
|
|
portage, and iron out the implementation details. |
| 17 |
|
|
|
| 18 |
|
|
Motivation |
| 19 |
|
|
========== |
| 20 |
|
|
|
| 21 |
|
|
Reduce the bandwidth load placed on our mirrors by decreasing the amount of |
| 22 |
|
|
bytes transferred when upgrading between versions. Side benefit of this is to |
| 23 |
|
|
significantly decrease the download requirements for users lacking broadband. |
| 24 |
|
|
|
| 25 |
|
|
Binary patches vs GNUDiff patches |
| 26 |
|
|
================================= |
| 27 |
|
|
|
| 28 |
|
|
Most people are familiar with diff patches (unified diff for example)- this |
| 29 |
|
|
glep is specifically proposing the use of an actual binary differencer. The |
| 30 |
|
|
reason for this is that diff patches are line based- you change a single |
| 31 |
|
|
character in a line, and the whole line must be included in the patch. Binary |
| 32 |
|
|
differencers work at the byte level- it encodes just that byte. In that |
| 33 |
|
|
respect binary patches are often much more efficient then diff patches. |
| 34 |
|
|
|
| 35 |
|
|
Further, the ability to reverse a unified patch is due to the fact the diff |
| 36 |
|
|
includes **both** the original line, and the modified line. The author isn't |
| 37 |
|
|
aware of any binary differencer that is able to create patches the can be |
| 38 |
|
|
reversed- basically they're unidirectional, the patch that is generated can |
| 39 |
|
|
only be used to upgrade or downgrade the version, not both. The plus side of |
| 40 |
|
|
this limitation is a significantly decreased patch size. |
| 41 |
|
|
|
| 42 |
|
|
The choice of binary patches over diff patches pretty much comes down to the |
| 43 |
|
|
fact they're smaller- example being a kdelibs binary patch for 3.1.4->3.1.5 is |
| 44 |
|
|
75kb, the equivalent diff patch is 123kb, and is unable to result in a correct |
| 45 |
|
|
md5 [1]_. |
| 46 |
|
|
|
| 47 |
|
|
Currently, this glep is proposing only the usage of binary patches- that's not |
| 48 |
|
|
to say (with a fair amount of work) it couldn't be extended to support |
| 49 |
|
|
standard diffs. |
| 50 |
|
|
|
| 51 |
|
|
Rationale |
| 52 |
|
|
========= |
| 53 |
|
|
|
| 54 |
|
|
The difference between source releases typically isn't very large, especially |
| 55 |
|
|
for minor releases. As an example, kdelibs-3.1.4.tar.bz2 is 10.53 MB, and |
| 56 |
|
|
kdelibs-3.1.5.tar.bz2 is 10.54 MB. A bzip2'ed patch between those versions is |
| 57 |
|
|
75.6 kb [2]_, less then 1% the size of 3.1.5's tbz2. |
| 58 |
|
|
|
| 59 |
|
|
Specification |
| 60 |
|
|
============= |
| 61 |
|
|
|
| 62 |
|
|
Quite a few sections of gentoo are affected- mirroring, the portage tree, and |
| 63 |
|
|
portage itself. |
| 64 |
|
|
|
| 65 |
|
|
Additions to the tree |
| 66 |
|
|
--------------------- |
| 67 |
|
|
|
| 68 |
|
|
For adding patch info into the tree, this glep proposes a global patch list |
| 69 |
|
|
(stored in profiles as patches.global), and individual patch lists stored in |
| 70 |
|
|
relevant package directories (named patches). Using the kernel packages as an |
| 71 |
|
|
example, a global list of patches enables us to create a patch once, add an |
| 72 |
|
|
entry, and have all kernel packages benefit from that single entry. Both |
| 73 |
|
|
patches.global, and individual package patch files share the same format: |
| 74 |
|
|
|
| 75 |
|
|
:: |
| 76 |
|
|
|
| 77 |
|
|
MD5 md5-value patch-url size MD5 md5-value ref-file size UMD5 md5-value new-file size |
| 78 |
|
|
|
| 79 |
|
|
For those familiar with digest file layout, this should look familiar. |
| 80 |
|
|
Essentially, chksum type, value, filename, size. The UMD5 chksum type is just |
| 81 |
|
|
the uncompressed md5/size of the file- so if the UMD5 were for a bzip2 |
| 82 |
|
|
compressed file, it would be the md5 value/size of the uncompressed file. |
| 83 |
|
|
And an example: |
| 84 |
|
|
|
| 85 |
|
|
:: |
| 86 |
|
|
|
| 87 |
|
|
MD5 ccd5411b3558326cbce0306fcae32e26 http://dev.gentoo.org/~ferringb/patches/kdelibs-3.1.4-3.1.5.patch.bz2 75687 MD5 82c265de78d53c7060a09c5cb1a78942 kdelibs-3.1.4.tar.bz2 10537433 UMD5 0b1908a51e739c07ff5a88e189d2f7a9 kdelibs-3.1.5.tar.bz2 48056320 |
| 88 |
|
|
|
| 89 |
|
|
In the above example, the md5sum of |
| 90 |
|
|
http://dev.gentoo.org/~ferringb/patches/kdelibs-3.1.4-3.1.5.patch.bz2 is |
| 91 |
|
|
calculated, compared to the stored value, and then the file size is checked. |
| 92 |
|
|
The one difference is the UMD5 checksum type- the md5 value and the size are |
| 93 |
|
|
specific to the *uncompressed* file. Continuing, for cases where the patch |
| 94 |
|
|
will reside on one of our mirrors, the patch filename would be sufficient. |
| 95 |
|
|
|
| 96 |
|
|
Finally, note that this is a unidirectional patch- using the above example, |
| 97 |
|
|
kdelibs-3.1.4-3.1.5 can **only** be used to upgrade from 3.1.4 to 3.1.5, not |
| 98 |
|
|
in reverse (originally explained in `Binary patches vs GNUDiff patches`_). |
| 99 |
|
|
|
| 100 |
|
|
Portage Implementation |
| 101 |
|
|
---------------------- |
| 102 |
|
|
|
| 103 |
|
|
This glep proposes the patching support should be (at this stage) optional- |
| 104 |
|
|
specifically, enabled via FEATURES="patching". |
| 105 |
|
|
|
| 106 |
|
|
Fetching |
| 107 |
|
|
'''''''' |
| 108 |
|
|
|
| 109 |
|
|
When patching is enabled, the global patch list is read, and the packages |
| 110 |
|
|
patch list is read. From there, portage determines what files could be used |
| 111 |
|
|
as a base for patching to the desired file- further, determining if it's |
| 112 |
|
|
actually worth patching (case where it wouldn't be is when the target file is |
| 113 |
|
|
less then the sum of the patches needed). Any patches to be used are fetched, |
| 114 |
|
|
and md5 verified. |
| 115 |
|
|
|
| 116 |
|
|
Reconstruction |
| 117 |
|
|
'''''''''''''' |
| 118 |
|
|
|
| 119 |
|
|
Upon fetching and md5 verification of patch(es), the desired file is |
| 120 |
|
|
reconstructed. Assuming reconstruction didn't return any errors, the target |
| 121 |
|
|
file has its uncompressed md5sum calculated and verified, then is recompressed |
| 122 |
|
|
and the compressed md5sum calculated. At this point, if the compressed md5 |
| 123 |
|
|
matches the md5 stored in the tree, then portage transfers the file into |
| 124 |
|
|
distfiles, and continues on it's merry way. |
| 125 |
|
|
|
| 126 |
|
|
If the compressed md5 is different from the tree's value, then the (proposed) |
| 127 |
|
|
md5 database is updated with new compressed md5. Details of this database |
| 128 |
|
|
(and the issue it addresses) follow. |
| 129 |
|
|
|
| 130 |
|
|
Compressed MD5sums: |
| 131 |
|
|
''''''''''''''''''' |
| 132 |
|
|
|
| 133 |
|
|
There will be instances where a file is reconstructed perfectly, recompressed, |
| 134 |
|
|
and the recompressed md5sum differs from what is stored in the tree- the |
| 135 |
|
|
problem is that the md5sum of a compressed file is inherently tied to the |
| 136 |
|
|
compressor version/options used to compress the original source. |
| 137 |
|
|
|
| 138 |
|
|
===================== |
| 139 |
|
|
The Problem in Detail |
| 140 |
|
|
===================== |
| 141 |
|
|
|
| 142 |
|
|
A good example of this problem is related to bzip2 versions used for |
| 143 |
|
|
compression. Between bzip2 0.9x and bzip2 1.x, there was a subtle change in |
| 144 |
|
|
the compressor resulting in a slightly better compression result- end result |
| 145 |
|
|
being a different file, eg a different md5sum. Assuming compressor versions |
| 146 |
|
|
are the same, there also is the issue of what compression level the target |
| 147 |
|
|
source was originally compressed at- was it compressed with -9, -8 or -7? |
| 148 |
|
|
That's just a sampling of the various original settings that must be accounted |
| 149 |
|
|
for, and that's limited to gzip/bzip2; other compressors will add to the |
| 150 |
|
|
number of variables to be accounted for to produce an exact recreation of the |
| 151 |
|
|
compressed md5sum. |
| 152 |
|
|
|
| 153 |
|
|
Tracking the compressor version and options originally used isn't really a |
| 154 |
|
|
valid option- assuming all options were accounted for, clients would still be |
| 155 |
|
|
required to have multiple versions of the same compressor installed just for |
| 156 |
|
|
the sake of recreating a compressed md5sum *even though* the uncompressed |
| 157 |
|
|
source's md5 has already been verified. |
| 158 |
|
|
|
| 159 |
|
|
===================== |
| 160 |
|
|
The Proposed Solution |
| 161 |
|
|
===================== |
| 162 |
|
|
|
| 163 |
|
|
The creation of a clientside flatfile/db of valid alternate md5/size pairs |
| 164 |
|
|
would enable portage to handle perfectly reconstructed files, that have a |
| 165 |
|
|
different md5sum due to compression differences. The proposed format is thus: |
| 166 |
|
|
|
| 167 |
|
|
:: |
| 168 |
|
|
|
| 169 |
|
|
MD5 md5sum orig-file size MD5 md5sum [ optional new-name ] size |
| 170 |
|
|
|
| 171 |
|
|
Example: |
| 172 |
|
|
|
| 173 |
|
|
:: |
| 174 |
|
|
|
| 175 |
|
|
MD5 984146931906a7d53300b29f58f6a899 OOo_1.0.3_source.tar.bz2 165475319 MD5 0733dd85ed44d88d1eabed704d579721 165444187 |
| 176 |
|
|
|
| 177 |
|
|
An alternate md5/size pair for a file would be added **only** when the |
| 178 |
|
|
uncompressed source's md5/size has been verified, yet upon recompression the |
| 179 |
|
|
md5 differs. For cleansing of older md5/size pairs from this db, a utility |
| 180 |
|
|
would be required- the author suggests the addition of a distfiles-cleaning |
| 181 |
|
|
utility to portage, with the ability to also cleanse old md5/size pairs when |
| 182 |
|
|
the file the pair was created for no longer exists in distfiles. |
| 183 |
|
|
|
| 184 |
|
|
Where to store the database is debatable- /etc/portage or /var/cache/edb are |
| 185 |
|
|
definite options. |
| 186 |
|
|
|
| 187 |
|
|
The reasoning for allowing for an optional new-name is that it provides needed |
| 188 |
|
|
functionality should anyone attempt to extend portage to allow for clients to |
| 189 |
|
|
change the compression used for a source (eg, recompress all gzip files as |
| 190 |
|
|
bzip2). Granted, no such code or attempt has been made, but nothing is lost |
| 191 |
|
|
by leaving the option open should the request/attempt be made. |
| 192 |
|
|
|
| 193 |
|
|
A potential gotcha of adding this support is that in environments where the |
| 194 |
|
|
distfiles directory is shared out to multiple systems, this db must be shared |
| 195 |
|
|
also. |
| 196 |
|
|
|
| 197 |
|
|
|
| 198 |
|
|
|
| 199 |
|
|
Distfile Mirror Additions |
| 200 |
|
|
------------------------- |
| 201 |
|
|
|
| 202 |
|
|
One issue of contention is where these files will actually be stored. As of |
| 203 |
|
|
the writing of this glep, a full distfiles mirror is roughly around 40 gb- a |
| 204 |
|
|
rough estimate by the author places the space requirements for patches for |
| 205 |
|
|
each version at a total of around 4gb. Note this isn't even remotely a hard |
| 206 |
|
|
figure yet, and a better figure is being checked into currently. |
| 207 |
|
|
|
| 208 |
|
|
Regardless of the exact space figure, finding a place to store the patches |
| 209 |
|
|
will be problematic. Expansion of the required mirror space (essentially just |
| 210 |
|
|
swallowing the patches storage requirement) is unlikely, since it was one of |
| 211 |
|
|
the main arguements against the now defunct glep9 attempt [2]_. A couple of |
| 212 |
|
|
ideas that have been put forth to handle the additional space requirements are |
| 213 |
|
|
as follows- |
| 214 |
|
|
|
| 215 |
|
|
1) Identification of mirrors willing to handle the extra space requirements- |
| 216 |
|
|
essentially create an additional patch mirror tier. |
| 217 |
|
|
|
| 218 |
|
|
2) Mirroring only a patch for certain package versions, rather then full |
| 219 |
|
|
source. Using kdelibs-3.1.5 as an example, only the patch would be mirrored |
| 220 |
|
|
(rather then the full 10.53 MB source). Downside to this approach is that a |
| 221 |
|
|
user who is downloading kdelibs for the first time would either need to pull |
| 222 |
|
|
it from the original SRC_URI (placing the burden onto the upstream mirror), or |
| 223 |
|
|
pull the 3.1.4 version, and the patch- pulling 63k more then if they had just |
| 224 |
|
|
pulled the full version. The kdelibs 3.1.4/3.1.5 example is something of an |
| 225 |
|
|
optimal case- not all versions will have such miniscule patches. |
| 226 |
|
|
|
| 227 |
|
|
3) A variation on the idea above, essentially mirroring only the patch for |
| 228 |
|
|
the oldest version(s) of a package; eg, kdelibs currently has version 3.05, |
| 229 |
|
|
3.1.5, 3.2.0, and 3.2.1- the mirrors would only carry a patch for 3.05, not |
| 230 |
|
|
full source (think RESTRICT="fetch"). One plus to this is that patches to |
| 231 |
|
|
downgrade in version are smaller then the patches to upgrade in version- there |
| 232 |
|
|
are exceptions to this, but they're hard to find. A major downside to this |
| 233 |
|
|
approach is A) a user would have to sync up to get the patchlists for that |
| 234 |
|
|
version, B) creation of a set of patches to go backwards in version (see |
| 235 |
|
|
`Binary patches vs GNUDiff patches`_).. |
| 236 |
|
|
|
| 237 |
|
|
Of the options listed above, the first is the easiest, although the second |
| 238 |
|
|
could be made to work. Feedback and any possible alternatives would be |
| 239 |
|
|
greatly appreciated. |
| 240 |
|
|
|
| 241 |
|
|
Patch Creation |
| 242 |
|
|
-------------- |
| 243 |
|
|
|
| 244 |
|
|
Maintenance of patch lists, and the actual patch creation ought to be managed |
| 245 |
|
|
by a high level script- essentally a dev says "I want a patch between this |
| 246 |
|
|
version, and that version: make it so", the script churns away |
| 247 |
|
|
creating/updating the patch list, and generating the patch locally. The |
| 248 |
|
|
utility next uploads the new patch to /space/distfiles-local on dev.gentoo.org |
| 249 |
|
|
(exempting if it's not a locally generated patch), and repoman is used to |
| 250 |
|
|
commit the updated patch list. |
| 251 |
|
|
|
| 252 |
|
|
What would be preferable (although possibly wishful thinking), is if hardware |
| 253 |
|
|
could be co-opted for automatic patch generation, rather then forcing it upon |
| 254 |
|
|
the devs- something akin to how files are pulled onto the mirror automatically |
| 255 |
|
|
for new ebuilds. |
| 256 |
|
|
|
| 257 |
|
|
The initial bulk of patches to get will be generated by the author, to ease |
| 258 |
|
|
the transition and offer patches for people to test out. |
| 259 |
|
|
|
| 260 |
|
|
Backwards Compatability |
| 261 |
|
|
======================= |
| 262 |
|
|
|
| 263 |
|
|
As noted in `The Proposed Solution`_, a system using patching and sharing out |
| 264 |
|
|
it's distfiles must share out it's alternate md5 db. Any system that uses the |
| 265 |
|
|
distfiles share must support the alternate md5 db also. If this is considered |
| 266 |
|
|
enough of an issue, it is conceivable to place reconstructed sources with an |
| 267 |
|
|
alternate md5 into a subdirectory of distdir- portage only looks within |
| 268 |
|
|
distdir, unwilling to descend into subdirectories. |
| 269 |
|
|
|
| 270 |
|
|
Also note that `Distfile Mirror Additions`_ may add additional backwards |
| 271 |
|
|
compatability issues, depending on what solution is accepted. |
| 272 |
|
|
|
| 273 |
|
|
Reference Implementation |
| 274 |
|
|
======================== |
| 275 |
|
|
|
| 276 |
|
|
TODO |
| 277 |
|
|
|
| 278 |
|
|
References |
| 279 |
|
|
========== |
| 280 |
|
|
.. [1] http://dev.gentoo.org/~ferringb/patches/kdelibs-3.1.4-3.1.5.{patch,diff}.bz2. |
| 281 |
|
|
.. [2] kdelibs-3.1.4-3.1.5.patch.bz2, switching format patch, created via diffball-0.4_pre4 (diffball is available at http://sourceforge.net/projects/diffball) |
| 282 |
|
|
Bzip2 -9 compressed, the patch is 75,687 bytes, uncompressed it is 337,649 bytes. The patch is available at http://dev.gentoo.org/~ferringb/kdelibs-3.1.4-3.1.5.patch.bz2 for those curious. |
| 283 |
|
|
.. [3] Glep9, 'Gentoo Package Update System' |
| 284 |
|
|
(http://glep.gentoo.org/glep-0009.html) |
| 285 |
|
|
|
| 286 |
|
|
Copyright |
| 287 |
|
|
========= |
| 288 |
|
|
|
| 289 |
|
|
This document has been placed in the public domain. |
Parent Directory
| 
Revision Log