--- xml/htdocs/proj/en/glep/glep-0025.html 2005/04/01 01:32:29 1.3 +++ xml/htdocs/proj/en/glep/glep-0025.html 2006/10/10 20:25:14 1.4 @@ -8,9 +8,252 @@ --> - + GLEP 25 -- Distfile Patching Support - + @@ -34,7 +277,7 @@ - + @@ -42,7 +285,7 @@ - + @@ -51,8 +294,8 @@
Version:1.3
Last-Modified:2005/04/01 01:32:19
Last-Modified:2005/04/01 01:32:19
Author:Brian Harring <ferringb at gentoo.org>
Type:Standards Track
Content-Type:text/x-rst
Content-Type:text/x-rst
Created:6-Mar-2004

-
-

Contents

+
+

Contents

-
-

Abstract

+
+

Abstract

The intention of this GLEP is to propose the creation of patching support for portage, and iron out the implementation details.

-
-

Status

+
+

Status

Timed out

-
-

Motivation

+
+

Motivation

Reduce the bandwidth load placed on our mirrors by decreasing the amount of bytes transferred when upgrading between versions. Side benefit of this is to significantly decrease the download requirements for users lacking broadband.

-
-

Binary patches vs GNUDiff patches

+
+

Binary patches vs GNUDiff patches

Most people are familiar with diff patches (unified diff for example)- this glep is specifically proposing the use of an actual binary differencer. The reason for this is that diff patches are line based- you change a single @@ -118,19 +361,19 @@ to say (with a fair amount of work) it couldn't be extended to support standard diffs.

-
-

Rationale

+
+

Rationale

The difference between source releases typically isn't very large, especially for minor releases. As an example, kdelibs-3.1.4.tar.bz2 is 10.53 MB, and kdelibs-3.1.5.tar.bz2 is 10.54 MB. A bzip2'ed patch between those versions is 75.6 kb [2], less then 1% the size of 3.1.5's tbz2.

-
-

Specification

+
+

Specification

Quite a few sections of gentoo are affected- mirroring, the portage tree, and portage itself.

-
-

Additions to the tree

+
+

Additions to the tree

For adding patch info into the tree, this glep proposes a global patch list (stored in profiles as patches.global), and individual patch lists stored in relevant package directories (named patches). Using the kernel packages as an @@ -158,12 +401,12 @@ kdelibs-3.1.4-3.1.5 can only be used to upgrade from 3.1.4 to 3.1.5, not in reverse (originally explained in Binary patches vs GNUDiff patches).

-
-

Portage Implementation

+
+

Portage Implementation

This glep proposes the patching support should be (at this stage) optional- specifically, enabled via FEATURES="patching".

-
-

Fetching

+
+

Fetching

When patching is enabled, the global patch list is read, and the packages patch list is read. From there, portage determines what files could be used as a base for patching to the desired file- further, determining if it's @@ -171,8 +414,8 @@ less then the sum of the patches needed). Any patches to be used are fetched, and md5 verified.

-
-

Reconstruction

+
+

Reconstruction

Upon fetching and md5 verification of patch(es), the desired file is reconstructed. Assuming reconstruction didn't return any errors, the target file has its uncompressed md5sum calculated and verified, then is recompressed @@ -183,14 +426,14 @@ md5 database is updated with new compressed md5. Details of this database (and the issue it addresses) follow.

-
-

Compressed MD5sums:

+
+

Compressed MD5sums:

There will be instances where a file is reconstructed perfectly, recompressed, and the recompressed md5sum differs from what is stored in the tree- the problem is that the md5sum of a compressed file is inherently tied to the compressor version/options used to compress the original source.

-
-

The Problem in Detail

+
+

The Problem in Detail

A good example of this problem is related to bzip2 versions used for compression. Between bzip2 0.9x and bzip2 1.x, there was a subtle change in the compressor resulting in a slightly better compression result- end result @@ -207,8 +450,8 @@ the sake of recreating a compressed md5sum even though the uncompressed source's md5 has already been verified.

-
-

The Proposed Solution

+
+

The Proposed Solution

The creation of a clientside flatfile/db of valid alternate md5/size pairs would enable portage to handle perfectly reconstructed files, that have a different md5sum due to compression differences. The proposed format is thus:

@@ -238,8 +481,8 @@
-
-

Distfile Mirror Additions

+
+

Distfile Mirror Additions

One issue of contention is where these files will actually be stored. As of the writing of this glep, a full distfiles mirror is roughly around 40 gb- a rough estimate by the author places the space requirements for patches for @@ -274,8 +517,8 @@ could be made to work. Feedback and any possible alternatives would be greatly appreciated.

-
-

Patch Creation

+
+

Patch Creation

Maintenance of patch lists, and the actual patch creation ought to be managed by a high level script- essentally a dev says "I want a patch between this version, and that version: make it so", the script churns away @@ -291,8 +534,8 @@ the transition and offer patches for people to test out.

-
-

Backwards Compatibility

+
+

Backwards Compatibility

As noted in The Proposed Solution, a system using patching and sharing out it's distfiles must share out it's alternate md5 db. Any system that uses the distfiles share must support the alternate md5 db also. If this is considered @@ -302,12 +545,12 @@

Also note that Distfile Mirror Additions may add additional backwards compatibility issues, depending on what solution is accepted.

-
-

Reference Implementation

+ -
-

References

+
+

References

@@ -317,29 +560,30 @@
-
[2](1, 2) kdelibs-3.1.4-3.1.5.patch.bz2, switching format patch, created via diffball-0.4_pre4 (diffball is available at http://sourceforge.net/projects/diffball) +
[2](1, 2) kdelibs-3.1.4-3.1.5.patch.bz2, switching format patch, created via diffball-0.4_pre4 (diffball is available at http://sourceforge.net/projects/diffball) Bzip2 -9 compressed, the patch is 75,687 bytes, uncompressed it is 337,649 bytes. The patch is available at http://dev.gentoo.org/~ferringb/kdelibs-3.1.4-3.1.5.patch.bz2 for those curious.
-
[3]Glep9, 'Gentoo Package Update System' +
[3]Glep9, 'Gentoo Package Update System' (http://glep.gentoo.org/glep-0009.html)
- -