en/glep/glep-0009.txt

GLEP: 9
Title: Gentoo Package Update System
Version: $Revision: $
Last-Modified: $Date: 2003/07/25 22:31:38 $
Author: John J. Whitney <jjw@linuxmail.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 19-Jul-2003
Post-History: 25-Jul-2003


Abstract
========

This document proposes an official package updating system for Gentoo Linux.
The Deltup project has been developed for this purpose. [#DELTUP]_

As packages grow larger the amount of redundant data keeps increasing.  Updating
existing tarballs by patching is the natural way to handle source updates.

Motivation
==========

This system will reduce mirror loads (potentially mirror size as well) and
significantly speed up downloads, making Gentoo much more attractive for users 
with low-bandwidth connections.

Server Implementation
=====================

I propose that the patches be put onto the Gentoo Mirrors and stored in a new 
directory called "patchfiles" which could be placed beside "distfiles".

It would be advantageous to have a list of available patches within the portage
tree so that it can be updated during "emerge sync".  A file named "dtu.list" 
can be created and placed in $PORTDIR/profiles.

If a machine can be set up to generate patches it should contain a local mirror
of distfiles which it can monitor for added packages.  When a package is added
to distfiles the machine can try to determine the previous tarball so a patch
can be made and placed in the patchfiles dir.  In addition, special-case patches
can be added manually.

The dtu.list file will be maintained by a special script.  Whenever patches 
are added or removed to the patchfiles dir, the script will make necessary 
additions/removals in dtu.list.  This will be done with minimal changes in the
file so it can be synchronized efficiently.

Client Implementation
=====================

The system will be optional for users and can be enabled by making portage
invoke efetch through the FETCHCOMMAND environment variable [#TINYHOWTO]_.

When a package fetch is requested, the efetch/fetchcommand scripts (part of 
Deltup) will scan the dtu.list file for updates and try downloading and applying
them if they exist, or fall back to a full package download if they don't or if
the patching process fails.

Rationale
=========
The most controversial feature has been the addition of dtu.list to the portage
tree, so in this section I will list the reasons I support it.

- Flexibility.  Without it, there must be a standard naming scheme which we
  would be stuck with once the system is in place.  Changing the system would 
  require serious compatibility breaks.  With the dtu.list file we can change
  the naming scheme easily without problems, or even have several different
  naming schemes.

- Features.  Without patch information detecting different upgrade paths would 
  be impossible.  Split package patching would also be impossible.  If the info
  is available we can use it to find the quickest upgrade path, like jumping 
  from a .0 release, or even disable certain patches if there are problems with 
  them.

- It would be impossible to know which packages to upgrade from in some cases,
  including renamed packages.

- Knowing which patches are available will eliminate the overhead of attempting
  to download patches which don't exist.

The dtu.list file will contain several hundred kilobytes of data.  That has
caused some concern over how efficiently it can be rsynced.  To address these
concerns the file's format will be plaintext and care has been taken to
minimize the number of changes as removals/additions are made.

Backwards Compatibility
=======================

There are no backwards compatibility issues since Deltup can generate correct 
package MD5sums.


Conclusion
==========

I suggest we start with a scaled-down implementation and provide more as the
demand increases.  All of the necesary code is already written and working in
non-official tests.

References
==========

.. [#DELTUP] http://sourceforge.net/projects/deltup
.. [#PATCHES] ftp://sunsite.dk/projects/deltup/patchfiles
.. [#TINYHOWTO] Tiny Deltup HOWTO
   (http://www.thedoh.com/linux/HOWTO/deltup)

Copyright
=========

This document has been placed in the public domain.
1	GLEP: 9
2	Title: Gentoo Package Update System
3	Version: $Revision: $
4	Last-Modified: $Date: 2003/07/25 22:31:38 $
5	Author: John J. Whitney <jjw@linuxmail.org>
6	Status: Draft
7	Type: Standards Track
8	Content-Type: text/x-rst
9	Created: 19-Jul-2003
10	Post-History: 25-Jul-2003
11
12
13	Abstract
14	========
15
16	This document proposes an official package updating system for Gentoo Linux.
17	The Deltup project has been developed for this purpose. [#DELTUP]_
18
19	As packages grow larger the amount of redundant data keeps increasing. Updating
20	existing tarballs by patching is the natural way to handle source updates.
21
22	Motivation
23	==========
24
25	This system will reduce mirror loads (potentially mirror size as well) and
26	significantly speed up downloads, making Gentoo much more attractive for users
27	with low-bandwidth connections.
28
29	Server Implementation
30	=====================
31
32	I propose that the patches be put onto the Gentoo Mirrors and stored in a new
33	directory called "patchfiles" which could be placed beside "distfiles".
34
35	It would be advantageous to have a list of available patches within the portage
36	tree so that it can be updated during "emerge sync". A file named "dtu.list"
37	can be created and placed in $PORTDIR/profiles.
38
39	If a machine can be set up to generate patches it should contain a local mirror
40	of distfiles which it can monitor for added packages. When a package is added
41	to distfiles the machine can try to determine the previous tarball so a patch
42	can be made and placed in the patchfiles dir. In addition, special-case patches
43	can be added manually.
44
45	The dtu.list file will be maintained by a special script. Whenever patches
46	are added or removed to the patchfiles dir, the script will make necessary
47	additions/removals in dtu.list. This will be done with minimal changes in the
48	file so it can be synchronized efficiently.
49
50	Client Implementation
51	=====================
52
53	The system will be optional for users and can be enabled by making portage
54	invoke efetch through the FETCHCOMMAND environment variable [#TINYHOWTO]_.
55
56	When a package fetch is requested, the efetch/fetchcommand scripts (part of
57	Deltup) will scan the dtu.list file for updates and try downloading and applying
58	them if they exist, or fall back to a full package download if they don't or if
59	the patching process fails.
60
61	Rationale
62	=========
63	The most controversial feature has been the addition of dtu.list to the portage
64	tree, so in this section I will list the reasons I support it.
65
66	- Flexibility. Without it, there must be a standard naming scheme which we
67	would be stuck with once the system is in place. Changing the system would
68	require serious compatibility breaks. With the dtu.list file we can change
69	the naming scheme easily without problems, or even have several different
70	naming schemes.
71
72	- Features. Without patch information detecting different upgrade paths would
73	be impossible. Split package patching would also be impossible. If the info
74	is available we can use it to find the quickest upgrade path, like jumping
75	from a .0 release, or even disable certain patches if there are problems with
76	them.
77
78	- It would be impossible to know which packages to upgrade from in some cases,
79	including renamed packages.
80
81	- Knowing which patches are available will eliminate the overhead of attempting
82	to download patches which don't exist.
83
84	The dtu.list file will contain several hundred kilobytes of data. That has
85	caused some concern over how efficiently it can be rsynced. To address these
86	concerns the file's format will be plaintext and care has been taken to
87	minimize the number of changes as removals/additions are made.
88
89	Backwards Compatibility
90	=======================
91
92	There are no backwards compatibility issues since Deltup can generate correct
93	package MD5sums.
94
95
96	Conclusion
97	==========
98
99	I suggest we start with a scaled-down implementation and provide more as the
100	demand increases. All of the necesary code is already written and working in
101	non-official tests.
102
103	References
104	==========
105
106	.. [#DELTUP] http://sourceforge.net/projects/deltup
107	.. [#PATCHES] ftp://sunsite.dk/projects/deltup/patchfiles
108	.. [#TINYHOWTO] Tiny Deltup HOWTO
109	(http://www.thedoh.com/linux/HOWTO/deltup)
110
111	Copyright
112	=========
113
114	This document has been placed in the public domain.