en/glep/glep-0037.txt

GLEP: 37
Title: Virtuals Deprecation
Version: $Revision: 1.1 $
Last-Modified: $Date: 2005/04/30 22:35:58 $
Author: Jason Stubbs <jstubbs@gentoo.org>
Status: deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 30-April-2005
Post-History: 30-April-2005, 5-Sep-2006


Status
======

What has been implemented so far diverges somewhat from what is 
stated in this GLEP.  As such, this GLEP (in its current form)
has been marked "deferred".

Credits
=======

Most ideas in this GLEP came out of discussion with Thomas de Grenier de
Latour.  Ciaran McCreesh, Brian Harring and Stephen Bennett have also provided
help in fleshing out the idea.


Abstract
========

This GLEP covers the pitfalls of the current virtuals system, the benefits of
using regular ebuilds to serve the purpose of virtuals and what needs to be
supported to make it viable.


Motivation
==========

The current virtuals system is decentralized; that is there is no way to find
information about a specific virtual other than to scan all packages for what
they provide.  There is also no way to tell whether an atom is a virtual or
not - yes, the "virtual/" prefix could have been used but it isn't which has
led to its abuse.

What this means is that portage must scan all installed packages for the
virtuals they provide, that profiles must provide a default for every single
virtual that portage might encounter and that every single atom that portage
processes must be checked against the list of virtuals.  Needless to say that
this causes quite a performance decrease.

The current virtuals system also has some other major shortcomings.  The most
well known case is virtual/jdk and kaffe.  Kaffe-1.1.4 implements the Java 1.4
API but can not satisfy a package that requires >=virtual/jdk-1.4 because
kaffe's versioning scheme differs.  (ED: Need to add some more here. ;)


Specification
=============

This GLEP recommends that virtuals become no different to regular packages.
Specifically, the standard virtual would include the DESCRIPTION, KEYWORDS,
IUSE and RDEPEND metadata.  An example would be something like this::

        DESCRIPTION="Java Development Kit 1.4"
        KEYWORDS="amd64 hppa ia64 ppc ppc64 sparc x86"
        RDEPEND="|| (
                     =dev-java/blackdown-jdk-1.4\*
                     =dev-java/ibm-jdk-bin-1.4*
                     =dev-java/jrockit-jdk-bin-1.4*
                     =dev-java/kaffe-1.1.4*
                     =dev-java/sun-jdk-1.4*
                    )"
        IUSE=""

However, there are some issues that have been brought up with doing this.


Consistency
-----------

Presently, it is very easy to remove packages even if others are dependent
on them, which can lead to broken emerges when packages rely on indirect
dependencies.  For example, if kdelibs is merged bringing in qt and then
qt is unmerged, attempting to merge kdebase will likely fail.  This becomes
a much bigger problem with virtuals as packages because the dependencies
are always indirect.

The resolution for this issue will be to add full dependency tracking and
verification to portage.  The details of how it will be done are outside the
scope of this GLEP, but essentially this means that portage will need to be
forced to unmerge a package that is depended on by another and will also be
able to scan and fix any broken dependencies.


Overrides
---------

Profiles currently specify the default provider of each virtual and users are
able to override these defaults using /etc/portage/profile/virtuals.  If
virtuals are replaced by regular packages and thus able to have arbitrarily
complex DEPENDs, the current method of overriding default virtuals can not
be extended to support this.

Before looking at a solution, lets look at how the current system works.  When
portage initializes, it searches installed packages for available virtuals.
It then searches profiles and user overrides and adds them to the available
providers list and/or changes the order of the providers so that overrides are
listed earlier.  Portage then expands any virtual atom it finds into an OR
list using the order decided upon at initialization.

To keep this behaviour available, this GLEP proposes a new file named
package.prefer.  In its basic form, this is just a list of package names
ordered by preference.  Portage would use it by reordering the atoms of any
OR list it processes to fit the order given by package.prefer.  For example,
if package.prefer contained "dev-java/kaffe" then:

::
        
        || (
            dev-java/blackdown-jdk
            dev-java/sun-jdk
            dev-java/kaffe
           )

would be processed as:

::
        
        || (
            dev-java/kaffe
            dev-java/blackdown-jdk
            dev-java/sun-jdk
           )

In its basic form, package.prefer already covers profile and user overrides.
However, this GLEP proposes that any type of atom be usable.  This will be
accomplished by checking for intersections of the atoms in the OR list and
atoms in the preferred list.  When an intersection is found, both atoms
would be specified in a sublist, which would then be treated as a ranged dep.
For example, if package.prefer contained "<=dev-java/sun-jdk-1.4" then:

::
        
        || (
            >=dev-java/blackdown-jdk-1.3
            >=dev-java/sun-jdk-1.3
           )

would be processed as:

::
        
        || (
            (
              <=dev-java/sun-jdk-1.4
              >=dev-java/sun-jdk-1.3
            )
            >=dev-java/blackdown-jdk-1.3
            >=dev-java/sun-jdk-1.3
           )

Ranged deps are outside of the scope of this GLEP.


Rationale
=========

The number one advantage is that it offers more power to both the user and
the developer.  Flexibility of virtuals is far greater in this scheme and
fulfills requirements that exist already.  It also means that the maintainers
of profiles will not need to list a default for every virtual.  The user
benefits by being able to easily gather a list of providers of a virtual as
well as their control being extended to allow selection where there is a
choice within any package.

Portage code also benefits from this scheme as virtuals will no longer
require special handling or dual implementations of essentially the same
feature, for example USE-based PROVIDEs.  This scheme is also much easier to
optimize which will benefit the processing of all packages.  It also means
that any additions to the DEPEND vocabulary become available for use in the
definitions of virtuals.


Backwards Compatibility
=======================

Compatibility will begin by making 2.0.51.20 treat unknown virtuals like
regular packages.  When the tree is stripped of PROVIDEs and "virtuals"
override files, the only virtuals that these portages will use are those that
the user has specified and those gleaned from installed packages.  Any
unknown virtual will be treated like a regular package and looked for in the
tree.

The next major version of portage (2.1.0) will support consistency
checking.  The only remaining issue is that of user overrides.  The old
method will work even with new style virtuals.  The only catch is that
complex virtuals - that is virtuals that will install more than one package -
may not be overridable satisfactorally.

Dropping of support of current style virtuals is planned for the following
major version of portage (2.2.0).  When the time comes to release it, scripts
will be written to create packages from the existing virtuals system as well
as to create appropriate package.prefer overrides within the profiles.


Copyright
=========

This document has been placed in the public domain.
1	GLEP: 37
2	Title: Virtuals Deprecation
3	Version: $Revision: 1.1 $
4	Last-Modified: $Date: 2005/04/30 22:35:58 $
5	Author: Jason Stubbs <jstubbs@gentoo.org>
6	Status: deferred
7	Type: Standards Track
8	Content-Type: text/x-rst
9	Created: 30-April-2005
10	Post-History: 30-April-2005, 5-Sep-2006
11
12
13	Status
14	======
15
16	What has been implemented so far diverges somewhat from what is
17	stated in this GLEP. As such, this GLEP (in its current form)
18	has been marked "deferred".
19
20	Credits
21	=======
22
23	Most ideas in this GLEP came out of discussion with Thomas de Grenier de
24	Latour. Ciaran McCreesh, Brian Harring and Stephen Bennett have also provided
25	help in fleshing out the idea.
26
27
28	Abstract
29	========
30
31	This GLEP covers the pitfalls of the current virtuals system, the benefits of
32	using regular ebuilds to serve the purpose of virtuals and what needs to be
33	supported to make it viable.
34
35
36	Motivation
37	==========
38
39	The current virtuals system is decentralized; that is there is no way to find
40	information about a specific virtual other than to scan all packages for what
41	they provide. There is also no way to tell whether an atom is a virtual or
42	not - yes, the "virtual/" prefix could have been used but it isn't which has
43	led to its abuse.
44
45	What this means is that portage must scan all installed packages for the
46	virtuals they provide, that profiles must provide a default for every single
47	virtual that portage might encounter and that every single atom that portage
48	processes must be checked against the list of virtuals. Needless to say that
49	this causes quite a performance decrease.
50
51	The current virtuals system also has some other major shortcomings. The most
52	well known case is virtual/jdk and kaffe. Kaffe-1.1.4 implements the Java 1.4
53	API but can not satisfy a package that requires >=virtual/jdk-1.4 because
54	kaffe's versioning scheme differs. (ED: Need to add some more here. ;)
55
56
57	Specification
58	=============
59
60	This GLEP recommends that virtuals become no different to regular packages.
61	Specifically, the standard virtual would include the DESCRIPTION, KEYWORDS,
62	IUSE and RDEPEND metadata. An example would be something like this::
63
64	DESCRIPTION="Java Development Kit 1.4"
65	KEYWORDS="amd64 hppa ia64 ppc ppc64 sparc x86"
66	RDEPEND="\|\| (
67	=dev-java/blackdown-jdk-1.4\*
68	=dev-java/ibm-jdk-bin-1.4*
69	=dev-java/jrockit-jdk-bin-1.4*
70	=dev-java/kaffe-1.1.4*
71	=dev-java/sun-jdk-1.4*
72	)"
73	IUSE=""
74
75	However, there are some issues that have been brought up with doing this.
76
77
78	Consistency
79	-----------
80
81	Presently, it is very easy to remove packages even if others are dependent
82	on them, which can lead to broken emerges when packages rely on indirect
83	dependencies. For example, if kdelibs is merged bringing in qt and then
84	qt is unmerged, attempting to merge kdebase will likely fail. This becomes
85	a much bigger problem with virtuals as packages because the dependencies
86	are always indirect.
87
88	The resolution for this issue will be to add full dependency tracking and
89	verification to portage. The details of how it will be done are outside the
90	scope of this GLEP, but essentially this means that portage will need to be
91	forced to unmerge a package that is depended on by another and will also be
92	able to scan and fix any broken dependencies.
93
94
95	Overrides
96	---------
97
98	Profiles currently specify the default provider of each virtual and users are
99	able to override these defaults using /etc/portage/profile/virtuals. If
100	virtuals are replaced by regular packages and thus able to have arbitrarily
101	complex DEPENDs, the current method of overriding default virtuals can not
102	be extended to support this.
103
104	Before looking at a solution, lets look at how the current system works. When
105	portage initializes, it searches installed packages for available virtuals.
106	It then searches profiles and user overrides and adds them to the available
107	providers list and/or changes the order of the providers so that overrides are
108	listed earlier. Portage then expands any virtual atom it finds into an OR
109	list using the order decided upon at initialization.
110
111	To keep this behaviour available, this GLEP proposes a new file named
112	package.prefer. In its basic form, this is just a list of package names
113	ordered by preference. Portage would use it by reordering the atoms of any
114	OR list it processes to fit the order given by package.prefer. For example,
115	if package.prefer contained "dev-java/kaffe" then:
116
117	::
118
119	\|\| (
120	dev-java/blackdown-jdk
121	dev-java/sun-jdk
122	dev-java/kaffe
123	)
124
125	would be processed as:
126
127	::
128
129	\|\| (
130	dev-java/kaffe
131	dev-java/blackdown-jdk
132	dev-java/sun-jdk
133	)
134
135	In its basic form, package.prefer already covers profile and user overrides.
136	However, this GLEP proposes that any type of atom be usable. This will be
137	accomplished by checking for intersections of the atoms in the OR list and
138	atoms in the preferred list. When an intersection is found, both atoms
139	would be specified in a sublist, which would then be treated as a ranged dep.
140	For example, if package.prefer contained "<=dev-java/sun-jdk-1.4" then:
141
142	::
143
144	\|\| (
145	>=dev-java/blackdown-jdk-1.3
146	>=dev-java/sun-jdk-1.3
147	)
148
149	would be processed as:
150
151	::
152
153	\|\| (
154	(
155	<=dev-java/sun-jdk-1.4
156	>=dev-java/sun-jdk-1.3
157	)
158	>=dev-java/blackdown-jdk-1.3
159	>=dev-java/sun-jdk-1.3
160	)
161
162	Ranged deps are outside of the scope of this GLEP.
163
164
165	Rationale
166	=========
167
168	The number one advantage is that it offers more power to both the user and
169	the developer. Flexibility of virtuals is far greater in this scheme and
170	fulfills requirements that exist already. It also means that the maintainers
171	of profiles will not need to list a default for every virtual. The user
172	benefits by being able to easily gather a list of providers of a virtual as
173	well as their control being extended to allow selection where there is a
174	choice within any package.
175
176	Portage code also benefits from this scheme as virtuals will no longer
177	require special handling or dual implementations of essentially the same
178	feature, for example USE-based PROVIDEs. This scheme is also much easier to
179	optimize which will benefit the processing of all packages. It also means
180	that any additions to the DEPEND vocabulary become available for use in the
181	definitions of virtuals.
182
183
184	Backwards Compatibility
185	=======================
186
187	Compatibility will begin by making 2.0.51.20 treat unknown virtuals like
188	regular packages. When the tree is stripped of PROVIDEs and "virtuals"
189	override files, the only virtuals that these portages will use are those that
190	the user has specified and those gleaned from installed packages. Any
191	unknown virtual will be treated like a regular package and looked for in the
192	tree.
193
194	The next major version of portage (2.1.0) will support consistency
195	checking. The only remaining issue is that of user overrides. The old
196	method will work even with new style virtuals. The only catch is that
197	complex virtuals - that is virtuals that will install more than one package -
198	may not be overridable satisfactorally.
199
200	Dropping of support of current style virtuals is planned for the following
201	major version of portage (2.2.0). When the time comes to release it, scripts
202	will be written to create packages from the existing virtuals system as well
203	as to create appropriate package.prefer overrides within the profiles.
204
205
206	Copyright
207	=========
208
209	This document has been placed in the public domain.