/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Diff of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.19 Revision 1.40
1<?xml version='1.0' encoding="UTF-8"?> 1<?xml version='1.0' encoding="UTF-8"?>
2<!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> 2<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.19 2004/07/02 15:47:15 swift Exp $ --> 3<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.40 2007/06/21 23:46:16 rane Exp $ -->
4
5<!--
6 Synchronised with rev 1.15 on
7 http://www.thedoh.com/viewcvs/distcc/html/distcc.xml
8-->
9 4
10<guide link="/doc/en/distcc.xml"> 5<guide link="/doc/en/distcc.xml">
11 6
12<title>Gentoo Distcc Documentation</title> 7<title>Gentoo Distcc Documentation</title>
13 8
14<author title="Author"> 9<author title="Author">
15 <mail link="lisa@gentoo.org">Lisa Seelye</mail> 10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
16</author> 11</author>
17
18<author title="Editor"> 12<author title="Editor">
19 <mail link="vapier@gentoo.org">Mike Frysinger</mail> 13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
20</author> 14</author>
21
22<author title="Editor"> 15<author title="Editor">
23 <mail link="erwin@gentoo.org">Erwin</mail> 16 <mail link="erwin@gentoo.org">Erwin</mail>
24</author> 17</author>
25
26<author title="Editor"> 18<author title="Editor">
27 <mail link="swift@gentoo.org">Sven Vermeulen</mail> 19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
28</author> 20</author>
29 21<author title="Editor">
22 <mail link="pylon@gentoo.org">Lars Weiler</mail>
23</author>
30<author title="Reviewer"> 24<author title="Reviewer">
31 <mail link="blubber@gentoo.org">Tiemo Kieft</mail> 25 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
32</author> 26</author>
33 27
34<abstract> 28<abstract>
35This document serves as a HOWTO for using distcc with Gentoo. 29This document serves as a HOWTO for using distcc with Gentoo.
36</abstract> 30</abstract>
37 31
38<!-- The content of this document is licensed under the CC-BY-SA license --> 32<!-- The content of this document is licensed under the CC-BY-SA license -->
39<!-- See http://creativecommons.org/licenses/by-sa/1.0 --> 33<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
40<license/> 34<license/>
41 35
42<version>1.7</version> 36<version>1.19</version>
43<date>July 02, 2004</date> 37<date>2007-06-22</date>
44 38
45<chapter> 39<chapter>
46<title>Introduction</title> 40<title>Introduction</title>
47<section> 41<section>
48<title>What is distcc?</title> 42<title>What is distcc?</title>
49<body> 43<body>
50 44
51<p> 45<p>
52Distcc is a program designed to distribute compiling tasks across a network to 46Distcc is a program designed to distribute compiling tasks across a network to
53participating hosts. It is comprised of a server, <c>distccd</c> and a client 47participating hosts. It is comprised of a server, <c>distccd</c>, and a client
54program, <c>distcc</c>. Distcc can work transparently with <uri 48program, <c>distcc</c>. Distcc can work transparently with <uri
55link="http://ccache.samba.org">ccache</uri> and Portage with a little work. 49link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
50little setup.
56</p> 51</p>
57 52
58</body> 53</body>
59</section> 54</section>
60<section> 55<section>
61<title>Using distcc to bootstrap</title> 56<title>Using distcc to bootstrap</title>
62<body> 57<body>
63 58
64<p> 59<p>
65If you are planning on using distcc to help you bootstrap a Gentoo installation, 60If you are planning on using distcc to help you bootstrap a Gentoo
66make sure you read the part on <uri link="#bootstrapping">Using distcc to 61installation, make sure you read the section <uri link="#bootstrapping">Using
67Bootstrap</uri> which is situated further down in this document. 62distcc to Bootstrap</uri>, which is situated further down in this document.
68</p> 63</p>
69 64
70</body> 65</body>
71</section> 66</section>
72</chapter> 67</chapter>
… …
76<section> 71<section>
77<title>Dependencies</title> 72<title>Dependencies</title>
78<body> 73<body>
79 74
80<p> 75<p>
81To start off with, to use Distcc all of the computers on your network need to 76In order to use Distcc, all of the computers on your network need to have the
82have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but 77same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
83mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it. 78mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
79errors.
84</p> 80</p>
85 81
86</body> 82</body>
87</section> 83</section>
88<section> 84<section>
89<title>Installing Distcc</title> 85<title>Installing Distcc</title>
90<body> 86<body>
91 87
92<p> 88<p>
93There are a couple of options you should know about before you start 89There are a couple of options you should be aware of before you start
94installing distcc. 90installing distcc.
95</p> 91</p>
96 92
97<p> 93<p>
98Distcc ships with a graphical monitor to monitor tasks that your computer is 94Distcc ships with a graphical monitor to monitor tasks that your computer is
… …
103 99
104<pre caption="Installing distcc"> 100<pre caption="Installing distcc">
105# <i>emerge distcc</i> 101# <i>emerge distcc</i>
106</pre> 102</pre>
107 103
104<impo>
105Remember, you must be sure to install distcc on all of your participating
106machines.
107</impo>
108
108</body> 109</body>
109</section> 110</section>
110<section> 111<section>
111<title>Setting up Portage to use Distcc</title> 112<title>Setting up Portage to use Distcc</title>
112<body> 113<body>
113 114
114<p> 115<p>
115Setting up distcc is easy to do with Portage. Execute the following steps on 116Setting up Portage to use distcc is easy. Execute the following steps on
116each system that should participate in the distributed compiling: 117each system that should participate in the distributed compiling:
117</p> 118</p>
118 119
119<pre caption="Integrating Distcc and Portage"> 120<pre caption="Integrating Distcc and Portage">
120# <i>emerge distcc</i> 121# <i>emerge distcc distcc-config</i>
121# <i>nano -w /etc/make.conf</i> 122# <i>nano -w /etc/make.conf</i>
122<comment>(Set N to a suitable number for your particular setup)</comment> 123<comment>(Set N to a suitable number for your particular setup)</comment>
123<comment>(A common heuristic strategy is twice the number of CPUs + 1)</comment> 124<comment>(A common strategy is setting N as twice the number of total CPUs + 1 available)</comment>
124MAKEOPTS="-jN" 125MAKEOPTS="-jN"
125<comment>(Add distcc to your FEATURES variable)</comment> 126<comment>(Add distcc to your FEATURES)</comment>
126FEATURES="distcc" 127FEATURES="distcc"
127</pre> 128</pre>
128 129
130</body>
131</section>
132<section>
133<title>Specifying Participating Hosts</title>
134<body>
135
129<p> 136<p>
130Next you have to specify what hosts you want to use. Use the
131<c>distcc-config</c> command to set the list of hosts. Here is an example of 137Use the <c>distcc-config</c> command to set the list of hosts. Here is an
132some hosts that might be in your list: 138example of some hosts that might be in your list:
133</p> 139</p>
134 140
135<pre caption="Examples of host definitions"> 141<pre caption="Examples of host definitions">
136192.168.0.1 192.168.0.2 192.168.0.3 142192.168.0.1 192.168.0.2 192.168.0.3
137192.168.0.1/2 192.168.0.2 192.168.0.3/10 143192.168.0.1/2 192.168.0.2 192.168.0.3/10
138192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4 144192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
139@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3 145@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
140<comment>(There are also several other methods of setting up hosts. See the 146<comment>(There are also several other methods of setting up hosts. See the
141distcc manpage for more details.)</comment> 147distcc manpage for more details.)</comment>
148<comment>If you wish to compile on the local machine you should put 'localhost'
149in the hosts list. Conversely if you do not wish to use the local machine to
150compile (which is often the case) omit it from the hosts list. On a slow
151machine using localhost may actually slow things down. Make sure to test your
152settings for performance.</comment>
142</pre> 153</pre>
143 154
144<p> 155<p>
145It may all look complicated, but in most cases a variant of line 1 or 2 will 156It may all look complicated, but in most cases a variant of line 1 or 2 will
146work. 157work.
147</p> 158</p>
148 159
149<ul>
150 <li>Line 1 is just a space-delimited list of hosts</li>
151 <li>
152 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
153 defining the <c>/N</c> setting) to send that host at any given time
154 </li>
155</ul>
156
157<p> 160<p>
158Since most people won't be using lines 3 or 4, I'll <uri 161Since most people won't be using lines 3 or 4, I'll <uri
159link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc 162link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
160docs for more information. 163docs (man distcc) for more information.
161</p> 164</p>
162 165
163<p> 166<p>
164For instance, to set the first line in the previous example: 167For instance, to set the first line in the previous example:
165</p> 168</p>
166 169
167<pre caption="Sample command to set the hosts"> 170<pre caption="Sample command to set the hosts">
168# <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i> 171# <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
169</pre> 172</pre>
173
174<p>
175Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
176<c>--allow</c> directive to allow only hosts you trust. For added security,
177you should also use the <c>--listen</c> directive to tell the distcc daemon
178what IP to listen on (for multi-homed systems). More information on distcc
179security can be found at <uri
180link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
181</p>
182
183<impo>
184It is important to use --allow and --listen. Please read the distccd manpage
185or the above security document for more information.
186</impo>
187
170 188
171<p> 189<p>
172Now start the distcc daemon on all the participating computers: 190Now start the distcc daemon on all the participating computers:
173</p> 191</p>
174 192
… …
184<section> 202<section>
185<title>Setting up Distcc to Work With Automake</title> 203<title>Setting up Distcc to Work With Automake</title>
186<body> 204<body>
187 205
188<p> 206<p>
189This is in some cases easier than the Portage setup. What you have to do is 207This is, in some cases, easier than the Portage setup. What you have to do is
190update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path> 208update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
191in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>). 209in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
192However, there is a caveat. If you use ccache you have to put distcc after 210However, there is a caveat. If you use ccache you have to put distcc after
193the ccache part: 211the ccache part:
194</p> 212</p>
195 213
196<pre caption="Setting your path"> 214<pre caption="Setting your path">
197# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i> 215# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
216<comment>(You can put this in your ~/.bashrc or equivalent file to have the PATH
217set every time you log in)</comment>
198</pre> 218</pre>
199 219
200<p> 220<p>
201Then, as you would normally type <c>make</c>, you would type <c>make -jN</c> 221Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
202(where N is an integer). The value of N depends on your network and the types 222(where N is an integer). The value of N depends on your network and the types
203of computers you are using to compile. For me, I have a dual processor P3 and 223of computers you are using to compile. Test your own settings to find the
204a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon 224number that yields the best performance.
205Thunderbird) and I use -j5. Test your own settings to find the most
206performant number.
207</p> 225</p>
208 226
209</body> 227</body>
210</section> 228</section>
211</chapter> 229</chapter>
212 230
213<chapter> 231<chapter>
214<title>Cross-Compiling</title> 232<title>Cross-Compiling</title>
215<section> 233<section>
216<title>A Note on Cross-Compiling</title>
217<body> 234<body>
218 235
219<p> 236<p>
220Cross-compiling is using one architecture to build programs for another 237Cross-compiling is using one architecture to build programs for another
221architecture. This can be as simple as using an Athlon (i686) to build a 238architecture. This can be as simple as using an Athlon (i686) to build a
222program for a K6-2 (i586), or using a Sparc to build a program for a ppc. 239program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
223</p> 240is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
224 241Cross-compiling Guide</uri>.
225</body>
226</section>
227<section>
228<title>A Personal Note on Cross-compiling</title>
229<body>
230
231<p>
232I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
233don't have any non-x86 machines. I can create things that work <e>in
234theory</e>, but I have to rely on other people to test what I write.
235That works to an extent, but it's tough. ;-)
236</p>
237
238<p>
239In the near future I hope to acquire a Sparc of some type to put Gentoo on so
240I can play around in-house.
241</p> 242</p>
242 243
243</body> 244</body>
244</section> 245</section>
245</chapter> 246</chapter>
246 247
247<chapter id="bootstrapping"> 248<chapter id="bootstrapping">
248<title>Using Distcc to Bootstrap</title> 249<title>Using Distcc to Bootstrap</title>
249<section> 250<section>
250<title>Step 1: Configure Portage </title> 251<title>Step 1: Configure Portage</title>
251<body> 252<body>
252 253<!--
254Note that this will need to become a link to the old 2005.1 networked handbooks
255once the 2006.0 handbooks are released. Until then, the link can stay.
256-->
253<p> 257<p>
254Boot your new box with a Gentoo Linux LiveCD and follow the <uri 258Boot your new box with a Gentoo Linux LiveCD and follow the <uri
255link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri> 259link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
260up until the bootstrapping part. (See the <uri
261link="/doc/en/faq.xml#stage12">Gentoo FAQ</uri> for more information about
256up until the bootstrapping part. Then configure Portage to use distcc: 262bootstrapping.) Then configure Portage to use distcc:
257</p> 263</p>
258 264
259<pre caption="Preliminary Setup"> 265<pre caption="Preliminary Setup">
260# <i>nano -w /etc/make.conf</i> 266# <i>nano -w /etc/make.conf</i>
261<comment>(Add distcc to the FEATURES variable)</comment> 267<comment>(Add distcc to the FEATURES</comment>
262FEATURES="distcc" 268FEATURES="distcc"
263<comment>(Set distcc's directory for temporary space)</comment> 269<comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
264DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc" 270+1 available)</comment>
265<comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
266MAKEOPTS="-jN" 271MAKEOPTS="-jN"
267</pre> 272</pre>
268 273
274<pre caption="Setting your path">
275# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
276</pre>
277
269</body> 278</body>
270</section> 279</section>
271<section> 280<section>
272<title>Step 2: Getting Distcc</title> 281<title>Step 2: Getting Distcc</title>
273<body> 282<body>
… …
275<p> 284<p>
276Install distcc: 285Install distcc:
277</p> 286</p>
278 287
279<pre caption="Getting Distcc on the new box"> 288<pre caption="Getting Distcc on the new box">
280# <i>USE='-*' emerge --nodeps distcc</i> 289# <i>USE='-*' emerge --nodeps sys-devel/distcc sys-devel/distcc-config</i>
281</pre> 290</pre>
282 291
283</body> 292</body>
284</section> 293</section>
285<section> 294<section>
286<title>Step 3: Setting Up Distcc</title> 295<title>Step 3: Setting Up Distcc</title>
287<body> 296<body>
288 297
289<p> 298<p>
290Add a "distcc" user to the <path>/etc/passwd</path> file and run
291<c>distcc-config</c> to setup distcc: 299Run <c>distcc-config --install</c> to setup distcc:
292</p> 300</p>
293 301
294<pre caption="Final distcc setup"> 302<pre caption="Final distcc setup">
295# <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
296<comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment> 303<comment>(Substitute host1, host2, ... with the IP number(s) of the
304participating hosts)</comment>
297# <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i> 305# <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
306<comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
307192.168.0.4 192.168.0.6"</i></comment>
298</pre> 308</pre>
299 309
300<p> 310<p>
301distcc is now set up to bootstrap! Continue the official installation 311Distcc is now set up to bootstrap! Continue with the official installation
302instructions and <e>don't forget</e> to re-emerge distcc after <c>emerge 312instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
303system</c>. This is to make sure that all of the dependencies you want are 313system</c>. This is to make sure that all of the dependencies you want are
304installed as well. 314installed as well.
305</p> 315</p>
306 316
307<note> 317<note>
308During bootstrap and <c>emerge system</c> Distcc may not appear to be used. 318During bootstrap and <c>emerge system</c> distcc may not appear to be used.
309This is expected as some ebuilds do not work well with distcc, so they 319This is expected as some ebuilds do not work well with distcc, so they
310intentionally disable it. 320intentionally disable it.
311</note> 321</note>
312 322
313</body> 323</body>
… …
315</chapter> 325</chapter>
316 326
317<chapter> 327<chapter>
318<title>Troubleshooting</title> 328<title>Troubleshooting</title>
319<section> 329<section>
320<title>Mozilla and Xfree</title> 330<title>Some Packages Don't Use Distcc</title>
321<body> 331<body>
322 332
323<p> 333<p>
324As you emerge various packages, you'll notice that some of them aren't being 334As you emerge various packages, you'll notice that some of them aren't being
325distributed (and aren't being built in parallel). This is because the 335distributed (and aren't being built in parallel). This may happen because the
326developers of the Mozilla and Xfree ebuilds intentionally disable parallel 336package's Makefile doesn't support parallel operations or the maintainer of the
327building because it is known to cause problems. 337ebuild has explicitly disabled parallel operations due to a known problem.
328</p> 338</p>
329 339
330<p> 340<p>
331Sometimes distcc might cause a package to fail to compile. If this happens 341Sometimes distcc might cause a package to fail to compile. If this happens
332for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us. 342for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
333</p>
334
335</body>
336</section>
337<section>
338<title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
339<body>
340
341<p>
342With such a long title any explanation here is almost irrelevent. However, if
343you plan to use distcc across hosts that have the <uri
344link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
345not, you will run into problems.
346</p>
347
348<p>
349The solution requires a little foresight on your part; you have to run
350<c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
351enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
352which are a good thing to do since for the most part the protections offered
353by both packages is a good thing and is transparent to the user.
354</p> 343</p>
355 344
356</body> 345</body>
357</section> 346</section>
358<section> 347<section>
… …
362<p> 351<p>
363If you have different GCC versions on your hosts, there will likely be very 352If you have different GCC versions on your hosts, there will likely be very
364weird problems. The solution is to make certain all hosts have the same GCC 353weird problems. The solution is to make certain all hosts have the same GCC
365version. 354version.
366</p> 355</p>
356
357<p>
358Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
359<c>gcc</c>. This means that if you're mixing i686 machines with other types
360(i386, i586) you will run into problems. A workaround for this may be to
361<c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
362</p>
363
364<impo>
365Doing this explicitly redefines some behaviour of Portage and may have some
366weird results in the future. Only do this if you're mixing CHOSTs.
367</impo>
368
367 369
368</body> 370</body>
369</section> 371</section>
370</chapter> 372</chapter>
371 373
… …
382parameter it will run once. However, if you pass it a number it will update 384parameter it will run once. However, if you pass it a number it will update
383every N seconds, where N is the argument you passed. 385every N seconds, where N is the argument you passed.
384</p> 386</p>
385 387
386<p> 388<p>
387The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c> 389The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
388in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment 390in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
389and it is quite lovely. 391and it is quite lovely. For Gentoo the GUI monitor has been called
392<c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
393<c>distccmon-gnome</c>.
390</p> 394</p>
391 395
392<pre caption="Starting the monitors"> 396<pre caption="Starting the monitors">
393# <i>distccmon-text N</i> 397# <i>distccmon-text N</i>
394<comment>(Or ...)</comment> 398<comment>(Or)</comment>
395# <i>distccmon-gnome</i> 399# <i>distccmon-gui</i>
396</pre> 400<comment>To monitor Portage's distcc usage you can use:</comment>
397 401# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
398</body> 402<comment>(Or)</comment>
399</section> 403# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
400</chapter>
401
402<chapter>
403<title>Future Plans for Distcc and Gentoo</title>
404<section>
405<title>distcc-subnetscan</title>
406<body>
407
408<p>
409<c>distcc-subnetscan</c> is a perl program in development that will scan a
410subnet for hosts that have a participating distcc daemon. Even this will be
411enhanced to test if the daemon on a remote host will conform to a specific
412<c>CHOST</c> setting to make cross-compiling easier.
413</p> 404</pre>
414 405
415<p> 406<impo>
416The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">lisa's 407If your distcc directory is elsewhere, change the DISTCC_DIR variable
417dev page</uri> until a more formal home can be found. 408accordingly.
418</p> 409</impo>
419 410
420</body> 411</body>
421</section> 412</section>
422</chapter> 413</chapter>
423</guide> 414</guide>
Legend:
Removed from v.1.19  
changed lines
  Added in v.1.40
  ViewVC Help
Powered by ViewVC 1.1.13