/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Diff of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.27 Revision 1.40
1<?xml version='1.0' encoding="UTF-8"?> 1<?xml version='1.0' encoding="UTF-8"?>
2<!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> 2<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.27 2005/07/01 22:35:19 smithj Exp $ --> 3<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.40 2007/06/21 23:46:16 rane Exp $ -->
4 4
5<guide link="/doc/en/distcc.xml"> 5<guide link="/doc/en/distcc.xml">
6 6
7<title>Gentoo Distcc Documentation</title> 7<title>Gentoo Distcc Documentation</title>
8 8
9<author title="Author"> 9<author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail> 10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11</author> 11</author>
12
13<author title="Editor"> 12<author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail> 13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15</author> 14</author>
16
17<author title="Editor"> 15<author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail> 16 <mail link="erwin@gentoo.org">Erwin</mail>
19</author> 17</author>
20
21<author title="Editor"> 18<author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail> 19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23</author> 20</author>
24
25<author title="Editor"> 21<author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail> 22 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27</author> 23</author>
28
29<author title="Reviewer"> 24<author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail> 25 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31</author> 26</author>
32 27
33<abstract> 28<abstract>
34This document serves as a HOWTO for using distcc with Gentoo. 29This document serves as a HOWTO for using distcc with Gentoo.
35</abstract> 30</abstract>
36 31
37<!-- The content of this document is licensed under the CC-BY-SA license --> 32<!-- The content of this document is licensed under the CC-BY-SA license -->
38<!-- See http://creativecommons.org/licenses/by-sa/1.0 --> 33<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
39<license/> 34<license/>
40 35
41<version>1.11</version> 36<version>1.19</version>
42<date>2005-03-25</date> 37<date>2007-06-22</date>
43 38
44<chapter> 39<chapter>
45<title>Introduction</title> 40<title>Introduction</title>
46<section> 41<section>
47<title>What is distcc?</title> 42<title>What is distcc?</title>
49 44
50<p> 45<p>
51Distcc is a program designed to distribute compiling tasks across a network to 46Distcc is a program designed to distribute compiling tasks across a network to
52participating hosts. It is comprised of a server, <c>distccd</c>, and a client 47participating hosts. It is comprised of a server, <c>distccd</c>, and a client
53program, <c>distcc</c>. Distcc can work transparently with <uri 48program, <c>distcc</c>. Distcc can work transparently with <uri
54link="http://ccache.samba.org">ccache</uri> and Portage with a little work. 49link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
50little setup.
55</p> 51</p>
56 52
57</body> 53</body>
58</section> 54</section>
59<section> 55<section>
60<title>Using distcc to bootstrap</title> 56<title>Using distcc to bootstrap</title>
61<body> 57<body>
62 58
63<p> 59<p>
64If you are planning on using distcc to help you bootstrap a Gentoo installation, 60If you are planning on using distcc to help you bootstrap a Gentoo
65make sure you read the section <uri link="#bootstrapping">Using distcc to 61installation, make sure you read the section <uri link="#bootstrapping">Using
66Bootstrap</uri>, which is situated further down in this document. 62distcc to Bootstrap</uri>, which is situated further down in this document.
67</p> 63</p>
68 64
69</body> 65</body>
70</section> 66</section>
71</chapter> 67</chapter>
75<section> 71<section>
76<title>Dependencies</title> 72<title>Dependencies</title>
77<body> 73<body>
78 74
79<p> 75<p>
80In order to use Distcc, all of the computers on your network need to 76In order to use Distcc, all of the computers on your network need to have the
81have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but 77same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
82mixing 3.3.x and 3.2.x <b>will</b> result in your programs having errors in them. 78mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
79errors.
83</p> 80</p>
84 81
85</body> 82</body>
86</section> 83</section>
87<section> 84<section>
102 99
103<pre caption="Installing distcc"> 100<pre caption="Installing distcc">
104# <i>emerge distcc</i> 101# <i>emerge distcc</i>
105</pre> 102</pre>
106 103
104<impo>
105Remember, you must be sure to install distcc on all of your participating
106machines.
107</impo>
108
107</body> 109</body>
108</section> 110</section>
109<section> 111<section>
110<title>Setting up Portage to use Distcc</title> 112<title>Setting up Portage to use Distcc</title>
111<body> 113<body>
114Setting up Portage to use distcc is easy. Execute the following steps on 116Setting up Portage to use distcc is easy. Execute the following steps on
115each system that should participate in the distributed compiling: 117each system that should participate in the distributed compiling:
116</p> 118</p>
117 119
118<pre caption="Integrating Distcc and Portage"> 120<pre caption="Integrating Distcc and Portage">
119# <i>emerge distcc</i> 121# <i>emerge distcc distcc-config</i>
120# <i>nano -w /etc/make.conf</i> 122# <i>nano -w /etc/make.conf</i>
121<comment>(Set N to a suitable number for your particular setup)</comment> 123<comment>(Set N to a suitable number for your particular setup)</comment>
122<comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment> 124<comment>(A common strategy is setting N as twice the number of total CPUs + 1 available)</comment>
123MAKEOPTS="-jN" 125MAKEOPTS="-jN"
124<comment>(Add distcc to your FEATURES variable)</comment> 126<comment>(Add distcc to your FEATURES)</comment>
125FEATURES="distcc" 127FEATURES="distcc"
126</pre> 128</pre>
127 129
128</body> 130</body>
129</section> 131</section>
130<section> 132<section>
131<title>Specifying Participating Hosts</title> 133<title>Specifying Participating Hosts</title>
132<body> 134<body>
133 135
134<p> 136<p>
135Use the <c>distcc-config</c> command to set the list of hosts. Here is an example of 137Use the <c>distcc-config</c> command to set the list of hosts. Here is an
136some hosts that might be in your list: 138example of some hosts that might be in your list:
137</p> 139</p>
138 140
139<pre caption="Examples of host definitions"> 141<pre caption="Examples of host definitions">
140192.168.0.1 192.168.0.2 192.168.0.3 142192.168.0.1 192.168.0.2 192.168.0.3
141192.168.0.1/2 192.168.0.2 192.168.0.3/10 143192.168.0.1/2 192.168.0.2 192.168.0.3/10
142192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4 144192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
143@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3 145@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
144<comment>(There are also several other methods of setting up hosts. See the 146<comment>(There are also several other methods of setting up hosts. See the
145distcc manpage for more details.)</comment> 147distcc manpage for more details.)</comment>
148<comment>If you wish to compile on the local machine you should put 'localhost'
149in the hosts list. Conversely if you do not wish to use the local machine to
150compile (which is often the case) omit it from the hosts list. On a slow
151machine using localhost may actually slow things down. Make sure to test your
152settings for performance.</comment>
146</pre> 153</pre>
147 154
148<p> 155<p>
149It may all look complicated, but in most cases a variant of line 1 or 2 will 156It may all look complicated, but in most cases a variant of line 1 or 2 will
150work. 157work.
151</p> 158</p>
152 159
153<ul>
154 <li>Line 1 is just a space-delimited list of hosts</li>
155 <li>
156 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
157 defining the <c>/N</c> setting) to send that host at any given time
158 </li>
159</ul>
160
161<p> 160<p>
162Since most people won't be using lines 3 or 4, I'll <uri 161Since most people won't be using lines 3 or 4, I'll <uri
163link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc 162link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
164docs for more information. 163docs (man distcc) for more information.
165</p> 164</p>
166 165
167<p> 166<p>
168For instance, to set the first line in the previous example: 167For instance, to set the first line in the previous example:
169</p> 168</p>
172# <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i> 171# <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
173</pre> 172</pre>
174 173
175<p> 174<p>
176Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the 175Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
177<c>--allow</c> directive to allow only hosts you trust. For added security, you should also use the <c>--listen</c> 176<c>--allow</c> directive to allow only hosts you trust. For added security,
178directive to tell the distcc daemon what IP to listen on (for multi-homed 177you should also use the <c>--listen</c> directive to tell the distcc daemon
179systems). More information on distcc security can be found at <uri link="http://distcc.samba.org/security.html">Distcc Security 178what IP to listen on (for multi-homed systems). More information on distcc
180Design</uri>. 179security can be found at <uri
180link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
181</p> 181</p>
182
183<impo>
184It is important to use --allow and --listen. Please read the distccd manpage
185or the above security document for more information.
186</impo>
187
182 188
183<p> 189<p>
184Now start the distcc daemon on all the participating computers: 190Now start the distcc daemon on all the participating computers:
185</p> 191</p>
186 192
205the ccache part: 211the ccache part:
206</p> 212</p>
207 213
208<pre caption="Setting your path"> 214<pre caption="Setting your path">
209# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i> 215# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
216<comment>(You can put this in your ~/.bashrc or equivalent file to have the PATH
217set every time you log in)</comment>
210</pre> 218</pre>
211 219
212<p> 220<p>
213Then, as you would normally type <c>make</c>, you would type <c>make -jN</c> 221Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
214(where N is an integer). The value of N depends on your network and the types 222(where N is an integer). The value of N depends on your network and the types
215of computers you are using to compile. For me, I have a dual processor P3 and 223of computers you are using to compile. Test your own settings to find the
216a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
217Thunderbird) and I use -j5. Test your own settings to find the number that
218yields the best performance. 224number that yields the best performance.
219</p> 225</p>
220 226
221</body> 227</body>
222</section> 228</section>
223</chapter> 229</chapter>
224 230
225<chapter> 231<chapter>
226<title>Cross-Compiling</title> 232<title>Cross-Compiling</title>
227<section> 233<section>
228<title>A Note on Cross-Compiling</title>
229<body> 234<body>
230 235
231<p> 236<p>
232Cross-compiling is using one architecture to build programs for another 237Cross-compiling is using one architecture to build programs for another
233architecture. This can be as simple as using an Athlon (i686) to build a 238architecture. This can be as simple as using an Athlon (i686) to build a
234program for a K6-2 (i586), or using a Sparc to build a program for a ppc. 239program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
235</p> 240is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
236 241Cross-compiling Guide</uri>.
237</body>
238</section>
239<section>
240<title>An Introduction to Cross-Compiling</title>
241<body>
242
243<p>
244If you want to give cross-compiling a try you can to follow <uri
245link="http://dev.gentoo.org/~vapier/CROSS-COMPILE-HOWTO">The Cross Compile
246HOWTO</uri>; Crossdev is deprecated.
247</p> 242</p>
248 243
249</body> 244</body>
250</section> 245</section>
251</chapter> 246</chapter>
253<chapter id="bootstrapping"> 248<chapter id="bootstrapping">
254<title>Using Distcc to Bootstrap</title> 249<title>Using Distcc to Bootstrap</title>
255<section> 250<section>
256<title>Step 1: Configure Portage</title> 251<title>Step 1: Configure Portage</title>
257<body> 252<body>
258 253<!--
254Note that this will need to become a link to the old 2005.1 networked handbooks
255once the 2006.0 handbooks are released. Until then, the link can stay.
256-->
259<p> 257<p>
260Boot your new box with a Gentoo Linux LiveCD and follow the <uri 258Boot your new box with a Gentoo Linux LiveCD and follow the <uri
261link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri> 259link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
260up until the bootstrapping part. (See the <uri
261link="/doc/en/faq.xml#stage12">Gentoo FAQ</uri> for more information about
262up until the bootstrapping part. Then configure Portage to use distcc: 262bootstrapping.) Then configure Portage to use distcc:
263</p> 263</p>
264 264
265<pre caption="Preliminary Setup"> 265<pre caption="Preliminary Setup">
266# <i>nano -w /etc/make.conf</i> 266# <i>nano -w /etc/make.conf</i>
267<comment>(Add distcc to the FEATURES variable)</comment> 267<comment>(Add distcc to the FEATURES</comment>
268FEATURES="distcc" 268FEATURES="distcc"
269<comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment> 269<comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
270+1 available)</comment>
270MAKEOPTS="-jN" 271MAKEOPTS="-jN"
271</pre> 272</pre>
272 273
273<pre caption="Setting your path"> 274<pre caption="Setting your path">
274# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i> 275# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
279<section> 280<section>
280<title>Step 2: Getting Distcc</title> 281<title>Step 2: Getting Distcc</title>
281<body> 282<body>
282 283
283<p> 284<p>
284Before the installation of distcc, a user called distcc must be added to the
285<path>/etc/passwd</path>:
286</p>
287
288<pre caption="Create user distcc">
289# <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
290</pre>
291
292<impo>
293It is important to note that adding users like this is very bad. We only do
294it here because there is no <c>useradd</c> utility (which you normally use for
295adding users) yet at this stage of installation.
296</impo>
297
298<p>
299Install distcc: 285Install distcc:
300</p> 286</p>
301 287
302<pre caption="Getting Distcc on the new box"> 288<pre caption="Getting Distcc on the new box">
303# <i>USE='-*' emerge --nodeps sys-devel/distcc</i> 289# <i>USE='-*' emerge --nodeps sys-devel/distcc sys-devel/distcc-config</i>
304</pre> 290</pre>
305 291
306</body> 292</body>
307</section> 293</section>
308<section> 294<section>
312<p> 298<p>
313Run <c>distcc-config --install</c> to setup distcc: 299Run <c>distcc-config --install</c> to setup distcc:
314</p> 300</p>
315 301
316<pre caption="Final distcc setup"> 302<pre caption="Final distcc setup">
317<comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment> 303<comment>(Substitute host1, host2, ... with the IP number(s) of the
304participating hosts)</comment>
318# <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i> 305# <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
319<comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment> 306<comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
307192.168.0.4 192.168.0.6"</i></comment>
320</pre> 308</pre>
321 309
322<p> 310<p>
323Distcc is now set up to bootstrap! Continue with the official installation 311Distcc is now set up to bootstrap! Continue with the official installation
324instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge 312instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
325system</c>. This is to make sure that all of the dependencies you want are 313system</c>. This is to make sure that all of the dependencies you want are
337</chapter> 325</chapter>
338 326
339<chapter> 327<chapter>
340<title>Troubleshooting</title> 328<title>Troubleshooting</title>
341<section> 329<section>
342<title>Mozilla and Xfree</title> 330<title>Some Packages Don't Use Distcc</title>
343<body> 331<body>
344 332
345<p> 333<p>
346As you emerge various packages, you'll notice that some of them aren't being 334As you emerge various packages, you'll notice that some of them aren't being
347distributed (and aren't being built in parallel). This is because the 335distributed (and aren't being built in parallel). This may happen because the
348developers of the Mozilla and Xfree ebuilds intentionally disable parallel 336package's Makefile doesn't support parallel operations or the maintainer of the
349building because it is known to cause problems. 337ebuild has explicitly disabled parallel operations due to a known problem.
350</p> 338</p>
351 339
352<p> 340<p>
353Sometimes distcc might cause a package to fail to compile. If this happens 341Sometimes distcc might cause a package to fail to compile. If this happens
354for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us. 342for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
355</p>
356
357</body>
358</section>
359<section>
360<title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
361<body>
362
363<p>
364With such a long title any explanation here is almost irrelevent. However, if
365you plan to use distcc across hosts that have the <uri
366link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
367not, you will run into problems.
368</p>
369
370<p>
371The solution requires a little foresight on your part; you have to run
372<c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
373enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
374which are a good thing to do since for the most part the protections offered
375by both packages is a good thing and is transparent to the user.
376</p> 343</p>
377 344
378</body> 345</body>
379</section> 346</section>
380<section> 347<section>
389 356
390<p> 357<p>
391Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of 358Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
392<c>gcc</c>. This means that if you're mixing i686 machines with other types 359<c>gcc</c>. This means that if you're mixing i686 machines with other types
393(i386, i586) you will run into problems. A workaround for this may be to 360(i386, i586) you will run into problems. A workaround for this may be to
394<c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>, 361<c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
395but that needs testing. If it works for you please email me,
396<mail link="lisa@gentoo.org">Lisa Seelye</mail>.
397</p> 362</p>
363
364<impo>
365Doing this explicitly redefines some behaviour of Portage and may have some
366weird results in the future. Only do this if you're mixing CHOSTs.
367</impo>
368
398 369
399</body> 370</body>
400</section> 371</section>
401</chapter> 372</chapter>
402 373
413parameter it will run once. However, if you pass it a number it will update 384parameter it will run once. However, if you pass it a number it will update
414every N seconds, where N is the argument you passed. 385every N seconds, where N is the argument you passed.
415</p> 386</p>
416 387
417<p> 388<p>
418The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c> 389The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
419in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment 390in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
420and it is quite lovely. 391and it is quite lovely. For Gentoo the GUI monitor has been called
392<c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
393<c>distccmon-gnome</c>.
421</p> 394</p>
422 395
423<pre caption="Starting the monitors"> 396<pre caption="Starting the monitors">
424# <i>distccmon-text N</i> 397# <i>distccmon-text N</i>
425<comment>(Or)</comment> 398<comment>(Or)</comment>
426# <i>distccmon-gnome</i> 399# <i>distccmon-gui</i>
427<comment>To monitor Portage's distcc usage you can use:</comment> 400<comment>To monitor Portage's distcc usage you can use:</comment>
428# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i> 401# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
429<comment>(Or)</comment> 402<comment>(Or)</comment>
430# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gnome</i> 403# <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
431</pre> 404</pre>
432 405
433<impo> 406<impo>
434If your distcc directory is elsewhere, change the DISTCC_DIR variable 407If your distcc directory is elsewhere, change the DISTCC_DIR variable
435accordingly. 408accordingly.

Legend:
Removed from v.1.27  
changed lines
  Added in v.1.40

  ViewVC Help
Powered by ViewVC 1.1.20