/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.30 - (show annotations) (download) (as text)
Sun Jul 3 17:33:01 2005 UTC (8 years, 9 months ago) by alin
Branch: MAIN
Changes since 1.29: +2 -2 lines
File MIME type: application/xml
Fixed minor typo. No content change

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.29 2005/07/02 04:50:55 smithj Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12
13 <author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 </author>
16
17 <author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail>
19 </author>
20
21 <author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 </author>
24
25 <author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27 </author>
28
29 <author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 </author>
32
33 <abstract>
34 This document serves as a HOWTO for using distcc with Gentoo.
35 </abstract>
36
37 <!-- The content of this document is licensed under the CC-BY-SA license -->
38 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 <license/>
40
41 <version>1.12</version>
42 <date>2005-07-01</date>
43
44 <chapter>
45 <title>Introduction</title>
46 <section>
47 <title>What is distcc?</title>
48 <body>
49
50 <p>
51 Distcc is a program designed to distribute compiling tasks across a network to
52 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
53 program, <c>distcc</c>. Distcc can work transparently with <uri
54 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
55 little setup.
56 </p>
57
58 </body>
59 </section>
60 <section>
61 <title>Using distcc to bootstrap</title>
62 <body>
63
64 <p>
65 If you are planning on using distcc to help you bootstrap a Gentoo
66 installation, make sure you read the section <uri link="#bootstrapping">Using
67 distcc to Bootstrap</uri>, which is situated further down in this document.
68 </p>
69
70 </body>
71 </section>
72 </chapter>
73
74 <chapter>
75 <title>Setup</title>
76 <section>
77 <title>Dependencies</title>
78 <body>
79
80 <p>
81 In order to use Distcc, all of the computers on your network need to have the
82 same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
83 mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
84 errors.
85 </p>
86
87 </body>
88 </section>
89 <section>
90 <title>Installing Distcc</title>
91 <body>
92
93 <p>
94 There are a couple of options you should be aware of before you start
95 installing distcc.
96 </p>
97
98 <p>
99 Distcc ships with a graphical monitor to monitor tasks that your computer is
100 sending away for compilation. If you use Gnome then put 'gnome' in your USE
101 flags. However, if you don't use Gnome and would still like to have the
102 monitor then you should put 'gtk' in your USE flags.
103 </p>
104
105 <pre caption="Installing distcc">
106 # <i>emerge distcc</i>
107 </pre>
108
109 </body>
110 </section>
111 <section>
112 <title>Setting up Portage to use Distcc</title>
113 <body>
114
115 <p>
116 Setting up Portage to use distcc is easy. Execute the following steps on
117 each system that should participate in the distributed compiling:
118 </p>
119
120 <pre caption="Integrating Distcc and Portage">
121 # <i>emerge distcc</i>
122 # <i>nano -w /etc/make.conf</i>
123 <comment>(Set N to a suitable number for your particular setup)</comment>
124 <comment>(A common strategy is setting N as twice the number of total CPUs + 1
125 available)</comment>
126 MAKEOPTS="-jN"
127 <comment>(Add distcc to your FEATURES)</comment>
128 FEATURES="distcc"
129 </pre>
130
131 </body>
132 </section>
133 <section>
134 <title>Specifying Participating Hosts</title>
135 <body>
136
137 <p>
138 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
139 example of some hosts that might be in your list:
140 </p>
141
142 <pre caption="Examples of host definitions">
143 192.168.0.1 192.168.0.2 192.168.0.3
144 192.168.0.1/2 192.168.0.2 192.168.0.3/10
145 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
146 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
147 <comment>(There are also several other methods of setting up hosts. See the
148 distcc manpage for more details.)</comment>
149 <comment>If you wish to compile on the local machine you should put 'localhost'
150 in the hosts list. Conversely if you do not wish to use the local machine to
151 compile (which is often the case) omit it from the hosts list. On a slow
152 machine using localhost may actually slow things down. Make sure to test your
153 settings for performance.</comment>
154 </pre>
155
156 <p>
157 It may all look complicated, but in most cases a variant of line 1 or 2 will
158 work.
159 </p>
160
161 <p>
162 Since most people won't be using lines 3 or 4, I'll <uri
163 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
164 docs (man distcc) for more information.
165 </p>
166
167 <p>
168 For instance, to set the first line in the previous example:
169 </p>
170
171 <pre caption="Sample command to set the hosts">
172 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
173 </pre>
174
175 <p>
176 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
177 <c>--allow</c> directive to allow only hosts you trust. For added security,
178 you should also use the <c>--listen</c> directive to tell the distcc daemon
179 what IP to listen on (for multi-homed systems). More information on distcc
180 security can be found at <uri
181 link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
182 </p>
183
184 <impo>
185 It is important to use --allow and --listen. Please read the distccd manpage
186 or the above security document for more information.
187 </impo>
188
189
190 <p>
191 Now start the distcc daemon on all the participating computers:
192 </p>
193
194 <pre caption="Starting the distcc daemon">
195 <comment>(Add distccd to the default runlevel)</comment>
196 # <i>rc-update add distccd default</i>
197 <comment>(Start the distcc daemon)</comment>
198 # <i>/etc/init.d/distccd start</i>
199 </pre>
200
201 </body>
202 </section>
203 <section>
204 <title>Setting up Distcc to Work With Automake</title>
205 <body>
206
207 <p>
208 This is, in some cases, easier than the Portage setup. What you have to do is
209 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
210 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
211 However, there is a caveat. If you use ccache you have to put distcc after
212 the ccache part:
213 </p>
214
215 <pre caption="Setting your path">
216 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
217 <comment>You can put this in your .bashrc or equivelant file to have the PATH
218 set every time you log in</comment>
219 </pre>
220
221 <p>
222 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
223 (where N is an integer). The value of N depends on your network and the types
224 of computers you are using to compile. Test your own settings to find the
225 number that yields the best performance.
226 </p>
227
228 </body>
229 </section>
230 </chapter>
231
232 <chapter>
233 <title>Cross-Compiling</title>
234 <section>
235 <title>A Note on Cross-Compiling</title>
236 <body>
237
238 <p>
239 Cross-compiling is using one architecture to build programs for another
240 architecture. This can be as simple as using an Athlon (i686) to build a
241 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
242 </p>
243
244 </body>
245 </section>
246 <section>
247 <title>An Introduction to Cross-Compiling</title>
248 <body>
249
250 <p>
251 If you want to give cross-compiling a try you can to follow <uri
252 link="http://dev.gentoo.org/~vapier/CROSS-COMPILE-HOWTO">The Cross Compile
253 HOWTO</uri>; Crossdev is deprecated.
254 </p>
255
256 </body>
257 </section>
258 </chapter>
259
260 <chapter id="bootstrapping">
261 <title>Using Distcc to Bootstrap</title>
262 <section>
263 <title>Step 1: Configure Portage</title>
264 <body>
265
266 <p>
267 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
268 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
269 up until the bootstrapping part. Then configure Portage to use distcc:
270 </p>
271
272 <pre caption="Preliminary Setup">
273 # <i>nano -w /etc/make.conf</i>
274 <comment>(Add distcc to the FEATURES</comment>
275 FEATURES="distcc"
276 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
277 +1 available)</comment>
278 MAKEOPTS="-jN"
279 </pre>
280
281 <pre caption="Setting your path">
282 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
283 </pre>
284
285 </body>
286 </section>
287 <section>
288 <title>Step 2: Getting Distcc</title>
289 <body>
290
291 <p>
292 Before the installation of distcc, a user called distcc must be added to the
293 <path>/etc/passwd</path>:
294 </p>
295
296 <pre caption="Create user distcc">
297 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
298 </pre>
299
300 <impo>
301 It is important to note that adding users like this is very bad. We only do
302 it here because there is no <c>useradd</c> utility (which you normally use for
303 adding users) yet at this stage of installation.
304 </impo>
305
306 <p>
307 Install distcc:
308 </p>
309
310 <pre caption="Getting Distcc on the new box">
311 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
312 </pre>
313
314 </body>
315 </section>
316 <section>
317 <title>Step 3: Setting Up Distcc</title>
318 <body>
319
320 <p>
321 Run <c>distcc-config --install</c> to setup distcc:
322 </p>
323
324 <pre caption="Final distcc setup">
325 <comment>(Substitute host1, host2, ... with the IP number(s) of the
326 participating hosts)</comment>
327 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
328 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
329 192.168.0.4 192.168.0.6"</i></comment>
330 </pre>
331
332 <p>
333 Distcc is now set up to bootstrap! Continue with the official installation
334 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
335 system</c>. This is to make sure that all of the dependencies you want are
336 installed as well.
337 </p>
338
339 <note>
340 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
341 This is expected as some ebuilds do not work well with distcc, so they
342 intentionally disable it.
343 </note>
344
345 </body>
346 </section>
347 </chapter>
348
349 <chapter>
350 <title>Troubleshooting</title>
351 <section>
352 <title>Mozilla and Xfree</title>
353 <body>
354
355 <p>
356 As you emerge various packages, you'll notice that some of them aren't being
357 distributed (and aren't being built in parallel). This is because the
358 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
359 building because it is known to cause problems.
360 </p>
361
362 <p>
363 Sometimes distcc might cause a package to fail to compile. If this happens
364 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
365 </p>
366
367 </body>
368 </section>
369 <section>
370 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
371 <body>
372
373 <p>
374 With such a long title any explanation here is almost irrelevent. However, if
375 you plan to use distcc across hosts that have the <uri
376 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
377 not, you will run into problems.
378 </p>
379
380 <p>
381 The solution requires a little foresight on your part; you have to run
382 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
383 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
384 which are a good thing to do since for the most part the protections offered
385 by both packages is a good thing and is transparent to the user.
386 </p>
387
388 </body>
389 </section>
390 <section>
391 <title>Mixed GCC Versions</title>
392 <body>
393
394 <p>
395 If you have different GCC versions on your hosts, there will likely be very
396 weird problems. The solution is to make certain all hosts have the same GCC
397 version.
398 </p>
399
400 <p>
401 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
402 <c>gcc</c>. This means that if you're mixing i686 machines with other types
403 (i386, i586) you will run into problems. A workaround for this may be to
404 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
405 </p>
406
407 <impo>
408 Doing this explicitly redefines some behaviour of Portage and may have some
409 weird results in the future. Only do this if you're mixing CHOSTs.
410 </impo>
411
412
413 </body>
414 </section>
415 </chapter>
416
417 <chapter>
418 <title>Distcc Extras</title>
419 <section>
420 <title>Distcc Monitors</title>
421 <body>
422
423 <p>
424 Distcc ships with two monitors. The text-based one is always built and is
425 called <c>distccmon-text</c>. Running it for the first time can be a bit
426 confusing, but it is really quite easy to use. If you run the program with no
427 parameter it will run once. However, if you pass it a number it will update
428 every N seconds, where N is the argument you passed.
429 </p>
430
431 <p>
432 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
433 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
434 and it is quite lovely. For Gentoo the GUI monitor has been called
435 <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
436 <c>distccmon-gnome</c>.
437 </p>
438
439 <pre caption="Starting the monitors">
440 # <i>distccmon-text N</i>
441 <comment>(Or)</comment>
442 # <i>distccmon-gui</i>
443 <comment>To monitor Portage's distcc usage you can use:</comment>
444 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
445 <comment>(Or)</comment>
446 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
447 </pre>
448
449 <impo>
450 If your distcc directory is elsewhere, change the DISTCC_DIR variable
451 accordingly.
452 </impo>
453
454 </body>
455 </section>
456 </chapter>
457 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20