/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.32 - (show annotations) (download) (as text)
Wed Jan 25 09:16:29 2006 UTC (8 years, 6 months ago) by neysx
Branch: MAIN
Changes since 1.31: +8 -24 lines
File MIME type: application/xml
link to cross-compiling-distcc.xml

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.31 2006/01/01 11:51:43 neysx Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12 <author title="Editor">
13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 </author>
15 <author title="Editor">
16 <mail link="erwin@gentoo.org">Erwin</mail>
17 </author>
18 <author title="Editor">
19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 </author>
21 <author title="Editor">
22 <mail link="pylon@gentoo.org">Lars Weiler</mail>
23 </author>
24 <author title="Reviewer">
25 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
26 </author>
27
28 <abstract>
29 This document serves as a HOWTO for using distcc with Gentoo.
30 </abstract>
31
32 <!-- The content of this document is licensed under the CC-BY-SA license -->
33 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
34 <license/>
35
36 <version>1.13</version>
37 <date>2006-01-25</date>
38
39 <chapter>
40 <title>Introduction</title>
41 <section>
42 <title>What is distcc?</title>
43 <body>
44
45 <p>
46 Distcc is a program designed to distribute compiling tasks across a network to
47 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
48 program, <c>distcc</c>. Distcc can work transparently with <uri
49 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
50 little setup.
51 </p>
52
53 </body>
54 </section>
55 <section>
56 <title>Using distcc to bootstrap</title>
57 <body>
58
59 <p>
60 If you are planning on using distcc to help you bootstrap a Gentoo
61 installation, make sure you read the section <uri link="#bootstrapping">Using
62 distcc to Bootstrap</uri>, which is situated further down in this document.
63 </p>
64
65 </body>
66 </section>
67 </chapter>
68
69 <chapter>
70 <title>Setup</title>
71 <section>
72 <title>Dependencies</title>
73 <body>
74
75 <p>
76 In order to use Distcc, all of the computers on your network need to have the
77 same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
78 mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
79 errors.
80 </p>
81
82 </body>
83 </section>
84 <section>
85 <title>Installing Distcc</title>
86 <body>
87
88 <p>
89 There are a couple of options you should be aware of before you start
90 installing distcc.
91 </p>
92
93 <p>
94 Distcc ships with a graphical monitor to monitor tasks that your computer is
95 sending away for compilation. If you use Gnome then put 'gnome' in your USE
96 flags. However, if you don't use Gnome and would still like to have the
97 monitor then you should put 'gtk' in your USE flags.
98 </p>
99
100 <pre caption="Installing distcc">
101 # <i>emerge distcc</i>
102 </pre>
103
104 </body>
105 </section>
106 <section>
107 <title>Setting up Portage to use Distcc</title>
108 <body>
109
110 <p>
111 Setting up Portage to use distcc is easy. Execute the following steps on
112 each system that should participate in the distributed compiling:
113 </p>
114
115 <pre caption="Integrating Distcc and Portage">
116 # <i>emerge distcc</i>
117 # <i>nano -w /etc/make.conf</i>
118 <comment>(Set N to a suitable number for your particular setup)</comment>
119 <comment>(A common strategy is setting N as twice the number of total CPUs + 1
120 available)</comment>
121 MAKEOPTS="-jN"
122 <comment>(Add distcc to your FEATURES)</comment>
123 FEATURES="distcc"
124 </pre>
125
126 </body>
127 </section>
128 <section>
129 <title>Specifying Participating Hosts</title>
130 <body>
131
132 <p>
133 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
134 example of some hosts that might be in your list:
135 </p>
136
137 <pre caption="Examples of host definitions">
138 192.168.0.1 192.168.0.2 192.168.0.3
139 192.168.0.1/2 192.168.0.2 192.168.0.3/10
140 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
141 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
142 <comment>(There are also several other methods of setting up hosts. See the
143 distcc manpage for more details.)</comment>
144 <comment>If you wish to compile on the local machine you should put 'localhost'
145 in the hosts list. Conversely if you do not wish to use the local machine to
146 compile (which is often the case) omit it from the hosts list. On a slow
147 machine using localhost may actually slow things down. Make sure to test your
148 settings for performance.</comment>
149 </pre>
150
151 <p>
152 It may all look complicated, but in most cases a variant of line 1 or 2 will
153 work.
154 </p>
155
156 <p>
157 Since most people won't be using lines 3 or 4, I'll <uri
158 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
159 docs (man distcc) for more information.
160 </p>
161
162 <p>
163 For instance, to set the first line in the previous example:
164 </p>
165
166 <pre caption="Sample command to set the hosts">
167 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 </pre>
169
170 <p>
171 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
172 <c>--allow</c> directive to allow only hosts you trust. For added security,
173 you should also use the <c>--listen</c> directive to tell the distcc daemon
174 what IP to listen on (for multi-homed systems). More information on distcc
175 security can be found at <uri
176 link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
177 </p>
178
179 <impo>
180 It is important to use --allow and --listen. Please read the distccd manpage
181 or the above security document for more information.
182 </impo>
183
184
185 <p>
186 Now start the distcc daemon on all the participating computers:
187 </p>
188
189 <pre caption="Starting the distcc daemon">
190 <comment>(Add distccd to the default runlevel)</comment>
191 # <i>rc-update add distccd default</i>
192 <comment>(Start the distcc daemon)</comment>
193 # <i>/etc/init.d/distccd start</i>
194 </pre>
195
196 </body>
197 </section>
198 <section>
199 <title>Setting up Distcc to Work With Automake</title>
200 <body>
201
202 <p>
203 This is, in some cases, easier than the Portage setup. What you have to do is
204 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
205 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
206 However, there is a caveat. If you use ccache you have to put distcc after
207 the ccache part:
208 </p>
209
210 <pre caption="Setting your path">
211 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
212 <comment>You can put this in your .bashrc or equivelant file to have the PATH
213 set every time you log in</comment>
214 </pre>
215
216 <p>
217 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
218 (where N is an integer). The value of N depends on your network and the types
219 of computers you are using to compile. Test your own settings to find the
220 number that yields the best performance.
221 </p>
222
223 </body>
224 </section>
225 </chapter>
226
227 <chapter>
228 <title>Cross-Compiling</title>
229 <section>
230 <body>
231
232 <p>
233 Cross-compiling is using one architecture to build programs for another
234 architecture. This can be as simple as using an Athlon (i686) to build a
235 program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
236 is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
237 Cross-compiling Guide</uri>.
238 </p>
239
240 </body>
241 </section>
242 </chapter>
243
244 <chapter id="bootstrapping">
245 <title>Using Distcc to Bootstrap</title>
246 <section>
247 <title>Step 1: Configure Portage</title>
248 <body>
249
250 <p>
251 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
252 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
253 up until the bootstrapping part. Then configure Portage to use distcc:
254 </p>
255
256 <pre caption="Preliminary Setup">
257 # <i>nano -w /etc/make.conf</i>
258 <comment>(Add distcc to the FEATURES</comment>
259 FEATURES="distcc"
260 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
261 +1 available)</comment>
262 MAKEOPTS="-jN"
263 </pre>
264
265 <pre caption="Setting your path">
266 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
267 </pre>
268
269 </body>
270 </section>
271 <section>
272 <title>Step 2: Getting Distcc</title>
273 <body>
274
275 <p>
276 Before the installation of distcc, a user called distcc must be added to the
277 <path>/etc/passwd</path>:
278 </p>
279
280 <pre caption="Create user distcc">
281 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
282 </pre>
283
284 <impo>
285 It is important to note that adding users like this is very bad. We only do
286 it here because there is no <c>useradd</c> utility (which you normally use for
287 adding users) yet at this stage of installation.
288 </impo>
289
290 <p>
291 Install distcc:
292 </p>
293
294 <pre caption="Getting Distcc on the new box">
295 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
296 </pre>
297
298 </body>
299 </section>
300 <section>
301 <title>Step 3: Setting Up Distcc</title>
302 <body>
303
304 <p>
305 Run <c>distcc-config --install</c> to setup distcc:
306 </p>
307
308 <pre caption="Final distcc setup">
309 <comment>(Substitute host1, host2, ... with the IP number(s) of the
310 participating hosts)</comment>
311 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
312 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
313 192.168.0.4 192.168.0.6"</i></comment>
314 </pre>
315
316 <p>
317 Distcc is now set up to bootstrap! Continue with the official installation
318 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
319 system</c>. This is to make sure that all of the dependencies you want are
320 installed as well.
321 </p>
322
323 <note>
324 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
325 This is expected as some ebuilds do not work well with distcc, so they
326 intentionally disable it.
327 </note>
328
329 </body>
330 </section>
331 </chapter>
332
333 <chapter>
334 <title>Troubleshooting</title>
335 <section>
336 <title>Mozilla and Xfree</title>
337 <body>
338
339 <p>
340 As you emerge various packages, you'll notice that some of them aren't being
341 distributed (and aren't being built in parallel). This is because the
342 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
343 building because it is known to cause problems.
344 </p>
345
346 <p>
347 Sometimes distcc might cause a package to fail to compile. If this happens
348 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
349 </p>
350
351 </body>
352 </section>
353 <section>
354 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
355 <body>
356
357 <p>
358 With such a long title any explanation here is almost irrelevent. However, if
359 you plan to use distcc across hosts that have the <uri
360 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
361 not, you will run into problems.
362 </p>
363
364 <p>
365 The solution requires a little foresight on your part; you have to run
366 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
367 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
368 which are a good thing to do since for the most part the protections offered
369 by both packages is a good thing and is transparent to the user.
370 </p>
371
372 </body>
373 </section>
374 <section>
375 <title>Mixed GCC Versions</title>
376 <body>
377
378 <p>
379 If you have different GCC versions on your hosts, there will likely be very
380 weird problems. The solution is to make certain all hosts have the same GCC
381 version.
382 </p>
383
384 <p>
385 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
386 <c>gcc</c>. This means that if you're mixing i686 machines with other types
387 (i386, i586) you will run into problems. A workaround for this may be to
388 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
389 </p>
390
391 <impo>
392 Doing this explicitly redefines some behaviour of Portage and may have some
393 weird results in the future. Only do this if you're mixing CHOSTs.
394 </impo>
395
396
397 </body>
398 </section>
399 </chapter>
400
401 <chapter>
402 <title>Distcc Extras</title>
403 <section>
404 <title>Distcc Monitors</title>
405 <body>
406
407 <p>
408 Distcc ships with two monitors. The text-based one is always built and is
409 called <c>distccmon-text</c>. Running it for the first time can be a bit
410 confusing, but it is really quite easy to use. If you run the program with no
411 parameter it will run once. However, if you pass it a number it will update
412 every N seconds, where N is the argument you passed.
413 </p>
414
415 <p>
416 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
417 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
418 and it is quite lovely. For Gentoo the GUI monitor has been called
419 <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
420 <c>distccmon-gnome</c>.
421 </p>
422
423 <pre caption="Starting the monitors">
424 # <i>distccmon-text N</i>
425 <comment>(Or)</comment>
426 # <i>distccmon-gui</i>
427 <comment>To monitor Portage's distcc usage you can use:</comment>
428 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
429 <comment>(Or)</comment>
430 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
431 </pre>
432
433 <impo>
434 If your distcc directory is elsewhere, change the DISTCC_DIR variable
435 accordingly.
436 </impo>
437
438 </body>
439 </section>
440 </chapter>
441 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20