/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.23 - (show annotations) (download) (as text)
Tue Nov 9 11:37:30 2004 UTC (10 years, 1 month ago) by swift
Branch: MAIN
Changes since 1.22: +10 -3 lines
File MIME type: application/xml
#67848 - Warn about security and --allow directive

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.22 2004/08/22 20:51:08 neysx Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12
13 <author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 </author>
16
17 <author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail>
19 </author>
20
21 <author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 </author>
24
25 <author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27 </author>
28
29 <author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 </author>
32
33 <abstract>
34 This document serves as a HOWTO for using distcc with Gentoo.
35 </abstract>
36
37 <!-- The content of this document is licensed under the CC-BY-SA license -->
38 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 <license/>
40
41 <version>1.9</version>
42 <date>November 09, 2004</date>
43
44 <chapter>
45 <title>Introduction</title>
46 <section>
47 <title>What is distcc?</title>
48 <body>
49
50 <p>
51 Distcc is a program designed to distribute compiling tasks across a network to
52 participating hosts. It is comprised of a server, <c>distccd</c> and a client
53 program, <c>distcc</c>. Distcc can work transparently with <uri
54 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
55 </p>
56
57 </body>
58 </section>
59 <section>
60 <title>Using distcc to bootstrap</title>
61 <body>
62
63 <p>
64 If you are planning on using distcc to help you bootstrap a Gentoo installation,
65 make sure you read the part on <uri link="#bootstrapping">Using distcc to
66 Bootstrap</uri> which is situated further down in this document.
67 </p>
68
69 </body>
70 </section>
71 </chapter>
72
73 <chapter>
74 <title>Setup</title>
75 <section>
76 <title>Dependencies</title>
77 <body>
78
79 <p>
80 To start off with, to use Distcc all of the computers on your network need to
81 have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
82 mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
83 </p>
84
85 </body>
86 </section>
87 <section>
88 <title>Installing Distcc</title>
89 <body>
90
91 <p>
92 There are a couple of options you should know about before you start
93 installing distcc.
94 </p>
95
96 <p>
97 Distcc ships with a graphical monitor to monitor tasks that your computer is
98 sending away for compilation. If you use Gnome then put 'gnome' in your USE
99 flags. However, if you don't use Gnome and would still like to have the
100 monitor then you should put 'gtk' in your USE flags.
101 </p>
102
103 <pre caption="Installing distcc">
104 # <i>emerge distcc</i>
105 </pre>
106
107 </body>
108 </section>
109 <section>
110 <title>Setting up Portage to use Distcc</title>
111 <body>
112
113 <p>
114 Setting up distcc is easy to do with Portage. Execute the following steps on
115 each system that should participate in the distributed compiling:
116 </p>
117
118 <pre caption="Integrating Distcc and Portage">
119 # <i>emerge distcc</i>
120 # <i>nano -w /etc/make.conf</i>
121 <comment>(Set N to a suitable number for your particular setup)</comment>
122 <comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment>
123 MAKEOPTS="-jN"
124 <comment>(Add distcc to your FEATURES variable)</comment>
125 FEATURES="distcc"
126 </pre>
127
128 <p>
129 Next you have to specify what hosts you want to use. Use the
130 <c>distcc-config</c> command to set the list of hosts. Here is an example of
131 some hosts that might be in your list:
132 </p>
133
134 <pre caption="Examples of host definitions">
135 192.168.0.1 192.168.0.2 192.168.0.3
136 192.168.0.1/2 192.168.0.2 192.168.0.3/10
137 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 <comment>(There are also several other methods of setting up hosts. See the
140 distcc manpage for more details.)</comment>
141 </pre>
142
143 <p>
144 It may all look complicated, but in most cases a variant of line 1 or 2 will
145 work.
146 </p>
147
148 <ul>
149 <li>Line 1 is just a space-delimited list of hosts</li>
150 <li>
151 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
152 defining the <c>/N</c> setting) to send that host at any given time
153 </li>
154 </ul>
155
156 <p>
157 Since most people won't be using lines 3 or 4, I'll <uri
158 link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
159 docs for more information.
160 </p>
161
162 <p>
163 For instance, to set the first line in the previous example:
164 </p>
165
166 <pre caption="Sample command to set the hosts">
167 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 </pre>
169
170 <p>
171 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
172 <c>--allow</c> directive to allow only hosts you trust. For more information
173 please see the <uri link="http://distcc.samba.org/security.html">Distcc Security
174 Design</uri>.
175 </p>
176
177 <p>
178 Now start the distcc daemon on all the participating computers:
179 </p>
180
181 <pre caption="Starting the distcc daemon">
182 <comment>(Add distccd to the default runlevel)</comment>
183 # <i>rc-update add distccd default</i>
184 <comment>(Start the distcc daemon)</comment>
185 # <i>/etc/init.d/distccd start</i>
186 </pre>
187
188 </body>
189 </section>
190 <section>
191 <title>Setting up Distcc to Work With Automake</title>
192 <body>
193
194 <p>
195 This is in some cases easier than the Portage setup. What you have to do is
196 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
197 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
198 However, there is a caveat. If you use ccache you have to put distcc after
199 the ccache part:
200 </p>
201
202 <pre caption="Setting your path">
203 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
204 </pre>
205
206 <p>
207 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
208 (where N is an integer). The value of N depends on your network and the types
209 of computers you are using to compile. For me, I have a dual processor P3 and
210 a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
211 Thunderbird) and I use -j5. Test your own settings to find the number that
212 yields the best performance.
213 </p>
214
215 </body>
216 </section>
217 </chapter>
218
219 <chapter>
220 <title>Cross-Compiling</title>
221 <section>
222 <title>A Note on Cross-Compiling</title>
223 <body>
224
225 <p>
226 Cross-compiling is using one architecture to build programs for another
227 architecture. This can be as simple as using an Athlon (i686) to build a
228 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
229 </p>
230
231 </body>
232 </section>
233 <section>
234 <title>An Introduction to Cross-compiling</title>
235 <body>
236
237 <p>
238 If you want to give Cross-compiling a try you can emerge the crossdev package.
239 I can't support it since I didn't write it, but that's a good way to start
240 cross-compiling.
241 </p>
242
243 </body>
244 </section>
245 </chapter>
246
247 <chapter id="bootstrapping">
248 <title>Using Distcc to Bootstrap</title>
249 <section>
250 <title>Step 1: Configure Portage </title>
251 <body>
252
253 <p>
254 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
255 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
256 up until the bootstrapping part. Then configure Portage to use distcc:
257 </p>
258
259 <pre caption="Preliminary Setup">
260 # <i>nano -w /etc/make.conf</i>
261 <comment>(Add distcc to the FEATURES variable)</comment>
262 FEATURES="distcc"
263 <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
264 MAKEOPTS="-jN"
265 </pre>
266
267 <pre caption="Setting your path">
268 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
269 </pre>
270
271 </body>
272 </section>
273 <section>
274 <title>Step 2: Getting Distcc</title>
275 <body>
276
277 <p>
278 Before the installation of distcc, a user called distcc must be added to the
279 <path>/etc/passwd</path>:
280 </p>
281
282 <pre caption="Create user distcc">
283 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
284 </pre>
285
286 <p>
287 Install distcc:
288 </p>
289
290 <pre caption="Getting Distcc on the new box">
291 # <i>USE='-*' emerge --nodeps distcc</i>
292 </pre>
293
294 </body>
295 </section>
296 <section>
297 <title>Step 3: Setting Up Distcc</title>
298 <body>
299
300 <p>
301 Run <c>distcc-config</c> to setup distcc:
302 </p>
303
304 <pre caption="Final distcc setup">
305 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
306 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
307 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment>
308 </pre>
309
310 <p>
311 distcc is now set up to bootstrap! Continue the official installation
312 instructions and <e>don't forget</e> to re-emerge distcc after <c>emerge
313 system</c>. This is to make sure that all of the dependencies you want are
314 installed as well.
315 </p>
316
317 <note>
318 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
319 This is expected as some ebuilds do not work well with distcc, so they
320 intentionally disable it.
321 </note>
322
323 </body>
324 </section>
325 </chapter>
326
327 <chapter>
328 <title>Troubleshooting</title>
329 <section>
330 <title>Mozilla and Xfree</title>
331 <body>
332
333 <p>
334 As you emerge various packages, you'll notice that some of them aren't being
335 distributed (and aren't being built in parallel). This is because the
336 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
337 building because it is known to cause problems.
338 </p>
339
340 <p>
341 Sometimes distcc might cause a package to fail to compile. If this happens
342 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
343 </p>
344
345 </body>
346 </section>
347 <section>
348 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
349 <body>
350
351 <p>
352 With such a long title any explanation here is almost irrelevent. However, if
353 you plan to use distcc across hosts that have the <uri
354 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
355 not, you will run into problems.
356 </p>
357
358 <p>
359 The solution requires a little foresight on your part; you have to run
360 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
361 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
362 which are a good thing to do since for the most part the protections offered
363 by both packages is a good thing and is transparent to the user.
364 </p>
365
366 </body>
367 </section>
368 <section>
369 <title>Mixed GCC Versions</title>
370 <body>
371
372 <p>
373 If you have different GCC versions on your hosts, there will likely be very
374 weird problems. The solution is to make certain all hosts have the same GCC
375 version.
376 </p>
377
378 </body>
379 </section>
380 </chapter>
381
382 <chapter>
383 <title>Distcc Extras</title>
384 <section>
385 <title>Distcc Monitors</title>
386 <body>
387
388 <p>
389 Distcc ships with two monitors. The text-based one is always built and is
390 called <c>distccmon-text</c>. Running it for the first time can be a bit
391 confusing, but it is really quite easy to use. If you run the program with no
392 parameter it will run once. However, if you pass it a number it will update
393 every N seconds, where N is the argument you passed.
394 </p>
395
396 <p>
397 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
398 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
399 and it is quite lovely.
400 </p>
401
402 <pre caption="Starting the monitors">
403 # <i>distccmon-text N</i>
404 <comment>(Or)</comment>
405 # <i>distccmon-gnome</i>
406 <comment>To monitor Portage's distcc usage you can use:</comment>
407 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-text N</i>
408 <comment>(Or)</comment>
409 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-gnome</i>
410 </pre>
411
412 <impo>
413 Notice the backticks above. <c>portageq</c> is a part of Portage that will
414 return a piece of information (in this case what DISTCC_DIR is set to). For
415 simplicity, most users can use <c>DISTCC_DIR=/var/tmp/portage/.distcc
416 distccmon-text N</c>
417 </impo>
418
419 </body>
420 </section>
421 </chapter>
422 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20