/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.20 - (show annotations) (download) (as text)
Sun Jul 11 13:13:49 2004 UTC (10 years, 1 month ago) by pylon
Branch: MAIN
Changes since 1.19: +15 -6 lines
File MIME type: application/xml
The user distcc must be created before installing distcc for bootstrap usage.

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.19 2004/07/02 15:47:15 swift Exp $ -->
4
5 <!--
6 Synchronised with rev 1.15 on
7 http://www.thedoh.com/viewcvs/distcc/html/distcc.xml
8 -->
9
10 <guide link="/doc/en/distcc.xml">
11
12 <title>Gentoo Distcc Documentation</title>
13
14 <author title="Author">
15 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
16 </author>
17
18 <author title="Editor">
19 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
20 </author>
21
22 <author title="Editor">
23 <mail link="erwin@gentoo.org">Erwin</mail>
24 </author>
25
26 <author title="Editor">
27 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
28 </author>
29
30 <author title="Editor">
31 <mail link="pylon@gentoo.org">Lars Weiler</mail>
32 </author>
33
34 <author title="Reviewer">
35 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
36 </author>
37
38 <abstract>
39 This document serves as a HOWTO for using distcc with Gentoo.
40 </abstract>
41
42 <!-- The content of this document is licensed under the CC-BY-SA license -->
43 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
44 <license/>
45
46 <version>1.7.1</version>
47 <date>July 11, 2004</date>
48
49 <chapter>
50 <title>Introduction</title>
51 <section>
52 <title>What is distcc?</title>
53 <body>
54
55 <p>
56 Distcc is a program designed to distribute compiling tasks across a network to
57 participating hosts. It is comprised of a server, <c>distccd</c> and a client
58 program, <c>distcc</c>. Distcc can work transparently with <uri
59 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
60 </p>
61
62 </body>
63 </section>
64 <section>
65 <title>Using distcc to bootstrap</title>
66 <body>
67
68 <p>
69 If you are planning on using distcc to help you bootstrap a Gentoo installation,
70 make sure you read the part on <uri link="#bootstrapping">Using distcc to
71 Bootstrap</uri> which is situated further down in this document.
72 </p>
73
74 </body>
75 </section>
76 </chapter>
77
78 <chapter>
79 <title>Setup</title>
80 <section>
81 <title>Dependencies</title>
82 <body>
83
84 <p>
85 To start off with, to use Distcc all of the computers on your network need to
86 have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
87 mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
88 </p>
89
90 </body>
91 </section>
92 <section>
93 <title>Installing Distcc</title>
94 <body>
95
96 <p>
97 There are a couple of options you should know about before you start
98 installing distcc.
99 </p>
100
101 <p>
102 Distcc ships with a graphical monitor to monitor tasks that your computer is
103 sending away for compilation. If you use Gnome then put 'gnome' in your USE
104 flags. However, if you don't use Gnome and would still like to have the
105 monitor then you should put 'gtk' in your USE flags.
106 </p>
107
108 <pre caption="Installing distcc">
109 # <i>emerge distcc</i>
110 </pre>
111
112 </body>
113 </section>
114 <section>
115 <title>Setting up Portage to use Distcc</title>
116 <body>
117
118 <p>
119 Setting up distcc is easy to do with Portage. Execute the following steps on
120 each system that should participate in the distributed compiling:
121 </p>
122
123 <pre caption="Integrating Distcc and Portage">
124 # <i>emerge distcc</i>
125 # <i>nano -w /etc/make.conf</i>
126 <comment>(Set N to a suitable number for your particular setup)</comment>
127 <comment>(A common heuristic strategy is twice the number of CPUs + 1)</comment>
128 MAKEOPTS="-jN"
129 <comment>(Add distcc to your FEATURES variable)</comment>
130 FEATURES="distcc"
131 </pre>
132
133 <p>
134 Next you have to specify what hosts you want to use. Use the
135 <c>distcc-config</c> command to set the list of hosts. Here is an example of
136 some hosts that might be in your list:
137 </p>
138
139 <pre caption="Examples of host definitions">
140 192.168.0.1 192.168.0.2 192.168.0.3
141 192.168.0.1/2 192.168.0.2 192.168.0.3/10
142 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
143 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
144 <comment>(There are also several other methods of setting up hosts. See the
145 distcc manpage for more details.)</comment>
146 </pre>
147
148 <p>
149 It may all look complicated, but in most cases a variant of line 1 or 2 will
150 work.
151 </p>
152
153 <ul>
154 <li>Line 1 is just a space-delimited list of hosts</li>
155 <li>
156 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
157 defining the <c>/N</c> setting) to send that host at any given time
158 </li>
159 </ul>
160
161 <p>
162 Since most people won't be using lines 3 or 4, I'll <uri
163 link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
164 docs for more information.
165 </p>
166
167 <p>
168 For instance, to set the first line in the previous example:
169 </p>
170
171 <pre caption="Sample command to set the hosts">
172 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
173 </pre>
174
175 <p>
176 Now start the distcc daemon on all the participating computers:
177 </p>
178
179 <pre caption="Starting the distcc daemon">
180 <comment>(Add distccd to the default runlevel)</comment>
181 # <i>rc-update add distccd default</i>
182 <comment>(Start the distcc daemon)</comment>
183 # <i>/etc/init.d/distccd start</i>
184 </pre>
185
186 </body>
187 </section>
188 <section>
189 <title>Setting up Distcc to Work With Automake</title>
190 <body>
191
192 <p>
193 This is in some cases easier than the Portage setup. What you have to do is
194 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
195 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
196 However, there is a caveat. If you use ccache you have to put distcc after
197 the ccache part:
198 </p>
199
200 <pre caption="Setting your path">
201 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
202 </pre>
203
204 <p>
205 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
206 (where N is an integer). The value of N depends on your network and the types
207 of computers you are using to compile. For me, I have a dual processor P3 and
208 a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
209 Thunderbird) and I use -j5. Test your own settings to find the most
210 performant number.
211 </p>
212
213 </body>
214 </section>
215 </chapter>
216
217 <chapter>
218 <title>Cross-Compiling</title>
219 <section>
220 <title>A Note on Cross-Compiling</title>
221 <body>
222
223 <p>
224 Cross-compiling is using one architecture to build programs for another
225 architecture. This can be as simple as using an Athlon (i686) to build a
226 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
227 </p>
228
229 </body>
230 </section>
231 <section>
232 <title>A Personal Note on Cross-compiling</title>
233 <body>
234
235 <p>
236 I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
237 don't have any non-x86 machines. I can create things that work <e>in
238 theory</e>, but I have to rely on other people to test what I write.
239 That works to an extent, but it's tough. ;-)
240 </p>
241
242 <p>
243 In the near future I hope to acquire a Sparc of some type to put Gentoo on so
244 I can play around in-house.
245 </p>
246
247 </body>
248 </section>
249 </chapter>
250
251 <chapter id="bootstrapping">
252 <title>Using Distcc to Bootstrap</title>
253 <section>
254 <title>Step 1: Configure Portage </title>
255 <body>
256
257 <p>
258 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
259 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
260 up until the bootstrapping part. Then configure Portage to use distcc:
261 </p>
262
263 <pre caption="Preliminary Setup">
264 # <i>nano -w /etc/make.conf</i>
265 <comment>(Add distcc to the FEATURES variable)</comment>
266 FEATURES="distcc"
267 <comment>(Set distcc's directory for temporary space)</comment>
268 DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
269 <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
270 MAKEOPTS="-jN"
271 </pre>
272
273 </body>
274 </section>
275 <section>
276 <title>Step 2: Getting Distcc</title>
277 <body>
278
279 <p>Before the installation of distcc, a user called distcc must be added to the
280 <path>/etc/passwd</path>:</p>
281
282 <pre caption="Create user distcc">
283 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
284 </pre>
285
286 <p>
287 Install distcc:
288 </p>
289
290 <pre caption="Getting Distcc on the new box">
291 # <i>USE='-*' emerge --nodeps distcc</i>
292 </pre>
293
294 </body>
295 </section>
296 <section>
297 <title>Step 3: Setting Up Distcc</title>
298 <body>
299
300 <p>
301 Run <c>distcc-config</c> to setup distcc:
302 </p>
303
304 <pre caption="Final distcc setup">
305 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
306 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
307 </pre>
308
309 <p>
310 distcc is now set up to bootstrap! Continue the official installation
311 instructions and <e>don't forget</e> to re-emerge distcc after <c>emerge
312 system</c>. This is to make sure that all of the dependencies you want are
313 installed as well.
314 </p>
315
316 <note>
317 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
318 This is expected as some ebuilds do not work well with distcc, so they
319 intentionally disable it.
320 </note>
321
322 </body>
323 </section>
324 </chapter>
325
326 <chapter>
327 <title>Troubleshooting</title>
328 <section>
329 <title>Mozilla and Xfree</title>
330 <body>
331
332 <p>
333 As you emerge various packages, you'll notice that some of them aren't being
334 distributed (and aren't being built in parallel). This is because the
335 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
336 building because it is known to cause problems.
337 </p>
338
339 <p>
340 Sometimes distcc might cause a package to fail to compile. If this happens
341 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
342 </p>
343
344 </body>
345 </section>
346 <section>
347 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
348 <body>
349
350 <p>
351 With such a long title any explanation here is almost irrelevent. However, if
352 you plan to use distcc across hosts that have the <uri
353 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
354 not, you will run into problems.
355 </p>
356
357 <p>
358 The solution requires a little foresight on your part; you have to run
359 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
360 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
361 which are a good thing to do since for the most part the protections offered
362 by both packages is a good thing and is transparent to the user.
363 </p>
364
365 </body>
366 </section>
367 <section>
368 <title>Mixed GCC Versions</title>
369 <body>
370
371 <p>
372 If you have different GCC versions on your hosts, there will likely be very
373 weird problems. The solution is to make certain all hosts have the same GCC
374 version.
375 </p>
376
377 </body>
378 </section>
379 </chapter>
380
381 <chapter>
382 <title>Distcc Extras</title>
383 <section>
384 <title>Distcc Monitors</title>
385 <body>
386
387 <p>
388 Distcc ships with two monitors. The text-based one is always built and is
389 called <c>distccmon-text</c>. Running it for the first time can be a bit
390 confusing, but it is really quite easy to use. If you run the program with no
391 parameter it will run once. However, if you pass it a number it will update
392 every N seconds, where N is the argument you passed.
393 </p>
394
395 <p>
396 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
397 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
398 and it is quite lovely.
399 </p>
400
401 <pre caption="Starting the monitors">
402 # <i>distccmon-text N</i>
403 <comment>(Or ...)</comment>
404 # <i>distccmon-gnome</i>
405 </pre>
406
407 </body>
408 </section>
409 </chapter>
410
411 <chapter>
412 <title>Future Plans for Distcc and Gentoo</title>
413 <section>
414 <title>distcc-subnetscan</title>
415 <body>
416
417 <p>
418 <c>distcc-subnetscan</c> is a perl program in development that will scan a
419 subnet for hosts that have a participating distcc daemon. Even this will be
420 enhanced to test if the daemon on a remote host will conform to a specific
421 <c>CHOST</c> setting to make cross-compiling easier.
422 </p>
423
424 <p>
425 The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">lisa's
426 dev page</uri> until a more formal home can be found.
427 </p>
428
429 </body>
430 </section>
431 </chapter>
432 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20