/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.27 - (show annotations) (download) (as text)
Fri Jul 1 22:35:19 2005 UTC (9 years, 2 months ago) by smithj
Branch: MAIN
Changes since 1.26: +3 -3 lines
File MIME type: application/xml
#42712: a few minor english fixes

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.26 2005/03/25 14:43:23 swift Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12
13 <author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 </author>
16
17 <author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail>
19 </author>
20
21 <author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 </author>
24
25 <author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27 </author>
28
29 <author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 </author>
32
33 <abstract>
34 This document serves as a HOWTO for using distcc with Gentoo.
35 </abstract>
36
37 <!-- The content of this document is licensed under the CC-BY-SA license -->
38 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 <license/>
40
41 <version>1.11</version>
42 <date>2005-03-25</date>
43
44 <chapter>
45 <title>Introduction</title>
46 <section>
47 <title>What is distcc?</title>
48 <body>
49
50 <p>
51 Distcc is a program designed to distribute compiling tasks across a network to
52 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
53 program, <c>distcc</c>. Distcc can work transparently with <uri
54 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
55 </p>
56
57 </body>
58 </section>
59 <section>
60 <title>Using distcc to bootstrap</title>
61 <body>
62
63 <p>
64 If you are planning on using distcc to help you bootstrap a Gentoo installation,
65 make sure you read the section <uri link="#bootstrapping">Using distcc to
66 Bootstrap</uri>, which is situated further down in this document.
67 </p>
68
69 </body>
70 </section>
71 </chapter>
72
73 <chapter>
74 <title>Setup</title>
75 <section>
76 <title>Dependencies</title>
77 <body>
78
79 <p>
80 In order to use Distcc, all of the computers on your network need to
81 have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
82 mixing 3.3.x and 3.2.x <b>will</b> result in your programs having errors in them.
83 </p>
84
85 </body>
86 </section>
87 <section>
88 <title>Installing Distcc</title>
89 <body>
90
91 <p>
92 There are a couple of options you should be aware of before you start
93 installing distcc.
94 </p>
95
96 <p>
97 Distcc ships with a graphical monitor to monitor tasks that your computer is
98 sending away for compilation. If you use Gnome then put 'gnome' in your USE
99 flags. However, if you don't use Gnome and would still like to have the
100 monitor then you should put 'gtk' in your USE flags.
101 </p>
102
103 <pre caption="Installing distcc">
104 # <i>emerge distcc</i>
105 </pre>
106
107 </body>
108 </section>
109 <section>
110 <title>Setting up Portage to use Distcc</title>
111 <body>
112
113 <p>
114 Setting up Portage to use distcc is easy. Execute the following steps on
115 each system that should participate in the distributed compiling:
116 </p>
117
118 <pre caption="Integrating Distcc and Portage">
119 # <i>emerge distcc</i>
120 # <i>nano -w /etc/make.conf</i>
121 <comment>(Set N to a suitable number for your particular setup)</comment>
122 <comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment>
123 MAKEOPTS="-jN"
124 <comment>(Add distcc to your FEATURES variable)</comment>
125 FEATURES="distcc"
126 </pre>
127
128 </body>
129 </section>
130 <section>
131 <title>Specifying Participating Hosts</title>
132 <body>
133
134 <p>
135 Use the <c>distcc-config</c> command to set the list of hosts. Here is an example of
136 some hosts that might be in your list:
137 </p>
138
139 <pre caption="Examples of host definitions">
140 192.168.0.1 192.168.0.2 192.168.0.3
141 192.168.0.1/2 192.168.0.2 192.168.0.3/10
142 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
143 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
144 <comment>(There are also several other methods of setting up hosts. See the
145 distcc manpage for more details.)</comment>
146 </pre>
147
148 <p>
149 It may all look complicated, but in most cases a variant of line 1 or 2 will
150 work.
151 </p>
152
153 <ul>
154 <li>Line 1 is just a space-delimited list of hosts</li>
155 <li>
156 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
157 defining the <c>/N</c> setting) to send that host at any given time
158 </li>
159 </ul>
160
161 <p>
162 Since most people won't be using lines 3 or 4, I'll <uri
163 link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
164 docs for more information.
165 </p>
166
167 <p>
168 For instance, to set the first line in the previous example:
169 </p>
170
171 <pre caption="Sample command to set the hosts">
172 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
173 </pre>
174
175 <p>
176 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
177 <c>--allow</c> directive to allow only hosts you trust. For added security, you should also use the <c>--listen</c>
178 directive to tell the distcc daemon what IP to listen on (for multi-homed
179 systems). More information on distcc security can be found at <uri link="http://distcc.samba.org/security.html">Distcc Security
180 Design</uri>.
181 </p>
182
183 <p>
184 Now start the distcc daemon on all the participating computers:
185 </p>
186
187 <pre caption="Starting the distcc daemon">
188 <comment>(Add distccd to the default runlevel)</comment>
189 # <i>rc-update add distccd default</i>
190 <comment>(Start the distcc daemon)</comment>
191 # <i>/etc/init.d/distccd start</i>
192 </pre>
193
194 </body>
195 </section>
196 <section>
197 <title>Setting up Distcc to Work With Automake</title>
198 <body>
199
200 <p>
201 This is, in some cases, easier than the Portage setup. What you have to do is
202 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
203 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
204 However, there is a caveat. If you use ccache you have to put distcc after
205 the ccache part:
206 </p>
207
208 <pre caption="Setting your path">
209 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
210 </pre>
211
212 <p>
213 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
214 (where N is an integer). The value of N depends on your network and the types
215 of computers you are using to compile. For me, I have a dual processor P3 and
216 a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
217 Thunderbird) and I use -j5. Test your own settings to find the number that
218 yields the best performance.
219 </p>
220
221 </body>
222 </section>
223 </chapter>
224
225 <chapter>
226 <title>Cross-Compiling</title>
227 <section>
228 <title>A Note on Cross-Compiling</title>
229 <body>
230
231 <p>
232 Cross-compiling is using one architecture to build programs for another
233 architecture. This can be as simple as using an Athlon (i686) to build a
234 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
235 </p>
236
237 </body>
238 </section>
239 <section>
240 <title>An Introduction to Cross-Compiling</title>
241 <body>
242
243 <p>
244 If you want to give cross-compiling a try you can to follow <uri
245 link="http://dev.gentoo.org/~vapier/CROSS-COMPILE-HOWTO">The Cross Compile
246 HOWTO</uri>; Crossdev is deprecated.
247 </p>
248
249 </body>
250 </section>
251 </chapter>
252
253 <chapter id="bootstrapping">
254 <title>Using Distcc to Bootstrap</title>
255 <section>
256 <title>Step 1: Configure Portage</title>
257 <body>
258
259 <p>
260 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
261 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
262 up until the bootstrapping part. Then configure Portage to use distcc:
263 </p>
264
265 <pre caption="Preliminary Setup">
266 # <i>nano -w /etc/make.conf</i>
267 <comment>(Add distcc to the FEATURES variable)</comment>
268 FEATURES="distcc"
269 <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
270 MAKEOPTS="-jN"
271 </pre>
272
273 <pre caption="Setting your path">
274 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
275 </pre>
276
277 </body>
278 </section>
279 <section>
280 <title>Step 2: Getting Distcc</title>
281 <body>
282
283 <p>
284 Before the installation of distcc, a user called distcc must be added to the
285 <path>/etc/passwd</path>:
286 </p>
287
288 <pre caption="Create user distcc">
289 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
290 </pre>
291
292 <impo>
293 It is important to note that adding users like this is very bad. We only do
294 it here because there is no <c>useradd</c> utility (which you normally use for
295 adding users) yet at this stage of installation.
296 </impo>
297
298 <p>
299 Install distcc:
300 </p>
301
302 <pre caption="Getting Distcc on the new box">
303 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
304 </pre>
305
306 </body>
307 </section>
308 <section>
309 <title>Step 3: Setting Up Distcc</title>
310 <body>
311
312 <p>
313 Run <c>distcc-config --install</c> to setup distcc:
314 </p>
315
316 <pre caption="Final distcc setup">
317 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
318 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
319 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment>
320 </pre>
321
322 <p>
323 Distcc is now set up to bootstrap! Continue with the official installation
324 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
325 system</c>. This is to make sure that all of the dependencies you want are
326 installed as well.
327 </p>
328
329 <note>
330 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
331 This is expected as some ebuilds do not work well with distcc, so they
332 intentionally disable it.
333 </note>
334
335 </body>
336 </section>
337 </chapter>
338
339 <chapter>
340 <title>Troubleshooting</title>
341 <section>
342 <title>Mozilla and Xfree</title>
343 <body>
344
345 <p>
346 As you emerge various packages, you'll notice that some of them aren't being
347 distributed (and aren't being built in parallel). This is because the
348 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
349 building because it is known to cause problems.
350 </p>
351
352 <p>
353 Sometimes distcc might cause a package to fail to compile. If this happens
354 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
355 </p>
356
357 </body>
358 </section>
359 <section>
360 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
361 <body>
362
363 <p>
364 With such a long title any explanation here is almost irrelevent. However, if
365 you plan to use distcc across hosts that have the <uri
366 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
367 not, you will run into problems.
368 </p>
369
370 <p>
371 The solution requires a little foresight on your part; you have to run
372 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
373 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
374 which are a good thing to do since for the most part the protections offered
375 by both packages is a good thing and is transparent to the user.
376 </p>
377
378 </body>
379 </section>
380 <section>
381 <title>Mixed GCC Versions</title>
382 <body>
383
384 <p>
385 If you have different GCC versions on your hosts, there will likely be very
386 weird problems. The solution is to make certain all hosts have the same GCC
387 version.
388 </p>
389
390 <p>
391 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
392 <c>gcc</c>. This means that if you're mixing i686 machines with other types
393 (i386, i586) you will run into problems. A workaround for this may be to
394 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>,
395 but that needs testing. If it works for you please email me,
396 <mail link="lisa@gentoo.org">Lisa Seelye</mail>.
397 </p>
398
399 </body>
400 </section>
401 </chapter>
402
403 <chapter>
404 <title>Distcc Extras</title>
405 <section>
406 <title>Distcc Monitors</title>
407 <body>
408
409 <p>
410 Distcc ships with two monitors. The text-based one is always built and is
411 called <c>distccmon-text</c>. Running it for the first time can be a bit
412 confusing, but it is really quite easy to use. If you run the program with no
413 parameter it will run once. However, if you pass it a number it will update
414 every N seconds, where N is the argument you passed.
415 </p>
416
417 <p>
418 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
419 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
420 and it is quite lovely.
421 </p>
422
423 <pre caption="Starting the monitors">
424 # <i>distccmon-text N</i>
425 <comment>(Or)</comment>
426 # <i>distccmon-gnome</i>
427 <comment>To monitor Portage's distcc usage you can use:</comment>
428 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
429 <comment>(Or)</comment>
430 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gnome</i>
431 </pre>
432
433 <impo>
434 If your distcc directory is elsewhere, change the DISTCC_DIR variable
435 accordingly.
436 </impo>
437
438 </body>
439 </section>
440 </chapter>
441 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20