/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.24 - (show annotations) (download) (as text)
Wed Dec 22 22:43:45 2004 UTC (9 years, 8 months ago) by swift
Branch: MAIN
Changes since 1.23: +28 -11 lines
File MIME type: application/xml
#75369 - various distcc updates

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.23 2004/11/09 11:37:30 swift Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12
13 <author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 </author>
16
17 <author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail>
19 </author>
20
21 <author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 </author>
24
25 <author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27 </author>
28
29 <author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 </author>
32
33 <abstract>
34 This document serves as a HOWTO for using distcc with Gentoo.
35 </abstract>
36
37 <!-- The content of this document is licensed under the CC-BY-SA license -->
38 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 <license/>
40
41 <version>1.10</version>
42 <date>December 22, 2004</date>
43
44 <chapter>
45 <title>Introduction</title>
46 <section>
47 <title>What is distcc?</title>
48 <body>
49
50 <p>
51 Distcc is a program designed to distribute compiling tasks across a network to
52 participating hosts. It is comprised of a server, <c>distccd</c> and a client
53 program, <c>distcc</c>. Distcc can work transparently with <uri
54 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
55 </p>
56
57 </body>
58 </section>
59 <section>
60 <title>Using distcc to bootstrap</title>
61 <body>
62
63 <p>
64 If you are planning on using distcc to help you bootstrap a Gentoo installation,
65 make sure you read the part on <uri link="#bootstrapping">Using distcc to
66 Bootstrap</uri> which is situated further down in this document.
67 </p>
68
69 </body>
70 </section>
71 </chapter>
72
73 <chapter>
74 <title>Setup</title>
75 <section>
76 <title>Dependencies</title>
77 <body>
78
79 <p>
80 To start off with, to use Distcc all of the computers on your network need to
81 have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
82 mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
83 </p>
84
85 </body>
86 </section>
87 <section>
88 <title>Installing Distcc</title>
89 <body>
90
91 <p>
92 There are a couple of options you should know about before you start
93 installing distcc.
94 </p>
95
96 <p>
97 Distcc ships with a graphical monitor to monitor tasks that your computer is
98 sending away for compilation. If you use Gnome then put 'gnome' in your USE
99 flags. However, if you don't use Gnome and would still like to have the
100 monitor then you should put 'gtk' in your USE flags.
101 </p>
102
103 <pre caption="Installing distcc">
104 # <i>emerge distcc</i>
105 </pre>
106
107 </body>
108 </section>
109 <section>
110 <title>Setting up Portage to use Distcc</title>
111 <body>
112
113 <p>
114 Setting up distcc is easy to do with Portage. Execute the following steps on
115 each system that should participate in the distributed compiling:
116 </p>
117
118 <pre caption="Integrating Distcc and Portage">
119 # <i>emerge distcc</i>
120 # <i>nano -w /etc/make.conf</i>
121 <comment>(Set N to a suitable number for your particular setup)</comment>
122 <comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment>
123 MAKEOPTS="-jN"
124 <comment>(Add distcc to your FEATURES variable)</comment>
125 FEATURES="distcc"
126 </pre>
127
128 <p>
129 Next you have to specify what hosts you want to use. Use the
130 <c>distcc-config</c> command to set the list of hosts. Here is an example of
131 some hosts that might be in your list:
132 </p>
133
134 <pre caption="Examples of host definitions">
135 192.168.0.1 192.168.0.2 192.168.0.3
136 192.168.0.1/2 192.168.0.2 192.168.0.3/10
137 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 <comment>(There are also several other methods of setting up hosts. See the
140 distcc manpage for more details.)</comment>
141 </pre>
142
143 <p>
144 It may all look complicated, but in most cases a variant of line 1 or 2 will
145 work.
146 </p>
147
148 <ul>
149 <li>Line 1 is just a space-delimited list of hosts</li>
150 <li>
151 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
152 defining the <c>/N</c> setting) to send that host at any given time
153 </li>
154 </ul>
155
156 <p>
157 Since most people won't be using lines 3 or 4, I'll <uri
158 link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
159 docs for more information.
160 </p>
161
162 <p>
163 For instance, to set the first line in the previous example:
164 </p>
165
166 <pre caption="Sample command to set the hosts">
167 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 </pre>
169
170 <p>
171 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
172 <c>--allow</c> directive to allow only hosts you trust. For more information
173 please see the <uri link="http://distcc.samba.org/security.html">Distcc Security
174 Design</uri>. For added security you should also use the <c>--listen</c>
175 directive to tell the Distcc daemon what IP to listen on (for multi-homed
176 systems).
177 </p>
178
179 <p>
180 Now start the distcc daemon on all the participating computers:
181 </p>
182
183 <pre caption="Starting the distcc daemon">
184 <comment>(Add distccd to the default runlevel)</comment>
185 # <i>rc-update add distccd default</i>
186 <comment>(Start the distcc daemon)</comment>
187 # <i>/etc/init.d/distccd start</i>
188 </pre>
189
190 </body>
191 </section>
192 <section>
193 <title>Setting up Distcc to Work With Automake</title>
194 <body>
195
196 <p>
197 This is in some cases easier than the Portage setup. What you have to do is
198 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
199 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
200 However, there is a caveat. If you use ccache you have to put distcc after
201 the ccache part:
202 </p>
203
204 <pre caption="Setting your path">
205 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
206 </pre>
207
208 <p>
209 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
210 (where N is an integer). The value of N depends on your network and the types
211 of computers you are using to compile. For me, I have a dual processor P3 and
212 a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
213 Thunderbird) and I use -j5. Test your own settings to find the number that
214 yields the best performance.
215 </p>
216
217 </body>
218 </section>
219 </chapter>
220
221 <chapter>
222 <title>Cross-Compiling</title>
223 <section>
224 <title>A Note on Cross-Compiling</title>
225 <body>
226
227 <p>
228 Cross-compiling is using one architecture to build programs for another
229 architecture. This can be as simple as using an Athlon (i686) to build a
230 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
231 </p>
232
233 </body>
234 </section>
235 <section>
236 <title>An Introduction to Cross-compiling</title>
237 <body>
238
239 <p>
240 If you want to give Cross-compiling a try you can to follow <uri
241 link="http://dev.gentoo.org/~vapier/CROSS-COMPILE-HOWTO">The Cross Compile
242 HOWTO</uri>; Crossdev is depreciated.
243 </p>
244
245 </body>
246 </section>
247 </chapter>
248
249 <chapter id="bootstrapping">
250 <title>Using Distcc to Bootstrap</title>
251 <section>
252 <title>Step 1: Configure Portage</title>
253 <body>
254
255 <p>
256 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
257 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
258 up until the bootstrapping part. Then configure Portage to use distcc:
259 </p>
260
261 <pre caption="Preliminary Setup">
262 # <i>nano -w /etc/make.conf</i>
263 <comment>(Add distcc to the FEATURES variable)</comment>
264 FEATURES="distcc"
265 <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
266 MAKEOPTS="-jN"
267 </pre>
268
269 <pre caption="Setting your path">
270 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
271 </pre>
272
273 </body>
274 </section>
275 <section>
276 <title>Step 2: Getting Distcc</title>
277 <body>
278
279 <p>
280 Before the installation of distcc, a user called distcc must be added to the
281 <path>/etc/passwd</path>:
282 </p>
283
284 <pre caption="Create user distcc">
285 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
286 </pre>
287
288 <impo>
289 It is important to note that adding users like this is very bad. We only do
290 it here because there is no <c>useradd</c> utility (which you normally use for
291 adding users) yet at this stage of installation.
292 </impo>
293
294 <p>
295 Install distcc:
296 </p>
297
298 <pre caption="Getting Distcc on the new box">
299 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
300 </pre>
301
302 </body>
303 </section>
304 <section>
305 <title>Step 3: Setting Up Distcc</title>
306 <body>
307
308 <p>
309 Run <c>distcc-config --install</c> to setup distcc:
310 </p>
311
312 <pre caption="Final distcc setup">
313 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
314 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
315 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment>
316 </pre>
317
318 <p>
319 distcc is now set up to bootstrap! Continue the official installation
320 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
321 system</c>. This is to make sure that all of the dependencies you want are
322 installed as well.
323 </p>
324
325 <note>
326 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
327 This is expected as some ebuilds do not work well with distcc, so they
328 intentionally disable it.
329 </note>
330
331 </body>
332 </section>
333 </chapter>
334
335 <chapter>
336 <title>Troubleshooting</title>
337 <section>
338 <title>Mozilla and Xfree</title>
339 <body>
340
341 <p>
342 As you emerge various packages, you'll notice that some of them aren't being
343 distributed (and aren't being built in parallel). This is because the
344 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
345 building because it is known to cause problems.
346 </p>
347
348 <p>
349 Sometimes distcc might cause a package to fail to compile. If this happens
350 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
351 </p>
352
353 </body>
354 </section>
355 <section>
356 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
357 <body>
358
359 <p>
360 With such a long title any explanation here is almost irrelevent. However, if
361 you plan to use distcc across hosts that have the <uri
362 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
363 not, you will run into problems.
364 </p>
365
366 <p>
367 The solution requires a little foresight on your part; you have to run
368 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
369 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
370 which are a good thing to do since for the most part the protections offered
371 by both packages is a good thing and is transparent to the user.
372 </p>
373
374 </body>
375 </section>
376 <section>
377 <title>Mixed GCC Versions</title>
378 <body>
379
380 <p>
381 If you have different GCC versions on your hosts, there will likely be very
382 weird problems. The solution is to make certain all hosts have the same GCC
383 version.
384 </p>
385
386 <p>
387 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
388 <c>gcc</c>. This means that if you're mixing i686 machines with other types
389 (i386, i586) you will run into problems. A workaround for this may be to
390 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>,
391 but that needs testing. If it works for you please email me,
392 <mail link="lisa@gentoo.org">Lisa Seelye</mail>.
393 </p>
394
395 </body>
396 </section>
397 </chapter>
398
399 <chapter>
400 <title>Distcc Extras</title>
401 <section>
402 <title>Distcc Monitors</title>
403 <body>
404
405 <p>
406 Distcc ships with two monitors. The text-based one is always built and is
407 called <c>distccmon-text</c>. Running it for the first time can be a bit
408 confusing, but it is really quite easy to use. If you run the program with no
409 parameter it will run once. However, if you pass it a number it will update
410 every N seconds, where N is the argument you passed.
411 </p>
412
413 <p>
414 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
415 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
416 and it is quite lovely.
417 </p>
418
419 <pre caption="Starting the monitors">
420 # <i>distccmon-text N</i>
421 <comment>(Or)</comment>
422 # <i>distccmon-gnome</i>
423 <comment>To monitor Portage's distcc usage you can use:</comment>
424 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-text N</i>
425 <comment>(Or)</comment>
426 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-gnome</i>
427 </pre>
428
429 <impo>
430 Notice the backticks above. <c>portageq</c> is a part of Portage that will
431 return a piece of information (in this case what DISTCC_DIR is set to). For
432 simplicity, most users can use <c>DISTCC_DIR=/var/tmp/portage/.distcc
433 distccmon-text N</c>
434 </impo>
435
436 </body>
437 </section>
438 </chapter>
439 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20