/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.22 - (show annotations) (download) (as text)
Sun Aug 22 20:51:08 2004 UTC (10 years, 3 months ago) by neysx
Branch: MAIN
Changes since 1.21: +34 -51 lines
File MIME type: application/xml
#61292 New version from Lisa

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /home/cvsroot/distcc/html/distcc.xml,v 1.16 2004/08/22 20:28:16 lisa Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12
13 <author title="Editor">
14 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 </author>
16
17 <author title="Editor">
18 <mail link="erwin@gentoo.org">Erwin</mail>
19 </author>
20
21 <author title="Editor">
22 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 </author>
24
25 <author title="Editor">
26 <mail link="pylon@gentoo.org">Lars Weiler</mail>
27 </author>
28
29 <author title="Reviewer">
30 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 </author>
32
33 <abstract>
34 This document serves as a HOWTO for using distcc with Gentoo.
35 </abstract>
36
37 <!-- The content of this document is licensed under the CC-BY-SA license -->
38 <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 <license/>
40
41 <version>1.8</version>
42 <date>August 22, 2004</date>
43
44 <chapter>
45 <title>Introduction</title>
46 <section>
47 <title>What is distcc?</title>
48 <body>
49
50 <p>
51 Distcc is a program designed to distribute compiling tasks across a network to
52 participating hosts. It is comprised of a server, <c>distccd</c> and a client
53 program, <c>distcc</c>. Distcc can work transparently with <uri
54 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
55 </p>
56
57 </body>
58 </section>
59 <section>
60 <title>Using distcc to bootstrap</title>
61 <body>
62
63 <p>
64 If you are planning on using distcc to help you bootstrap a Gentoo installation,
65 make sure you read the part on <uri link="#bootstrapping">Using distcc to
66 Bootstrap</uri> which is situated further down in this document.
67 </p>
68
69 </body>
70 </section>
71 </chapter>
72
73 <chapter>
74 <title>Setup</title>
75 <section>
76 <title>Dependencies</title>
77 <body>
78
79 <p>
80 To start off with, to use Distcc all of the computers on your network need to
81 have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
82 mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
83 </p>
84
85 </body>
86 </section>
87 <section>
88 <title>Installing Distcc</title>
89 <body>
90
91 <p>
92 There are a couple of options you should know about before you start
93 installing distcc.
94 </p>
95
96 <p>
97 Distcc ships with a graphical monitor to monitor tasks that your computer is
98 sending away for compilation. If you use Gnome then put 'gnome' in your USE
99 flags. However, if you don't use Gnome and would still like to have the
100 monitor then you should put 'gtk' in your USE flags.
101 </p>
102
103 <pre caption="Installing distcc">
104 # <i>emerge distcc</i>
105 </pre>
106
107 </body>
108 </section>
109 <section>
110 <title>Setting up Portage to use Distcc</title>
111 <body>
112
113 <p>
114 Setting up distcc is easy to do with Portage. Execute the following steps on
115 each system that should participate in the distributed compiling:
116 </p>
117
118 <pre caption="Integrating Distcc and Portage">
119 # <i>emerge distcc</i>
120 # <i>nano -w /etc/make.conf</i>
121 <comment>(Set N to a suitable number for your particular setup)</comment>
122 <comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment>
123 MAKEOPTS="-jN"
124 <comment>(Add distcc to your FEATURES variable)</comment>
125 FEATURES="distcc"
126 </pre>
127
128 <p>
129 Next you have to specify what hosts you want to use. Use the
130 <c>distcc-config</c> command to set the list of hosts. Here is an example of
131 some hosts that might be in your list:
132 </p>
133
134 <pre caption="Examples of host definitions">
135 192.168.0.1 192.168.0.2 192.168.0.3
136 192.168.0.1/2 192.168.0.2 192.168.0.3/10
137 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 <comment>(There are also several other methods of setting up hosts. See the
140 distcc manpage for more details.)</comment>
141 </pre>
142
143 <p>
144 It may all look complicated, but in most cases a variant of line 1 or 2 will
145 work.
146 </p>
147
148 <ul>
149 <li>Line 1 is just a space-delimited list of hosts</li>
150 <li>
151 Line 2 is a list of hosts that specifies the maximum amount of jobs (by
152 defining the <c>/N</c> setting) to send that host at any given time
153 </li>
154 </ul>
155
156 <p>
157 Since most people won't be using lines 3 or 4, I'll <uri
158 link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
159 docs for more information.
160 </p>
161
162 <p>
163 For instance, to set the first line in the previous example:
164 </p>
165
166 <pre caption="Sample command to set the hosts">
167 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 </pre>
169
170 <p>
171 Now start the distcc daemon on all the participating computers:
172 </p>
173
174 <pre caption="Starting the distcc daemon">
175 <comment>(Add distccd to the default runlevel)</comment>
176 # <i>rc-update add distccd default</i>
177 <comment>(Start the distcc daemon)</comment>
178 # <i>/etc/init.d/distccd start</i>
179 </pre>
180
181 </body>
182 </section>
183 <section>
184 <title>Setting up Distcc to Work With Automake</title>
185 <body>
186
187 <p>
188 This is in some cases easier than the Portage setup. What you have to do is
189 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
190 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
191 However, there is a caveat. If you use ccache you have to put distcc after
192 the ccache part:
193 </p>
194
195 <pre caption="Setting your path">
196 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
197 </pre>
198
199 <p>
200 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
201 (where N is an integer). The value of N depends on your network and the types
202 of computers you are using to compile. For me, I have a dual processor P3 and
203 a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
204 Thunderbird) and I use -j5. Test your own settings to find the number that
205 yields the best performance.
206 </p>
207
208 </body>
209 </section>
210 </chapter>
211
212 <chapter>
213 <title>Cross-Compiling</title>
214 <section>
215 <title>A Note on Cross-Compiling</title>
216 <body>
217
218 <p>
219 Cross-compiling is using one architecture to build programs for another
220 architecture. This can be as simple as using an Athlon (i686) to build a
221 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
222 </p>
223
224 </body>
225 </section>
226 <section>
227 <title>An Introduction to Cross-compiling</title>
228 <body>
229
230 <p>
231 If you want to give Cross-compiling a try you can emerge the crossdev package.
232 I can't support it since I didn't write it, but that's a good way to start
233 cross-compiling.
234 </p>
235
236 </body>
237 </section>
238 </chapter>
239
240 <chapter id="bootstrapping">
241 <title>Using Distcc to Bootstrap</title>
242 <section>
243 <title>Step 1: Configure Portage </title>
244 <body>
245
246 <p>
247 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
248 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
249 up until the bootstrapping part. Then configure Portage to use distcc:
250 </p>
251
252 <pre caption="Preliminary Setup">
253 # <i>nano -w /etc/make.conf</i>
254 <comment>(Add distcc to the FEATURES variable)</comment>
255 FEATURES="distcc"
256 <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
257 MAKEOPTS="-jN"
258 </pre>
259
260 <pre caption="Setting your path">
261 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
262 </pre>
263
264 </body>
265 </section>
266 <section>
267 <title>Step 2: Getting Distcc</title>
268 <body>
269
270 <p>
271 Before the installation of distcc, a user called distcc must be added to the
272 <path>/etc/passwd</path>:
273 </p>
274
275 <pre caption="Create user distcc">
276 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
277 </pre>
278
279 <p>
280 Install distcc:
281 </p>
282
283 <pre caption="Getting Distcc on the new box">
284 # <i>USE='-*' emerge --nodeps distcc</i>
285 </pre>
286
287 </body>
288 </section>
289 <section>
290 <title>Step 3: Setting Up Distcc</title>
291 <body>
292
293 <p>
294 Run <c>distcc-config</c> to setup distcc:
295 </p>
296
297 <pre caption="Final distcc setup">
298 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
299 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
300 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment>
301 </pre>
302
303 <p>
304 distcc is now set up to bootstrap! Continue the official installation
305 instructions and <e>don't forget</e> to re-emerge distcc after <c>emerge
306 system</c>. This is to make sure that all of the dependencies you want are
307 installed as well.
308 </p>
309
310 <note>
311 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
312 This is expected as some ebuilds do not work well with distcc, so they
313 intentionally disable it.
314 </note>
315
316 </body>
317 </section>
318 </chapter>
319
320 <chapter>
321 <title>Troubleshooting</title>
322 <section>
323 <title>Mozilla and Xfree</title>
324 <body>
325
326 <p>
327 As you emerge various packages, you'll notice that some of them aren't being
328 distributed (and aren't being built in parallel). This is because the
329 developers of the Mozilla and Xfree ebuilds intentionally disable parallel
330 building because it is known to cause problems.
331 </p>
332
333 <p>
334 Sometimes distcc might cause a package to fail to compile. If this happens
335 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
336 </p>
337
338 </body>
339 </section>
340 <section>
341 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
342 <body>
343
344 <p>
345 With such a long title any explanation here is almost irrelevent. However, if
346 you plan to use distcc across hosts that have the <uri
347 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
348 not, you will run into problems.
349 </p>
350
351 <p>
352 The solution requires a little foresight on your part; you have to run
353 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
354 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
355 which are a good thing to do since for the most part the protections offered
356 by both packages is a good thing and is transparent to the user.
357 </p>
358
359 </body>
360 </section>
361 <section>
362 <title>Mixed GCC Versions</title>
363 <body>
364
365 <p>
366 If you have different GCC versions on your hosts, there will likely be very
367 weird problems. The solution is to make certain all hosts have the same GCC
368 version.
369 </p>
370
371 </body>
372 </section>
373 </chapter>
374
375 <chapter>
376 <title>Distcc Extras</title>
377 <section>
378 <title>Distcc Monitors</title>
379 <body>
380
381 <p>
382 Distcc ships with two monitors. The text-based one is always built and is
383 called <c>distccmon-text</c>. Running it for the first time can be a bit
384 confusing, but it is really quite easy to use. If you run the program with no
385 parameter it will run once. However, if you pass it a number it will update
386 every N seconds, where N is the argument you passed.
387 </p>
388
389 <p>
390 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
391 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
392 and it is quite lovely.
393 </p>
394
395 <pre caption="Starting the monitors">
396 # <i>distccmon-text N</i>
397 <comment>(Or)</comment>
398 # <i>distccmon-gnome</i>
399 <comment>To monitor Portage's distcc usage you can use:</comment>
400 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-text N</i>
401 <comment>(Or)</comment>
402 # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-gnome</i>
403 </pre>
404
405 <impo>
406 Notice the backticks above. <c>portageq</c> is a part of Portage that will
407 return a piece of information (in this case what DISTCC_DIR is set to). For
408 simplicity, most users can use <c>DISTCC_DIR=/var/tmp/portage/.distcc
409 distccmon-text N</c>
410 </impo>
411
412 </body>
413 </section>
414 </chapter>
415 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20