/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.12 - (show annotations) (download) (as text)
Fri Jan 9 20:48:14 2004 UTC (10 years, 7 months ago) by swift
Branch: MAIN
Changes since 1.11: +16 -13 lines
File MIME type: application/xml
Fix #36386

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /home/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.11 2003/11/29 09:45:07 swift Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12 <author title="Editor">
13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 </author>
15 <author title="Editor">
16 <mail link="erwin@gentoo.org">Erwin</mail>
17 </author>
18 <author title="Reviewer">
19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 </author>
21 <author title="Reviewer">
22 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
23 </author>
24
25 <license/>
26
27 <version>1.3</version>
28 <date>January 9, 2004</date>
29
30 <abstract>
31 This document serves as a HOWTO for using distcc with Gentoo.
32 </abstract>
33
34 <chapter>
35 <title>Introduction</title>
36 <section>
37 <title>What is distcc?</title>
38 <body>
39
40 <p>
41 Distcc is a program designed to distribute compiling tasks across a network to
42 participating hosts. It is comprised of a server, <c>distccd</c> and a client
43 program, <c>distcc</c>. Distcc can work transparently with <uri
44 link="http://ccache.samba.org">ccache</uri> and Portage with a little set up.
45 </p>
46
47 </body>
48 </section>
49 </chapter>
50
51 <chapter>
52 <title>Setup</title>
53 <section>
54 <title>Dependencies</title>
55 <body>
56
57 <pre caption="Distcc dependencies (2.8 through 2.10)">
58 &gt;=sys-apps/portage-2.0.46-r11
59 <codenote>Distcc-2.11 and on require &gt;=sys-apps/portage-2.0.49-r6</codenote>
60 &gt;=sys-devel/gcc-config-1.3.1
61 sys-apps/shadow
62 <codenote>(As of version 2.8 and up until 2.11) And the following optional dependencies when you have <i>gtk</i> in your <i>USE</i> flags</codenote>
63 &gt;=x11-libs/gtk+-2.2.1
64 </pre>
65
66 <pre caption="Distcc Dependencies (post 2.11.1)">
67 &gt;=sys-apps/portage-2.0.49-r6
68 &gt;=sys-devel/gcc-config-1.3.1
69 sys-apps/shadow
70 <codenote>In this revision you may chose between a Gnome and GTK GUI monitor, they have the following added dependencies</codenote>
71 <codenote>For GTK:</codenote>
72 &gt;=x11-libs/gtk+-2.0.0
73 &gt;=gnome-base/libglade-2.0.0
74 x11-libs/pango
75 <codenote>For Gnome:</codenote>
76 &gt;=x11-libs/gtk+-2.0.0
77 &gt;=gnome-base/libglade-2.0.0
78 x11-libs/pango
79 &gt;=gnome-base/libgnomeui-2.0.0.0
80 &gt;=gnome-base/libgnome-2.0.0
81 </pre>
82
83 </body>
84 </section>
85 <section>
86 <title>Installing Distcc</title>
87 <body>
88
89 <p>
90 Installing Distcc is very easy. Simply set your USE flags and <c>emerge
91 distcc</c>. But, there's a couple of options you should know about.
92 </p>
93
94 <p>
95 Distcc ships with a graphical monitor to monitor tasks that your computer is
96 sending away for compilation. If you use Gnome then put 'gnome' in your USE
97 flags. However, if you don't use Gnome and would still like to have the
98 monitor then you should put 'gtk' in your USE flags.
99 </p>
100
101 </body>
102 </section>
103 <section>
104 <title>Setting up Portage to use Distcc</title>
105 <body>
106
107 <p>
108 Setting up distcc is very easy to do with Portage. Follow these simple steps on
109 each computer you want to use distcc on:
110 </p>
111
112 <pre caption="Integrating Distcc and Portage">
113 # <i>emerge distcc</i>
114 # <i>nano -w /etc/make.conf</i>
115 </pre>
116
117 <impo>
118 If you use distcc-2.11.1 or greater with &gt;=portage-2.0.49-r6, then all you
119 need to do as far as setup is to set your hosts (see next code block), modify
120 MAKEOPTS in <path>/etc/make.conf</path>, and add 'distcc' to FEATURES in
121 <path>/etc/make.conf</path>.
122 </impo>
123
124 <note>
125 Edit your FEATURES flags to include &quot;distcc&quot;
126 </note>
127
128 <p>
129 Next you have to specify what hosts you want to use. To do this you can use
130 the <c>distcc-config</c> command to set the list of hosts. Here is an example
131 of some hosts that might be in your list:
132 </p>
133
134 <pre caption="Examples of host definitions">
135 192.168.0.1 192.168.0.2 192.168.0.3
136 192.168.0.1/2 192.168.0.2 192.168.0.3/10
137 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 <codenote>There are also several other methods of setting up hosts. </codenote>
140 <codenote>See the distcc manpage for more details.</codenote>
141 </pre>
142
143 <p>
144 It may all look complicated, but in most cases a variant of line 1 or 2 will
145 work. An explanation of each line is: Line 1 is just a space-delimited list
146 of hosts that will use default everything. Line 2 is a list of hosts that
147 specifies the maximum number of jobs (by use of the /N) to send that host at
148 any given time (specified with the <c>/n</c>). Since most people won't be
149 using lines 3 or 4, I'll <uri link="http://distcc.samba.org/man/distcc_1.html">
150 point you to</uri> the distcc docs for more information.
151 </p>
152
153 <p>
154 A sample command to set the hosts (for line 1) is:
155 </p>
156
157 <pre caption="Sample command to set the hosts">
158 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
159 </pre>
160
161 <p>
162 The final step to integrating distcc into Portage is to re-open your
163 <path>/etc/make.conf</path> and edit <c>MAKEOPTS</c> to include <c>-jN</c>
164 (where N is an integer). Typically you will want to set this to the total
165 number of processors in your network plus one.
166 </p>
167
168 <pre caption="Final steps in make.conf">
169 # <i>nano -w /etc/make.conf</i>
170 MAKEOPTS=-jN
171 </pre>
172
173 <impo>
174 Don't forget to start the distcc daemon! <c>/etc/init.d/distccd start</c>
175 </impo>
176
177 <note>
178 You will likely have to find a suitable 'N' for your particular setup. Another
179 strategy is 2 x #number of CPUs + 1.
180 </note>
181
182 </body>
183 </section>
184 <section>
185 <title>Setting up Distcc to Work With Automake</title>
186 <body>
187
188 <p>
189 This is in some cases easier than the Portage setup. What you have to do is
190 update your <c>PATH</c> environmental variable to include
191 <path>/usr/lib/distcc/bin</path> before where gcc (<path>/usr/bin</path>).
192 However, there is a caveat. If you use ccache you have to put distcc after
193 the ccache part. Like so:
194 </p>
195
196 <pre caption="Setting your path">
197 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
198 </pre>
199
200 <p>
201 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
202 (where N is an integer). A safe number to use for N varies on your network
203 and the types of computers you are using to compile. For me, I have a dual
204 processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz
205 Athlon Thunderbird) and I use -j5. Test your own settings to find the right
206 number.
207 </p>
208
209 </body>
210 </section>
211 </chapter>
212
213 <chapter>
214 <title>Cross-Compiling</title>
215 <section>
216 <title>A Note on Cross-Compiling</title>
217 <body>
218
219 <p>
220 Cross-compiling is using one architecture to build programs for another
221 architecture. This can be as simple as using an Athlon (i686) to build a
222 program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
223 </p>
224
225 </body>
226 </section>
227 <section>
228 <title>A Personal Note on Cross-compiling</title>
229 <body>
230
231 <p>
232 I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
233 don't have any non-x86 machines. I can create things that work <e>in
234 theory</e>, but I have to rely on other people to test what I write. That
235 works to an extent, but it's tough. ;-)
236 </p>
237
238 <p>
239 In the near future I hope to acquire a Sparc of some type to put Gentoo on so
240 I can play around in-house.
241 </p>
242
243 </body>
244 </section>
245 </chapter>
246
247 <chapter>
248 <title>Using Distcc to Bootstrap</title>
249 <section>
250 <title>Step 1: Setup on New Box</title>
251 <body>
252
253 <p>
254 Next, you have to boot your new box with a Gentoo Linux LiveCD and follow all
255 of the steps up until the bootstrapping. Then proceed with a little preliminary
256 setup on the new box.
257 </p>
258
259 <pre caption="Preliminary Setup">
260 <codenote>Here we add distcc to FEATURES</codenote>
261 # <i>nano -w /etc/make.conf</i>
262 <codenote>Add &quot;distcc&quot; to FEATURES</codenote>
263 <codenote>Modify MAKEOPTS in <path>/etc/make.conf to include -jN (as described above)</path></codenote>
264 </pre>
265
266 <p>
267 Then add the distcc user to your <path>/etc/passwd</path>:
268 </p>
269
270 <pre caption="Add the distcc user to /etc/passwd">
271 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
272 <codenote>Do not forget the `<i>&gt;&gt;</i>'</codenote>
273 </pre>
274
275 </body>
276 </section>
277 <section>
278 <title>Step 2: Getting Distcc</title>
279 <body>
280
281 <p>
282 Next you will want to issue the following command inside the chroot on the new
283 box.
284 </p>
285
286 <pre caption="Getting Distcc on the new box">
287 # <i>USE='-*' emerge --nodeps distcc</i>
288 </pre>
289
290 <note>
291 <uri link="http://bugs.gentoo.org/show_bug.cgi?id=34262">Bug 34262</uri>
292 mentions a problem with pkg-config (dev-util/pkgconfig), so it may be necessary
293 to also issue <c>emerge --nodeps pkgconfig</c> prior to the above emerge.
294 </note>
295
296 </body>
297 </section>
298 <section>
299 <title>Step 3: Setting Up Distcc Itself</title>
300 <body>
301
302 <p>
303 Now set up distcc itself...
304 </p>
305
306 <pre caption="Final distcc setup">
307 # <i>/usr/bin/distcc-config --install</i>
308 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
309 </pre>
310
311 <p>
312 distcc should be set up now to bootstrap! Continue the official install guide,
313 and don't forget to re-emerge distcc after <c>emerge system</c>.
314 </p>
315
316 <note>
317 It is important to note that during bootstrap and <c>emerge system</c> distcc
318 may not appear to be being used. This is not to be unexpected. Some ebuilds or
319 packages do not work well with distcc, so they intentionally disable distcc.
320 </note>
321
322 </body>
323 </section>
324 </chapter>
325
326 <chapter>
327 <title>Troubleshooting</title>
328 <section>
329 <title>Mozilla and Xfree</title>
330 <body>
331
332 <p>
333 As you emerge various packages, you'll notice that they aren't being distributed
334 (and indeed aren't being built in parallel). This is because the developers of
335 Mozilla and Xfree .ebuilds disabled parallel building because it is known to
336 cause problems. This isn't necessarily a distcc problem.
337 </p>
338
339 <p>
340 That isn't to say that sometimes distcc will cause a package to fail to compile.
341 </p>
342
343 </body>
344 </section>
345 <section>
346 <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
347 <body>
348
349 <p>
350 With such a long title any explanation here is almost irrelevent. However, if
351 you plan to use distcc across hosts that have the <uri
352 link="http://www.gentoo.org/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri>
353 and some that do not, you will run into problems.
354 </p>
355
356 <p>
357 The solution requires a little foresight on your part; you have to run
358 <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
359 enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
360 which are a good thing to do since for the most part the protections offered
361 by both packages is a good thing and is transparent to the user.
362 </p>
363
364 </body>
365 </section>
366 <section>
367 <title>Mixed GCC Versions</title>
368 <body>
369
370 <p>
371 If you have differing versions of GCC on your hosts, there will likely be very
372 weird problems. The solution is to make certain all hosts have the same GCC
373 version.
374 </p>
375
376 </body>
377 </section>
378 </chapter>
379
380 <chapter>
381 <title>Distcc Extras</title>
382 <section>
383 <title>Distcc Monitors</title>
384 <body>
385
386 <p>
387 Distcc ships with two monitors. The text-based one is always built, it is
388 called <c>distccmon-text</c>. Running it for the first time can be a bit
389 confusing, but it is really quite easy to use. If you run the program with no
390 parameter it will run once. However, if you pass it a number it will update
391 every N seconds, where N is the argument you passed.
392 </p>
393
394 <p>
395 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
396 in your <c>USE</c> flags. This one is GTK+ based and runs in an X environment,
397 and it is quite lovely.
398 </p>
399
400 <p>
401 There is a caveat to using these programs. If you want to monitor any emerge,
402 you have to start the monitor like so:
403 </p>
404
405 <pre caption="Starting the monitors">
406 # <i>DISTCC_DIR=/path/to/distccdir distccmon-text N</i>
407 <codenote>Or you can start the graphical monitor...</codenote>
408 # <i>DISTCC_DIR=/path/to/distccdir distccmon-gnome</i>
409 </pre>
410
411 <impo>
412 If you are using Portage and used the above example, this will be
413 <c>DISTCC_DIR=/var/tmp/portage/.distcc</c>.
414 </impo>
415
416 <note>
417 If you aren't using Portage you do not need to specify DISTCC_DIR (i.e., just
418 type distccmon-text N).
419 </note>
420
421 </body>
422 </section>
423 </chapter>
424
425 <chapter>
426 <title>Future Plans for Distcc and Gentoo</title>
427 <section>
428 <title>distcc-subnetscan</title>
429 <body>
430
431 <p>
432 <c>distcc-subnetscan</c> is a perl program in development that will scan a
433 subnet for hosts that have a participating distcc daemon. Even this will be
434 enhanced to test if the daemon on a remote host will conform to a specific
435 <c>CHOST</c> setting to make cross-compiling easier.
436 </p>
437
438 <p>
439 The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">
440 right here</uri> until a more formal home can be found.
441 </p>
442
443 </body>
444 </section>
445 <section>
446 <title>distcc-config</title>
447 <body>
448
449 <p>
450 <c>distcc-config</c>, the userland configuration tool for distcc is out of date.
451 It has been rewritten and is present in <path>distcc-2.11.1-r1</path> and up.
452 </p>
453
454 </body>
455 </section>
456 </chapter>
457 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20