/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.45 - (show annotations) (download) (as text)
Tue Jul 24 12:12:51 2012 UTC (23 months, 2 weeks ago) by swift
Branch: MAIN
Changes since 1.44: +6 -6 lines
File MIME type: application/xml
Fix bug #427860 - Use /etc/portage for make.conf and make.profile. Old location (/etc) is still supported, this is a heads up (new default)

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.44 2011/09/04 17:53:40 swift Exp $ -->
4
5 <guide>
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12 <author title="Editor">
13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 </author>
15 <author title="Editor">
16 <mail link="erwin@gentoo.org">Erwin</mail>
17 </author>
18 <author title="Editor">
19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 </author>
21 <author title="Editor">
22 <mail link="pylon@gentoo.org">Lars Weiler</mail>
23 </author>
24 <author title="Reviewer">
25 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
26 </author>
27 <author title="Editor">
28 <mail link="nightmorph"/>
29 </author>
30
31 <abstract>
32 This document serves as a HOWTO for using distcc with Gentoo.
33 </abstract>
34
35 <!-- The content of this document is licensed under the CC-BY-SA license -->
36 <!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
37 <license/>
38
39 <version>2</version>
40 <date>2012-07-24</date>
41
42 <chapter>
43 <title>Introduction</title>
44 <section>
45 <title>What is distcc?</title>
46 <body>
47
48 <p>
49 Distcc is a program designed to distribute compiling tasks across a network to
50 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
51 program, <c>distcc</c>. Distcc can work transparently with <uri
52 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
53 little setup.
54 </p>
55
56 </body>
57 </section>
58 <section>
59 <title>Using distcc to bootstrap</title>
60 <body>
61
62 <p>
63 If you are planning on using distcc to help you bootstrap a Gentoo
64 installation, make sure you read the section <uri link="#bootstrapping">Using
65 distcc to Bootstrap</uri>, which is situated further down in this document.
66 </p>
67
68 </body>
69 </section>
70 </chapter>
71
72 <chapter>
73 <title>Setup</title>
74 <section>
75 <title>Dependencies</title>
76 <body>
77
78 <p>
79 In order to use Distcc, all of the computers on your network need to have the
80 same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
81 mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
82 errors.
83 </p>
84
85 </body>
86 </section>
87 <section>
88 <title>Installing Distcc</title>
89 <body>
90
91 <p>
92 There are a couple of options you should be aware of before you start
93 installing distcc.
94 </p>
95
96 <p>
97 Distcc ships with a graphical monitor to monitor tasks that your computer is
98 sending away for compilation. If you use Gnome then put 'gnome' in your USE
99 flags. However, if you don't use Gnome and would still like to have the
100 monitor then you should put 'gtk' in your USE flags.
101 </p>
102
103 <pre caption="Installing distcc">
104 # <i>emerge distcc</i>
105 </pre>
106
107 <impo>
108 Remember, you must be sure to install distcc on all of your participating
109 machines.
110 </impo>
111
112 </body>
113 </section>
114 <section>
115 <title>Setting up Portage to use Distcc</title>
116 <body>
117
118 <p>
119 Setting up Portage to use distcc is easy. Execute the following steps on
120 each system that should participate in the distributed compiling:
121 </p>
122
123 <pre caption="Integrating Distcc and Portage">
124 # <i>emerge distcc</i>
125 # <i>nano -w /etc/portage/make.conf</i>
126 <comment>(Set N to a suitable number for your particular setup)</comment>
127 <comment>(A common strategy is setting N as twice the number of total CPUs + 1 available)</comment>
128 MAKEOPTS="-jN"
129 <comment>(Add distcc to your FEATURES)</comment>
130 FEATURES="distcc"
131 </pre>
132
133 </body>
134 </section>
135 <section>
136 <title>Specifying Participating Hosts</title>
137 <body>
138
139 <p>
140 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
141 example of some hosts that might be in your list:
142 </p>
143
144 <pre caption="Examples of host definitions">
145 192.168.0.1 192.168.0.2 192.168.0.3
146 192.168.0.1/2 192.168.0.2 192.168.0.3/10
147 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
148 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
149 <comment>(There are also several other methods of setting up hosts. See the
150 distcc manpage for more details.)</comment>
151 <comment>If you wish to compile on the local machine you should put 'localhost'
152 in the hosts list. Conversely if you do not wish to use the local machine to
153 compile (which is often the case) omit it from the hosts list. On a slow
154 machine using localhost may actually slow things down. Make sure to test your
155 settings for performance.</comment>
156 </pre>
157
158 <p>
159 It may all look complicated, but in most cases a variant of line 1 or 2 will
160 work.
161 </p>
162
163 <p>
164 Since most people won't be using lines 3 or 4, I'll <uri
165 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
166 docs (man distcc) for more information, which includes being able to run distcc
167 over an SSH connection.
168 </p>
169
170 <p>
171 For instance, to set the first line in the previous example:
172 </p>
173
174 <pre caption="Sample command to set the hosts">
175 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
176 </pre>
177
178 <p>
179 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
180 <c>--allow</c> directive to allow only hosts you trust. For added security,
181 you should also use the <c>--listen</c> directive to tell the distcc daemon
182 what IP to listen on (for multi-homed systems). More information on distcc
183 security can be found at <uri
184 link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
185 </p>
186
187 <impo>
188 It is important to use --allow and --listen. Please read the distccd manpage
189 or the above security document for more information.
190 </impo>
191
192
193 <p>
194 Now start the distcc daemon on all the participating computers:
195 </p>
196
197 <pre caption="Starting the distcc daemon">
198 <comment>(Add distccd to the default runlevel)</comment>
199 # <i>rc-update add distccd default</i>
200 <comment>(Start the distcc daemon)</comment>
201 # <i>/etc/init.d/distccd start</i>
202 </pre>
203
204 </body>
205 </section>
206 <section>
207 <title>Setting up Distcc to Work With Automake</title>
208 <body>
209
210 <p>
211 This is, in some cases, easier than the Portage setup. What you have to do is
212 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
213 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
214 However, there is a caveat. If you use ccache you have to put distcc after
215 the ccache part:
216 </p>
217
218 <pre caption="Setting your path">
219 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
220 <comment>(You can put this in your ~/.bashrc or equivalent file to have the PATH
221 set every time you log in)</comment>
222 </pre>
223
224 <p>
225 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
226 (where N is an integer). The value of N depends on your network and the types
227 of computers you are using to compile. Test your own settings to find the
228 number that yields the best performance.
229 </p>
230
231 </body>
232 </section>
233 </chapter>
234
235 <chapter>
236 <title>Cross-Compiling</title>
237 <section>
238 <body>
239
240 <p>
241 Cross-compiling is using one architecture to build programs for another
242 architecture. This can be as simple as using an Athlon (i686) to build a
243 program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
244 is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
245 Cross-compiling Guide</uri>.
246 </p>
247
248 </body>
249 </section>
250 </chapter>
251
252 <chapter id="bootstrapping">
253 <title>Using Distcc to Bootstrap</title>
254 <section>
255 <title>Step 1: Configure Portage</title>
256 <body>
257 <!--
258 Note that this will need to become a link to the old 2005.1 networked handbooks
259 once the 2006.0 handbooks are released. Until then, the link can stay.
260 -->
261 <p>
262 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
263 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
264 up until the bootstrapping part. (See the <uri
265 link="/doc/en/faq.xml#stage12">Gentoo FAQ</uri> for more information about
266 bootstrapping.) Then configure Portage to use distcc:
267 </p>
268
269 <pre caption="Preliminary Setup">
270 # <i>nano -w /etc/portage/make.conf</i>
271 <comment>(Add distcc to the FEATURES</comment>
272 FEATURES="distcc"
273 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
274 +1 available)</comment>
275 MAKEOPTS="-jN"
276 </pre>
277
278 <pre caption="Setting your path">
279 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
280 </pre>
281
282 </body>
283 </section>
284 <section>
285 <title>Step 2: Getting Distcc</title>
286 <body>
287
288 <p>
289 Install distcc:
290 </p>
291
292 <pre caption="Getting Distcc on the new box">
293 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
294 </pre>
295
296 </body>
297 </section>
298 <section>
299 <title>Step 3: Setting Up Distcc</title>
300 <body>
301
302 <p>
303 Run <c>distcc-config --install</c> to setup distcc:
304 </p>
305
306 <pre caption="Final distcc setup">
307 <comment>(Substitute host1, host2, ... with the IP number(s) of the
308 participating hosts)</comment>
309 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
310 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
311 192.168.0.4 192.168.0.6"</i></comment>
312 </pre>
313
314 <p>
315 Distcc is now set up to bootstrap! Continue with the official installation
316 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
317 system</c>. This is to make sure that all of the dependencies you want are
318 installed as well.
319 </p>
320
321 <note>
322 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
323 This is expected as some ebuilds do not work well with distcc, so they
324 intentionally disable it.
325 </note>
326
327 </body>
328 </section>
329 </chapter>
330
331 <chapter>
332 <title>Troubleshooting</title>
333 <section>
334 <title>Some Packages Don't Use Distcc</title>
335 <body>
336
337 <p>
338 As you emerge various packages, you'll notice that some of them aren't being
339 distributed (and aren't being built in parallel). This may happen because the
340 package's Makefile doesn't support parallel operations or the maintainer of the
341 ebuild has explicitly disabled parallel operations due to a known problem.
342 </p>
343
344 <p>
345 Sometimes distcc might cause a package to fail to compile. If this happens
346 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
347 </p>
348
349 </body>
350 </section>
351 <section>
352 <title>Mixed GCC Versions</title>
353 <body>
354
355 <p>
356 If you have different GCC versions on your hosts, there will likely be very
357 weird problems. The solution is to make certain all hosts have the same GCC
358 version.
359 </p>
360
361 <p>
362 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
363 <c>gcc</c>. This means that if you're mixing i686 machines with other types
364 (i386, i586) you will run into problems. A workaround for this may be to
365 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/portage/make.conf</path>.
366 </p>
367
368 <impo>
369 Doing this explicitly redefines some behaviour of Portage and may have some
370 weird results in the future. Only do this if you're mixing CHOSTs.
371 </impo>
372
373
374 </body>
375 </section>
376 <section>
377 <title>-march=native</title>
378 <body>
379
380 <p>
381 Starting with GCC 4.3.0, the compiler supports the <c>-march=native</c> switch
382 which turns on CPU autodetection and optimizations that are worth being enabled
383 on the processor the GCC is running at. This is a problem with <c>distcc</c> as
384 it allows mixing of code optimized for different processors (like AMD Athlon and
385 Intel Pentium). <e>Don't</e> use <c>-march=native</c> or <c>-mtune=native</c> in
386 your <c>CFLAGS</c> or <c>CXXFLAGS</c> when compiling with <c>distcc</c>.
387 </p>
388
389 </body>
390 </section>
391 </chapter>
392
393 <chapter>
394 <title>Distcc Extras</title>
395 <section>
396 <title>Distcc Monitors</title>
397 <body>
398
399 <p>
400 Distcc ships with two monitors. The text-based one is always built and is
401 called <c>distccmon-text</c>. Running it for the first time can be a bit
402 confusing, but it is really quite easy to use. If you run the program with no
403 parameter it will run once. However, if you pass it a number it will update
404 every N seconds, where N is the argument you passed.
405 </p>
406
407 <p>
408 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
409 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
410 and it is quite lovely. For Gentoo the GUI monitor has been called
411 <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
412 <c>distccmon-gnome</c>.
413 </p>
414
415 <pre caption="Starting the monitors">
416 # <i>distccmon-text N</i>
417 <comment>(Or)</comment>
418 # <i>distccmon-gui</i>
419 <comment>To monitor Portage's distcc usage you can use:</comment>
420 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
421 <comment>(Or)</comment>
422 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
423 </pre>
424
425 <impo>
426 If your distcc directory is elsewhere, change the DISTCC_DIR variable
427 accordingly.
428 </impo>
429
430 </body>
431 </section>
432 </chapter>
433 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20