/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.46 - (hide annotations) (download) (as text)
Wed Jul 24 20:03:06 2013 UTC (13 months, 3 weeks ago) by swift
Branch: MAIN
CVS Tags: HEAD
Changes since 1.45: +4 -4 lines
File MIME type: application/xml
Distcc guide is now at https://wiki.gentoo.org/wiki/Distcc

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 swift 1.46 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.45 2012/07/24 12:12:51 swift Exp $ -->
4 blubber 1.2
5 swift 1.46 <guide disclaimer="obsolete" redirect="https://wiki.gentoo.org/wiki/Distcc">
6 swift 1.1
7     <title>Gentoo Distcc Documentation</title>
8    
9     <author title="Author">
10 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 erwin 1.4 </author>
12     <author title="Editor">
13 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 erwin 1.4 </author>
15     <author title="Editor">
16 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
17 blubber 1.2 </author>
18 swift 1.17 <author title="Editor">
19 nightmorph 1.38 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 swift 1.1 </author>
21 pylon 1.20 <author title="Editor">
22     <mail link="pylon@gentoo.org">Lars Weiler</mail>
23     </author>
24 swift 1.17 <author title="Reviewer">
25 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
26 swift 1.1 </author>
27 nightmorph 1.43 <author title="Editor">
28     <mail link="nightmorph"/>
29     </author>
30 swift 1.1
31 swift 1.15 <abstract>
32     This document serves as a HOWTO for using distcc with Gentoo.
33     </abstract>
34    
35 swift 1.17 <!-- The content of this document is licensed under the CC-BY-SA license -->
36 jkt 1.34 <!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
37 swift 1.6 <license/>
38    
39 swift 1.46 <version>3</version>
40     <date>2013-07-24</date>
41 swift 1.1
42     <chapter>
43 swift 1.10 <title>Introduction</title>
44     <section>
45     <title>What is distcc?</title>
46     <body>
47    
48     <p>
49     Distcc is a program designed to distribute compiling tasks across a network to
50 smithj 1.27 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
51 swift 1.10 program, <c>distcc</c>. Distcc can work transparently with <uri
52 smithj 1.28 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
53     little setup.
54 swift 1.17 </p>
55    
56     </body>
57     </section>
58     <section>
59     <title>Using distcc to bootstrap</title>
60     <body>
61    
62     <p>
63 smithj 1.28 If you are planning on using distcc to help you bootstrap a Gentoo
64     installation, make sure you read the section <uri link="#bootstrapping">Using
65     distcc to Bootstrap</uri>, which is situated further down in this document.
66 swift 1.10 </p>
67    
68     </body>
69     </section>
70 swift 1.8 </chapter>
71    
72     <chapter>
73 swift 1.10 <title>Setup</title>
74     <section>
75     <title>Dependencies</title>
76     <body>
77    
78 swift 1.17 <p>
79 smithj 1.28 In order to use Distcc, all of the computers on your network need to have the
80     same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
81     mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
82     errors.
83 swift 1.17 </p>
84 swift 1.10
85     </body>
86     </section>
87     <section>
88     <title>Installing Distcc</title>
89     <body>
90    
91     <p>
92 swift 1.25 There are a couple of options you should be aware of before you start
93 swift 1.17 installing distcc.
94 swift 1.10 </p>
95    
96     <p>
97     Distcc ships with a graphical monitor to monitor tasks that your computer is
98     sending away for compilation. If you use Gnome then put 'gnome' in your USE
99     flags. However, if you don't use Gnome and would still like to have the
100     monitor then you should put 'gtk' in your USE flags.
101     </p>
102    
103 swift 1.17 <pre caption="Installing distcc">
104     # <i>emerge distcc</i>
105     </pre>
106    
107 nightmorph 1.36 <impo>
108     Remember, you must be sure to install distcc on all of your participating
109     machines.
110     </impo>
111    
112 swift 1.10 </body>
113     </section>
114     <section>
115     <title>Setting up Portage to use Distcc</title>
116     <body>
117    
118     <p>
119 swift 1.25 Setting up Portage to use distcc is easy. Execute the following steps on
120 swift 1.17 each system that should participate in the distributed compiling:
121 swift 1.10 </p>
122    
123     <pre caption="Integrating Distcc and Portage">
124 nightmorph 1.41 # <i>emerge distcc</i>
125 swift 1.45 # <i>nano -w /etc/portage/make.conf</i>
126 swift 1.17 <comment>(Set N to a suitable number for your particular setup)</comment>
127 rane 1.40 <comment>(A common strategy is setting N as twice the number of total CPUs + 1 available)</comment>
128 swift 1.17 MAKEOPTS="-jN"
129 smithj 1.28 <comment>(Add distcc to your FEATURES)</comment>
130 swift 1.17 FEATURES="distcc"
131 swift 1.5 </pre>
132 swift 1.1
133 swift 1.25 </body>
134     </section>
135     <section>
136     <title>Specifying Participating Hosts</title>
137     <body>
138    
139 swift 1.10 <p>
140 smithj 1.28 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
141     example of some hosts that might be in your list:
142 swift 1.10 </p>
143    
144     <pre caption="Examples of host definitions">
145 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
146     192.168.0.1/2 192.168.0.2 192.168.0.3/10
147     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
148     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
149 swift 1.17 <comment>(There are also several other methods of setting up hosts. See the
150     distcc manpage for more details.)</comment>
151 smithj 1.28 <comment>If you wish to compile on the local machine you should put 'localhost'
152     in the hosts list. Conversely if you do not wish to use the local machine to
153     compile (which is often the case) omit it from the hosts list. On a slow
154     machine using localhost may actually slow things down. Make sure to test your
155     settings for performance.</comment>
156 swift 1.8 </pre>
157 swift 1.10
158     <p>
159     It may all look complicated, but in most cases a variant of line 1 or 2 will
160 swift 1.17 work.
161 swift 1.10 </p>
162    
163     <p>
164 swift 1.17 Since most people won't be using lines 3 or 4, I'll <uri
165 smithj 1.28 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
166 nightmorph 1.43 docs (man distcc) for more information, which includes being able to run distcc
167     over an SSH connection.
168 swift 1.17 </p>
169    
170     <p>
171     For instance, to set the first line in the previous example:
172 swift 1.10 </p>
173    
174     <pre caption="Sample command to set the hosts">
175 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
176 swift 1.8 </pre>
177 swift 1.1
178 swift 1.10 <p>
179 swift 1.23 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
180 smithj 1.28 <c>--allow</c> directive to allow only hosts you trust. For added security,
181     you should also use the <c>--listen</c> directive to tell the distcc daemon
182     what IP to listen on (for multi-homed systems). More information on distcc
183     security can be found at <uri
184     link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
185 swift 1.23 </p>
186    
187 smithj 1.28 <impo>
188     It is important to use --allow and --listen. Please read the distccd manpage
189 alin 1.30 or the above security document for more information.
190 smithj 1.28 </impo>
191    
192    
193 swift 1.23 <p>
194 swift 1.17 Now start the distcc daemon on all the participating computers:
195 swift 1.10 </p>
196    
197 swift 1.17 <pre caption="Starting the distcc daemon">
198     <comment>(Add distccd to the default runlevel)</comment>
199     # <i>rc-update add distccd default</i>
200     <comment>(Start the distcc daemon)</comment>
201     # <i>/etc/init.d/distccd start</i>
202 swift 1.8 </pre>
203 swift 1.10
204     </body>
205     </section>
206     <section>
207     <title>Setting up Distcc to Work With Automake</title>
208     <body>
209    
210     <p>
211 swift 1.25 This is, in some cases, easier than the Portage setup. What you have to do is
212 swift 1.17 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
213     in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
214 swift 1.10 However, there is a caveat. If you use ccache you have to put distcc after
215 swift 1.17 the ccache part:
216 swift 1.10 </p>
217    
218     <pre caption="Setting your path">
219 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
220 nightmorph 1.39 <comment>(You can put this in your ~/.bashrc or equivalent file to have the PATH
221     set every time you log in)</comment>
222 swift 1.8 </pre>
223 swift 1.10
224     <p>
225 neysx 1.22 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
226     (where N is an integer). The value of N depends on your network and the types
227 smithj 1.28 of computers you are using to compile. Test your own settings to find the
228     number that yields the best performance.
229 swift 1.10 </p>
230    
231     </body>
232     </section>
233 blubber 1.2 </chapter>
234 swift 1.1
235 blubber 1.2 <chapter>
236 swift 1.17 <title>Cross-Compiling</title>
237     <section>
238 swift 1.10 <body>
239    
240     <p>
241 neysx 1.32 Cross-compiling is using one architecture to build programs for another
242     architecture. This can be as simple as using an Athlon (i686) to build a
243     program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
244     is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
245     Cross-compiling Guide</uri>.
246 swift 1.17 </p>
247    
248     </body>
249     </section>
250     </chapter>
251    
252     <chapter id="bootstrapping">
253     <title>Using Distcc to Bootstrap</title>
254     <section>
255 swift 1.24 <title>Step 1: Configure Portage</title>
256 swift 1.17 <body>
257 nightmorph 1.33 <!--
258     Note that this will need to become a link to the old 2005.1 networked handbooks
259     once the 2006.0 handbooks are released. Until then, the link can stay.
260     -->
261     <p>
262     Boot your new box with a Gentoo Linux LiveCD and follow the <uri
263     link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
264     up until the bootstrapping part. (See the <uri
265 jkt 1.34 link="/doc/en/faq.xml#stage12">Gentoo FAQ</uri> for more information about
266     bootstrapping.) Then configure Portage to use distcc:
267 swift 1.10 </p>
268    
269 swift 1.17 <pre caption="Preliminary Setup">
270 swift 1.45 # <i>nano -w /etc/portage/make.conf</i>
271 smithj 1.28 <comment>(Add distcc to the FEATURES</comment>
272 swift 1.17 FEATURES="distcc"
273 smithj 1.28 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
274     +1 available)</comment>
275 swift 1.17 MAKEOPTS="-jN"
276 swift 1.8 </pre>
277 swift 1.10
278 neysx 1.22 <pre caption="Setting your path">
279     # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
280     </pre>
281    
282 swift 1.10 </body>
283     </section>
284     <section>
285     <title>Step 2: Getting Distcc</title>
286     <body>
287    
288 neysx 1.22 <p>
289 swift 1.17 Install distcc:
290 swift 1.10 </p>
291    
292     <pre caption="Getting Distcc on the new box">
293 nightmorph 1.41 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
294 swift 1.8 </pre>
295 swift 1.10
296     </body>
297     </section>
298     <section>
299 swift 1.17 <title>Step 3: Setting Up Distcc</title>
300 swift 1.10 <body>
301    
302     <p>
303 swift 1.24 Run <c>distcc-config --install</c> to setup distcc:
304 swift 1.10 </p>
305    
306     <pre caption="Final distcc setup">
307 smithj 1.28 <comment>(Substitute host1, host2, ... with the IP number(s) of the
308     participating hosts)</comment>
309 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
310 smithj 1.28 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
311     192.168.0.4 192.168.0.6"</i></comment>
312 neysx 1.31 </pre>
313 swift 1.10
314     <p>
315 swift 1.25 Distcc is now set up to bootstrap! Continue with the official installation
316 swift 1.24 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
317 swift 1.17 system</c>. This is to make sure that all of the dependencies you want are
318     installed as well.
319 swift 1.10 </p>
320    
321 swift 1.12 <note>
322 swift 1.25 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
323 swift 1.17 This is expected as some ebuilds do not work well with distcc, so they
324     intentionally disable it.
325 swift 1.12 </note>
326    
327 swift 1.10 </body>
328     </section>
329 blubber 1.2 </chapter>
330 swift 1.1
331 swift 1.5 <chapter>
332 swift 1.10 <title>Troubleshooting</title>
333     <section>
334 fox2mike 1.35 <title>Some Packages Don't Use Distcc</title>
335 swift 1.10 <body>
336    
337     <p>
338 nightmorph 1.33 As you emerge various packages, you'll notice that some of them aren't being
339 fox2mike 1.35 distributed (and aren't being built in parallel). This may happen because the
340     package's Makefile doesn't support parallel operations or the maintainer of the
341     ebuild has explicitly disabled parallel operations due to a known problem.
342 swift 1.10 </p>
343    
344     <p>
345 swift 1.17 Sometimes distcc might cause a package to fail to compile. If this happens
346     for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
347 swift 1.10 </p>
348    
349     </body>
350     </section>
351     <section>
352     <title>Mixed GCC Versions</title>
353     <body>
354    
355     <p>
356 swift 1.17 If you have different GCC versions on your hosts, there will likely be very
357 swift 1.10 weird problems. The solution is to make certain all hosts have the same GCC
358     version.
359     </p>
360    
361 swift 1.24 <p>
362     Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
363     <c>gcc</c>. This means that if you're mixing i686 machines with other types
364     (i386, i586) you will run into problems. A workaround for this may be to
365 swift 1.45 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/portage/make.conf</path>.
366 swift 1.24 </p>
367    
368 smithj 1.28 <impo>
369     Doing this explicitly redefines some behaviour of Portage and may have some
370     weird results in the future. Only do this if you're mixing CHOSTs.
371     </impo>
372    
373    
374 swift 1.10 </body>
375     </section>
376 jkt 1.42 <section>
377     <title>-march=native</title>
378     <body>
379    
380     <p>
381     Starting with GCC 4.3.0, the compiler supports the <c>-march=native</c> switch
382     which turns on CPU autodetection and optimizations that are worth being enabled
383     on the processor the GCC is running at. This is a problem with <c>distcc</c> as
384     it allows mixing of code optimized for different processors (like AMD Athlon and
385     Intel Pentium). <e>Don't</e> use <c>-march=native</c> or <c>-mtune=native</c> in
386     your <c>CFLAGS</c> or <c>CXXFLAGS</c> when compiling with <c>distcc</c>.
387     </p>
388    
389     </body>
390     </section>
391 swift 1.5 </chapter>
392 swift 1.1
393     <chapter>
394 swift 1.10 <title>Distcc Extras</title>
395     <section>
396     <title>Distcc Monitors</title>
397     <body>
398    
399     <p>
400 swift 1.17 Distcc ships with two monitors. The text-based one is always built and is
401 swift 1.10 called <c>distccmon-text</c>. Running it for the first time can be a bit
402     confusing, but it is really quite easy to use. If you run the program with no
403     parameter it will run once. However, if you pass it a number it will update
404     every N seconds, where N is the argument you passed.
405     </p>
406    
407     <p>
408 smithj 1.28 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
409     in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
410     and it is quite lovely. For Gentoo the GUI monitor has been called
411     <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
412     <c>distccmon-gnome</c>.
413 swift 1.10 </p>
414    
415     <pre caption="Starting the monitors">
416 swift 1.17 # <i>distccmon-text N</i>
417 neysx 1.22 <comment>(Or)</comment>
418 smithj 1.28 # <i>distccmon-gui</i>
419 neysx 1.22 <comment>To monitor Portage's distcc usage you can use:</comment>
420 swift 1.26 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
421 neysx 1.22 <comment>(Or)</comment>
422 smithj 1.28 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
423 swift 1.8 </pre>
424 swift 1.10
425 neysx 1.22 <impo>
426 swift 1.26 If your distcc directory is elsewhere, change the DISTCC_DIR variable
427     accordingly.
428 neysx 1.22 </impo>
429 swift 1.10
430     </body>
431     </section>
432 swift 1.1 </chapter>
433     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20