/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.32 - (hide annotations) (download) (as text)
Wed Jan 25 09:16:29 2006 UTC (8 years, 5 months ago) by neysx
Branch: MAIN
Changes since 1.31: +8 -24 lines
File MIME type: application/xml
link to cross-compiling-distcc.xml

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 neysx 1.32 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.31 2006/01/01 11:51:43 neysx Exp $ -->
4 blubber 1.2
5 swift 1.1 <guide link="/doc/en/distcc.xml">
6    
7     <title>Gentoo Distcc Documentation</title>
8    
9     <author title="Author">
10 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 erwin 1.4 </author>
12     <author title="Editor">
13 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 erwin 1.4 </author>
15     <author title="Editor">
16 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
17 blubber 1.2 </author>
18 swift 1.17 <author title="Editor">
19 swift 1.10 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 swift 1.1 </author>
21 pylon 1.20 <author title="Editor">
22     <mail link="pylon@gentoo.org">Lars Weiler</mail>
23     </author>
24 swift 1.17 <author title="Reviewer">
25 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
26 swift 1.1 </author>
27    
28 swift 1.15 <abstract>
29     This document serves as a HOWTO for using distcc with Gentoo.
30     </abstract>
31    
32 swift 1.17 <!-- The content of this document is licensed under the CC-BY-SA license -->
33     <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
34 swift 1.6 <license/>
35    
36 neysx 1.32 <version>1.13</version>
37     <date>2006-01-25</date>
38 swift 1.1
39     <chapter>
40 swift 1.10 <title>Introduction</title>
41     <section>
42     <title>What is distcc?</title>
43     <body>
44    
45     <p>
46     Distcc is a program designed to distribute compiling tasks across a network to
47 smithj 1.27 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
48 swift 1.10 program, <c>distcc</c>. Distcc can work transparently with <uri
49 smithj 1.28 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
50     little setup.
51 swift 1.17 </p>
52    
53     </body>
54     </section>
55     <section>
56     <title>Using distcc to bootstrap</title>
57     <body>
58    
59     <p>
60 smithj 1.28 If you are planning on using distcc to help you bootstrap a Gentoo
61     installation, make sure you read the section <uri link="#bootstrapping">Using
62     distcc to Bootstrap</uri>, which is situated further down in this document.
63 swift 1.10 </p>
64    
65     </body>
66     </section>
67 swift 1.8 </chapter>
68    
69     <chapter>
70 swift 1.10 <title>Setup</title>
71     <section>
72     <title>Dependencies</title>
73     <body>
74    
75 swift 1.17 <p>
76 smithj 1.28 In order to use Distcc, all of the computers on your network need to have the
77     same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
78     mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
79     errors.
80 swift 1.17 </p>
81 swift 1.10
82     </body>
83     </section>
84     <section>
85     <title>Installing Distcc</title>
86     <body>
87    
88     <p>
89 swift 1.25 There are a couple of options you should be aware of before you start
90 swift 1.17 installing distcc.
91 swift 1.10 </p>
92    
93     <p>
94     Distcc ships with a graphical monitor to monitor tasks that your computer is
95     sending away for compilation. If you use Gnome then put 'gnome' in your USE
96     flags. However, if you don't use Gnome and would still like to have the
97     monitor then you should put 'gtk' in your USE flags.
98     </p>
99    
100 swift 1.17 <pre caption="Installing distcc">
101     # <i>emerge distcc</i>
102     </pre>
103    
104 swift 1.10 </body>
105     </section>
106     <section>
107     <title>Setting up Portage to use Distcc</title>
108     <body>
109    
110     <p>
111 swift 1.25 Setting up Portage to use distcc is easy. Execute the following steps on
112 swift 1.17 each system that should participate in the distributed compiling:
113 swift 1.10 </p>
114    
115     <pre caption="Integrating Distcc and Portage">
116 swift 1.1 # <i>emerge distcc</i>
117     # <i>nano -w /etc/make.conf</i>
118 swift 1.17 <comment>(Set N to a suitable number for your particular setup)</comment>
119 smithj 1.28 <comment>(A common strategy is setting N as twice the number of total CPUs + 1
120     available)</comment>
121 swift 1.17 MAKEOPTS="-jN"
122 smithj 1.28 <comment>(Add distcc to your FEATURES)</comment>
123 swift 1.17 FEATURES="distcc"
124 swift 1.5 </pre>
125 swift 1.1
126 swift 1.25 </body>
127     </section>
128     <section>
129     <title>Specifying Participating Hosts</title>
130     <body>
131    
132 swift 1.10 <p>
133 smithj 1.28 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
134     example of some hosts that might be in your list:
135 swift 1.10 </p>
136    
137     <pre caption="Examples of host definitions">
138 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
139     192.168.0.1/2 192.168.0.2 192.168.0.3/10
140     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
141     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
142 swift 1.17 <comment>(There are also several other methods of setting up hosts. See the
143     distcc manpage for more details.)</comment>
144 smithj 1.28 <comment>If you wish to compile on the local machine you should put 'localhost'
145     in the hosts list. Conversely if you do not wish to use the local machine to
146     compile (which is often the case) omit it from the hosts list. On a slow
147     machine using localhost may actually slow things down. Make sure to test your
148     settings for performance.</comment>
149 swift 1.8 </pre>
150 swift 1.10
151     <p>
152     It may all look complicated, but in most cases a variant of line 1 or 2 will
153 swift 1.17 work.
154 swift 1.10 </p>
155    
156     <p>
157 swift 1.17 Since most people won't be using lines 3 or 4, I'll <uri
158 smithj 1.28 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
159     docs (man distcc) for more information.
160 swift 1.17 </p>
161    
162     <p>
163     For instance, to set the first line in the previous example:
164 swift 1.10 </p>
165    
166     <pre caption="Sample command to set the hosts">
167 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 swift 1.8 </pre>
169 swift 1.1
170 swift 1.10 <p>
171 swift 1.23 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
172 smithj 1.28 <c>--allow</c> directive to allow only hosts you trust. For added security,
173     you should also use the <c>--listen</c> directive to tell the distcc daemon
174     what IP to listen on (for multi-homed systems). More information on distcc
175     security can be found at <uri
176     link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
177 swift 1.23 </p>
178    
179 smithj 1.28 <impo>
180     It is important to use --allow and --listen. Please read the distccd manpage
181 alin 1.30 or the above security document for more information.
182 smithj 1.28 </impo>
183    
184    
185 swift 1.23 <p>
186 swift 1.17 Now start the distcc daemon on all the participating computers:
187 swift 1.10 </p>
188    
189 swift 1.17 <pre caption="Starting the distcc daemon">
190     <comment>(Add distccd to the default runlevel)</comment>
191     # <i>rc-update add distccd default</i>
192     <comment>(Start the distcc daemon)</comment>
193     # <i>/etc/init.d/distccd start</i>
194 swift 1.8 </pre>
195 swift 1.10
196     </body>
197     </section>
198     <section>
199     <title>Setting up Distcc to Work With Automake</title>
200     <body>
201    
202     <p>
203 swift 1.25 This is, in some cases, easier than the Portage setup. What you have to do is
204 swift 1.17 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
205     in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
206 swift 1.10 However, there is a caveat. If you use ccache you have to put distcc after
207 swift 1.17 the ccache part:
208 swift 1.10 </p>
209    
210     <pre caption="Setting your path">
211 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
212 smithj 1.28 <comment>You can put this in your .bashrc or equivelant file to have the PATH
213     set every time you log in</comment>
214 swift 1.8 </pre>
215 swift 1.10
216     <p>
217 neysx 1.22 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
218     (where N is an integer). The value of N depends on your network and the types
219 smithj 1.28 of computers you are using to compile. Test your own settings to find the
220     number that yields the best performance.
221 swift 1.10 </p>
222    
223     </body>
224     </section>
225 blubber 1.2 </chapter>
226 swift 1.1
227 blubber 1.2 <chapter>
228 swift 1.17 <title>Cross-Compiling</title>
229     <section>
230 swift 1.10 <body>
231    
232     <p>
233 neysx 1.32 Cross-compiling is using one architecture to build programs for another
234     architecture. This can be as simple as using an Athlon (i686) to build a
235     program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
236     is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
237     Cross-compiling Guide</uri>.
238 swift 1.17 </p>
239    
240     </body>
241     </section>
242     </chapter>
243    
244     <chapter id="bootstrapping">
245     <title>Using Distcc to Bootstrap</title>
246     <section>
247 swift 1.24 <title>Step 1: Configure Portage</title>
248 swift 1.17 <body>
249 swift 1.10
250     <p>
251 swift 1.17 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
252     link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
253     up until the bootstrapping part. Then configure Portage to use distcc:
254 swift 1.10 </p>
255    
256 swift 1.17 <pre caption="Preliminary Setup">
257     # <i>nano -w /etc/make.conf</i>
258 smithj 1.28 <comment>(Add distcc to the FEATURES</comment>
259 swift 1.17 FEATURES="distcc"
260 smithj 1.28 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
261     +1 available)</comment>
262 swift 1.17 MAKEOPTS="-jN"
263 swift 1.8 </pre>
264 swift 1.10
265 neysx 1.22 <pre caption="Setting your path">
266     # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
267     </pre>
268    
269 swift 1.10 </body>
270     </section>
271     <section>
272     <title>Step 2: Getting Distcc</title>
273     <body>
274    
275 neysx 1.22 <p>
276     Before the installation of distcc, a user called distcc must be added to the
277     <path>/etc/passwd</path>:
278     </p>
279 pylon 1.20
280     <pre caption="Create user distcc">
281 smithj 1.28 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
282 pylon 1.20 </pre>
283    
284 swift 1.24 <impo>
285     It is important to note that adding users like this is very bad. We only do
286     it here because there is no <c>useradd</c> utility (which you normally use for
287     adding users) yet at this stage of installation.
288     </impo>
289    
290 swift 1.10 <p>
291 swift 1.17 Install distcc:
292 swift 1.10 </p>
293    
294     <pre caption="Getting Distcc on the new box">
295 swift 1.24 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
296 swift 1.8 </pre>
297 swift 1.10
298     </body>
299     </section>
300     <section>
301 swift 1.17 <title>Step 3: Setting Up Distcc</title>
302 swift 1.10 <body>
303    
304     <p>
305 swift 1.24 Run <c>distcc-config --install</c> to setup distcc:
306 swift 1.10 </p>
307    
308     <pre caption="Final distcc setup">
309 smithj 1.28 <comment>(Substitute host1, host2, ... with the IP number(s) of the
310     participating hosts)</comment>
311 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
312 smithj 1.28 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
313     192.168.0.4 192.168.0.6"</i></comment>
314 neysx 1.31 </pre>
315 swift 1.10
316     <p>
317 swift 1.25 Distcc is now set up to bootstrap! Continue with the official installation
318 swift 1.24 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
319 swift 1.17 system</c>. This is to make sure that all of the dependencies you want are
320     installed as well.
321 swift 1.10 </p>
322    
323 swift 1.12 <note>
324 swift 1.25 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
325 swift 1.17 This is expected as some ebuilds do not work well with distcc, so they
326     intentionally disable it.
327 swift 1.12 </note>
328    
329 swift 1.10 </body>
330     </section>
331 blubber 1.2 </chapter>
332 swift 1.1
333 swift 1.5 <chapter>
334 swift 1.10 <title>Troubleshooting</title>
335     <section>
336     <title>Mozilla and Xfree</title>
337     <body>
338    
339     <p>
340 swift 1.17 As you emerge various packages, you'll notice that some of them aren't being
341     distributed (and aren't being built in parallel). This is because the
342     developers of the Mozilla and Xfree ebuilds intentionally disable parallel
343     building because it is known to cause problems.
344 swift 1.10 </p>
345    
346     <p>
347 swift 1.17 Sometimes distcc might cause a package to fail to compile. If this happens
348     for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
349 swift 1.10 </p>
350    
351     </body>
352     </section>
353     <section>
354     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
355     <body>
356    
357     <p>
358     With such a long title any explanation here is almost irrelevent. However, if
359     you plan to use distcc across hosts that have the <uri
360 swift 1.17 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
361     not, you will run into problems.
362 swift 1.10 </p>
363    
364     <p>
365     The solution requires a little foresight on your part; you have to run
366     <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
367     enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
368     which are a good thing to do since for the most part the protections offered
369     by both packages is a good thing and is transparent to the user.
370     </p>
371    
372     </body>
373     </section>
374     <section>
375     <title>Mixed GCC Versions</title>
376     <body>
377    
378     <p>
379 swift 1.17 If you have different GCC versions on your hosts, there will likely be very
380 swift 1.10 weird problems. The solution is to make certain all hosts have the same GCC
381     version.
382     </p>
383    
384 swift 1.24 <p>
385     Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
386     <c>gcc</c>. This means that if you're mixing i686 machines with other types
387     (i386, i586) you will run into problems. A workaround for this may be to
388 smithj 1.28 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
389 swift 1.24 </p>
390    
391 smithj 1.28 <impo>
392     Doing this explicitly redefines some behaviour of Portage and may have some
393     weird results in the future. Only do this if you're mixing CHOSTs.
394     </impo>
395    
396    
397 swift 1.10 </body>
398     </section>
399 swift 1.5 </chapter>
400 swift 1.1
401     <chapter>
402 swift 1.10 <title>Distcc Extras</title>
403     <section>
404     <title>Distcc Monitors</title>
405     <body>
406    
407     <p>
408 swift 1.17 Distcc ships with two monitors. The text-based one is always built and is
409 swift 1.10 called <c>distccmon-text</c>. Running it for the first time can be a bit
410     confusing, but it is really quite easy to use. If you run the program with no
411     parameter it will run once. However, if you pass it a number it will update
412     every N seconds, where N is the argument you passed.
413     </p>
414    
415     <p>
416 smithj 1.28 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
417     in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
418     and it is quite lovely. For Gentoo the GUI monitor has been called
419     <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
420     <c>distccmon-gnome</c>.
421 swift 1.10 </p>
422    
423     <pre caption="Starting the monitors">
424 swift 1.17 # <i>distccmon-text N</i>
425 neysx 1.22 <comment>(Or)</comment>
426 smithj 1.28 # <i>distccmon-gui</i>
427 neysx 1.22 <comment>To monitor Portage's distcc usage you can use:</comment>
428 swift 1.26 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
429 neysx 1.22 <comment>(Or)</comment>
430 smithj 1.28 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
431 swift 1.8 </pre>
432 swift 1.10
433 neysx 1.22 <impo>
434 swift 1.26 If your distcc directory is elsewhere, change the DISTCC_DIR variable
435     accordingly.
436 neysx 1.22 </impo>
437 swift 1.10
438     </body>
439     </section>
440 swift 1.1 </chapter>
441     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20