/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.24 - (hide annotations) (download) (as text)
Wed Dec 22 22:43:45 2004 UTC (9 years, 9 months ago) by swift
Branch: MAIN
Changes since 1.23: +28 -11 lines
File MIME type: application/xml
#75369 - various distcc updates

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 swift 1.24 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.23 2004/11/09 11:37:30 swift Exp $ -->
4 blubber 1.2
5 swift 1.1 <guide link="/doc/en/distcc.xml">
6    
7     <title>Gentoo Distcc Documentation</title>
8    
9     <author title="Author">
10 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 erwin 1.4 </author>
12 swift 1.17
13 erwin 1.4 <author title="Editor">
14 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15 erwin 1.4 </author>
16 swift 1.17
17 erwin 1.4 <author title="Editor">
18 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
19 blubber 1.2 </author>
20 swift 1.17
21     <author title="Editor">
22 swift 1.10 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 swift 1.1 </author>
24 swift 1.17
25 pylon 1.20 <author title="Editor">
26     <mail link="pylon@gentoo.org">Lars Weiler</mail>
27     </author>
28    
29 swift 1.17 <author title="Reviewer">
30 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
31 swift 1.1 </author>
32    
33 swift 1.15 <abstract>
34     This document serves as a HOWTO for using distcc with Gentoo.
35     </abstract>
36    
37 swift 1.17 <!-- The content of this document is licensed under the CC-BY-SA license -->
38     <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
39 swift 1.6 <license/>
40    
41 swift 1.24 <version>1.10</version>
42     <date>December 22, 2004</date>
43 swift 1.1
44     <chapter>
45 swift 1.10 <title>Introduction</title>
46     <section>
47     <title>What is distcc?</title>
48     <body>
49    
50     <p>
51     Distcc is a program designed to distribute compiling tasks across a network to
52     participating hosts. It is comprised of a server, <c>distccd</c> and a client
53     program, <c>distcc</c>. Distcc can work transparently with <uri
54 swift 1.17 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
55     </p>
56    
57     </body>
58     </section>
59     <section>
60     <title>Using distcc to bootstrap</title>
61     <body>
62    
63     <p>
64     If you are planning on using distcc to help you bootstrap a Gentoo installation,
65     make sure you read the part on <uri link="#bootstrapping">Using distcc to
66     Bootstrap</uri> which is situated further down in this document.
67 swift 1.10 </p>
68    
69     </body>
70     </section>
71 swift 1.8 </chapter>
72    
73     <chapter>
74 swift 1.10 <title>Setup</title>
75     <section>
76     <title>Dependencies</title>
77     <body>
78    
79 swift 1.17 <p>
80     To start off with, to use Distcc all of the computers on your network need to
81     have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
82     mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
83     </p>
84 swift 1.10
85     </body>
86     </section>
87     <section>
88     <title>Installing Distcc</title>
89     <body>
90    
91     <p>
92 swift 1.17 There are a couple of options you should know about before you start
93     installing distcc.
94 swift 1.10 </p>
95    
96     <p>
97     Distcc ships with a graphical monitor to monitor tasks that your computer is
98     sending away for compilation. If you use Gnome then put 'gnome' in your USE
99     flags. However, if you don't use Gnome and would still like to have the
100     monitor then you should put 'gtk' in your USE flags.
101     </p>
102    
103 swift 1.17 <pre caption="Installing distcc">
104     # <i>emerge distcc</i>
105     </pre>
106    
107 swift 1.10 </body>
108     </section>
109     <section>
110     <title>Setting up Portage to use Distcc</title>
111     <body>
112    
113     <p>
114 swift 1.17 Setting up distcc is easy to do with Portage. Execute the following steps on
115     each system that should participate in the distributed compiling:
116 swift 1.10 </p>
117    
118     <pre caption="Integrating Distcc and Portage">
119 swift 1.1 # <i>emerge distcc</i>
120     # <i>nano -w /etc/make.conf</i>
121 swift 1.17 <comment>(Set N to a suitable number for your particular setup)</comment>
122 neysx 1.22 <comment>(A common heuristic strategy is twice the number of total CPUs + 1)</comment>
123 swift 1.17 MAKEOPTS="-jN"
124     <comment>(Add distcc to your FEATURES variable)</comment>
125     FEATURES="distcc"
126 swift 1.5 </pre>
127 swift 1.1
128 swift 1.10 <p>
129 swift 1.17 Next you have to specify what hosts you want to use. Use the
130     <c>distcc-config</c> command to set the list of hosts. Here is an example of
131     some hosts that might be in your list:
132 swift 1.10 </p>
133    
134     <pre caption="Examples of host definitions">
135 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
136     192.168.0.1/2 192.168.0.2 192.168.0.3/10
137     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 swift 1.17 <comment>(There are also several other methods of setting up hosts. See the
140     distcc manpage for more details.)</comment>
141 swift 1.8 </pre>
142 swift 1.10
143     <p>
144     It may all look complicated, but in most cases a variant of line 1 or 2 will
145 swift 1.17 work.
146 swift 1.10 </p>
147    
148 swift 1.17 <ul>
149     <li>Line 1 is just a space-delimited list of hosts</li>
150     <li>
151     Line 2 is a list of hosts that specifies the maximum amount of jobs (by
152     defining the <c>/N</c> setting) to send that host at any given time
153     </li>
154     </ul>
155    
156 swift 1.10 <p>
157 swift 1.17 Since most people won't be using lines 3 or 4, I'll <uri
158     link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
159     docs for more information.
160     </p>
161    
162     <p>
163     For instance, to set the first line in the previous example:
164 swift 1.10 </p>
165    
166     <pre caption="Sample command to set the hosts">
167 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
168 swift 1.8 </pre>
169 swift 1.1
170 swift 1.10 <p>
171 swift 1.23 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
172     <c>--allow</c> directive to allow only hosts you trust. For more information
173     please see the <uri link="http://distcc.samba.org/security.html">Distcc Security
174 swift 1.24 Design</uri>. For added security you should also use the <c>--listen</c>
175     directive to tell the Distcc daemon what IP to listen on (for multi-homed
176     systems).
177 swift 1.23 </p>
178    
179     <p>
180 swift 1.17 Now start the distcc daemon on all the participating computers:
181 swift 1.10 </p>
182    
183 swift 1.17 <pre caption="Starting the distcc daemon">
184     <comment>(Add distccd to the default runlevel)</comment>
185     # <i>rc-update add distccd default</i>
186     <comment>(Start the distcc daemon)</comment>
187     # <i>/etc/init.d/distccd start</i>
188 swift 1.8 </pre>
189 swift 1.10
190     </body>
191     </section>
192     <section>
193     <title>Setting up Distcc to Work With Automake</title>
194     <body>
195    
196     <p>
197     This is in some cases easier than the Portage setup. What you have to do is
198 swift 1.17 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
199     in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
200 swift 1.10 However, there is a caveat. If you use ccache you have to put distcc after
201 swift 1.17 the ccache part:
202 swift 1.10 </p>
203    
204     <pre caption="Setting your path">
205 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
206 swift 1.8 </pre>
207 swift 1.10
208     <p>
209 neysx 1.22 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
210     (where N is an integer). The value of N depends on your network and the types
211     of computers you are using to compile. For me, I have a dual processor P3 and
212     a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
213     Thunderbird) and I use -j5. Test your own settings to find the number that
214     yields the best performance.
215 swift 1.10 </p>
216    
217     </body>
218     </section>
219 blubber 1.2 </chapter>
220 swift 1.1
221 blubber 1.2 <chapter>
222 swift 1.17 <title>Cross-Compiling</title>
223     <section>
224     <title>A Note on Cross-Compiling</title>
225     <body>
226    
227     <p>
228     Cross-compiling is using one architecture to build programs for another
229     architecture. This can be as simple as using an Athlon (i686) to build a
230     program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
231     </p>
232    
233     </body>
234     </section>
235 swift 1.10 <section>
236 neysx 1.22 <title>An Introduction to Cross-compiling</title>
237 swift 1.10 <body>
238    
239     <p>
240 swift 1.24 If you want to give Cross-compiling a try you can to follow <uri
241     link="http://dev.gentoo.org/~vapier/CROSS-COMPILE-HOWTO">The Cross Compile
242     HOWTO</uri>; Crossdev is depreciated.
243 swift 1.17 </p>
244    
245     </body>
246     </section>
247     </chapter>
248    
249     <chapter id="bootstrapping">
250     <title>Using Distcc to Bootstrap</title>
251     <section>
252 swift 1.24 <title>Step 1: Configure Portage</title>
253 swift 1.17 <body>
254 swift 1.10
255     <p>
256 swift 1.17 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
257     link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
258     up until the bootstrapping part. Then configure Portage to use distcc:
259 swift 1.10 </p>
260    
261 swift 1.17 <pre caption="Preliminary Setup">
262     # <i>nano -w /etc/make.conf</i>
263     <comment>(Add distcc to the FEATURES variable)</comment>
264     FEATURES="distcc"
265     <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
266     MAKEOPTS="-jN"
267 swift 1.8 </pre>
268 swift 1.10
269 neysx 1.22 <pre caption="Setting your path">
270     # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
271     </pre>
272    
273 swift 1.10 </body>
274     </section>
275     <section>
276     <title>Step 2: Getting Distcc</title>
277     <body>
278    
279 neysx 1.22 <p>
280     Before the installation of distcc, a user called distcc must be added to the
281     <path>/etc/passwd</path>:
282     </p>
283 pylon 1.20
284     <pre caption="Create user distcc">
285     # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
286     </pre>
287    
288 swift 1.24 <impo>
289     It is important to note that adding users like this is very bad. We only do
290     it here because there is no <c>useradd</c> utility (which you normally use for
291     adding users) yet at this stage of installation.
292     </impo>
293    
294 swift 1.10 <p>
295 swift 1.17 Install distcc:
296 swift 1.10 </p>
297    
298     <pre caption="Getting Distcc on the new box">
299 swift 1.24 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
300 swift 1.8 </pre>
301 swift 1.10
302     </body>
303     </section>
304     <section>
305 swift 1.17 <title>Step 3: Setting Up Distcc</title>
306 swift 1.10 <body>
307    
308     <p>
309 swift 1.24 Run <c>distcc-config --install</c> to setup distcc:
310 swift 1.10 </p>
311    
312     <pre caption="Final distcc setup">
313 swift 1.17 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
314 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
315 neysx 1.22 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"</i></comment>
316 swift 1.17 </pre>
317 swift 1.10
318     <p>
319 swift 1.17 distcc is now set up to bootstrap! Continue the official installation
320 swift 1.24 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
321 swift 1.17 system</c>. This is to make sure that all of the dependencies you want are
322     installed as well.
323 swift 1.10 </p>
324    
325 swift 1.12 <note>
326 swift 1.17 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
327     This is expected as some ebuilds do not work well with distcc, so they
328     intentionally disable it.
329 swift 1.12 </note>
330    
331 swift 1.10 </body>
332     </section>
333 blubber 1.2 </chapter>
334 swift 1.1
335 swift 1.5 <chapter>
336 swift 1.10 <title>Troubleshooting</title>
337     <section>
338     <title>Mozilla and Xfree</title>
339     <body>
340    
341     <p>
342 swift 1.17 As you emerge various packages, you'll notice that some of them aren't being
343     distributed (and aren't being built in parallel). This is because the
344     developers of the Mozilla and Xfree ebuilds intentionally disable parallel
345     building because it is known to cause problems.
346 swift 1.10 </p>
347    
348     <p>
349 swift 1.17 Sometimes distcc might cause a package to fail to compile. If this happens
350     for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
351 swift 1.10 </p>
352    
353     </body>
354     </section>
355     <section>
356     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
357     <body>
358    
359     <p>
360     With such a long title any explanation here is almost irrelevent. However, if
361     you plan to use distcc across hosts that have the <uri
362 swift 1.17 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
363     not, you will run into problems.
364 swift 1.10 </p>
365    
366     <p>
367     The solution requires a little foresight on your part; you have to run
368     <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
369     enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
370     which are a good thing to do since for the most part the protections offered
371     by both packages is a good thing and is transparent to the user.
372     </p>
373    
374     </body>
375     </section>
376     <section>
377     <title>Mixed GCC Versions</title>
378     <body>
379    
380     <p>
381 swift 1.17 If you have different GCC versions on your hosts, there will likely be very
382 swift 1.10 weird problems. The solution is to make certain all hosts have the same GCC
383     version.
384     </p>
385    
386 swift 1.24 <p>
387     Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
388     <c>gcc</c>. This means that if you're mixing i686 machines with other types
389     (i386, i586) you will run into problems. A workaround for this may be to
390     <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>,
391     but that needs testing. If it works for you please email me,
392     <mail link="lisa@gentoo.org">Lisa Seelye</mail>.
393     </p>
394    
395 swift 1.10 </body>
396     </section>
397 swift 1.5 </chapter>
398 swift 1.1
399     <chapter>
400 swift 1.10 <title>Distcc Extras</title>
401     <section>
402     <title>Distcc Monitors</title>
403     <body>
404    
405     <p>
406 swift 1.17 Distcc ships with two monitors. The text-based one is always built and is
407 swift 1.10 called <c>distccmon-text</c>. Running it for the first time can be a bit
408     confusing, but it is really quite easy to use. If you run the program with no
409     parameter it will run once. However, if you pass it a number it will update
410     every N seconds, where N is the argument you passed.
411     </p>
412    
413     <p>
414     The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
415 swift 1.17 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
416 swift 1.10 and it is quite lovely.
417     </p>
418    
419     <pre caption="Starting the monitors">
420 swift 1.17 # <i>distccmon-text N</i>
421 neysx 1.22 <comment>(Or)</comment>
422 swift 1.17 # <i>distccmon-gnome</i>
423 neysx 1.22 <comment>To monitor Portage's distcc usage you can use:</comment>
424     # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-text N</i>
425     <comment>(Or)</comment>
426     # <i>DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-gnome</i>
427 swift 1.8 </pre>
428 swift 1.10
429 neysx 1.22 <impo>
430     Notice the backticks above. <c>portageq</c> is a part of Portage that will
431     return a piece of information (in this case what DISTCC_DIR is set to). For
432     simplicity, most users can use <c>DISTCC_DIR=/var/tmp/portage/.distcc
433     distccmon-text N</c>
434     </impo>
435 swift 1.10
436     </body>
437     </section>
438 swift 1.1 </chapter>
439     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20