/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.18 - (hide annotations) (download) (as text)
Wed May 12 14:44:19 2004 UTC (11 years, 1 month ago) by swift
Branch: MAIN
Changes since 1.17: +7 -6 lines
File MIME type: application/xml
#50844 - distcc-config --install doesnt work during bootstrap due to absence of useradd/adduser command

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 swift 1.18 <!-- $Header: /home/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.17 2004/05/09 10:16:34 swift Exp $ -->
4 swift 1.17
5     <!--
6 swift 1.18 Synchronised with rev 1.15 on
7 swift 1.17 http://www.thedoh.com/viewcvs/distcc/html/distcc.xml
8     -->
9 blubber 1.2
10 swift 1.1 <guide link="/doc/en/distcc.xml">
11    
12     <title>Gentoo Distcc Documentation</title>
13    
14     <author title="Author">
15 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
16 erwin 1.4 </author>
17 swift 1.17
18 erwin 1.4 <author title="Editor">
19 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
20 erwin 1.4 </author>
21 swift 1.17
22 erwin 1.4 <author title="Editor">
23 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
24 blubber 1.2 </author>
25 swift 1.17
26     <author title="Editor">
27 swift 1.10 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
28 swift 1.1 </author>
29 swift 1.17
30     <author title="Reviewer">
31 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
32 swift 1.1 </author>
33    
34 swift 1.15 <abstract>
35     This document serves as a HOWTO for using distcc with Gentoo.
36     </abstract>
37    
38 swift 1.17 <!-- The content of this document is licensed under the CC-BY-SA license -->
39     <!-- See http://creativecommons.org/licenses/by-sa/1.0 -->
40 swift 1.6 <license/>
41    
42 swift 1.18 <version>1.6</version>
43     <date>May 12, 2004</date>
44 swift 1.1
45     <chapter>
46 swift 1.10 <title>Introduction</title>
47     <section>
48     <title>What is distcc?</title>
49     <body>
50    
51     <p>
52     Distcc is a program designed to distribute compiling tasks across a network to
53     participating hosts. It is comprised of a server, <c>distccd</c> and a client
54     program, <c>distcc</c>. Distcc can work transparently with <uri
55 swift 1.17 link="http://ccache.samba.org">ccache</uri> and Portage with a little work.
56     </p>
57    
58     </body>
59     </section>
60     <section>
61     <title>Using distcc to bootstrap</title>
62     <body>
63    
64     <p>
65     If you are planning on using distcc to help you bootstrap a Gentoo installation,
66     make sure you read the part on <uri link="#bootstrapping">Using distcc to
67     Bootstrap</uri> which is situated further down in this document.
68 swift 1.10 </p>
69    
70     </body>
71     </section>
72 swift 1.8 </chapter>
73    
74     <chapter>
75 swift 1.10 <title>Setup</title>
76     <section>
77     <title>Dependencies</title>
78     <body>
79    
80 swift 1.17 <p>
81     To start off with, to use Distcc all of the computers on your network need to
82     have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but
83     mixing 3.3.x and 3.2.x is not: your programs <b>will</b> have errors in it.
84     </p>
85 swift 1.10
86     </body>
87     </section>
88     <section>
89     <title>Installing Distcc</title>
90     <body>
91    
92     <p>
93 swift 1.17 There are a couple of options you should know about before you start
94     installing distcc.
95 swift 1.10 </p>
96    
97     <p>
98     Distcc ships with a graphical monitor to monitor tasks that your computer is
99     sending away for compilation. If you use Gnome then put 'gnome' in your USE
100     flags. However, if you don't use Gnome and would still like to have the
101     monitor then you should put 'gtk' in your USE flags.
102     </p>
103    
104 swift 1.17 <pre caption="Installing distcc">
105     # <i>emerge distcc</i>
106     </pre>
107    
108 swift 1.10 </body>
109     </section>
110     <section>
111     <title>Setting up Portage to use Distcc</title>
112     <body>
113    
114     <p>
115 swift 1.17 Setting up distcc is easy to do with Portage. Execute the following steps on
116     each system that should participate in the distributed compiling:
117 swift 1.10 </p>
118    
119     <pre caption="Integrating Distcc and Portage">
120 swift 1.1 # <i>emerge distcc</i>
121     # <i>nano -w /etc/make.conf</i>
122 swift 1.17 <comment>(Set N to a suitable number for your particular setup)</comment>
123     <comment>(A common heuristic strategy is twice the number of CPUs + 1)</comment>
124     MAKEOPTS="-jN"
125     <comment>(Add distcc to your FEATURES variable)</comment>
126     FEATURES="distcc"
127     <comment>(Set distcc's directory for temporary space)</comment>
128     DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
129 swift 1.5 </pre>
130 swift 1.1
131 swift 1.10 <p>
132 swift 1.17 Next you have to specify what hosts you want to use. Use the
133     <c>distcc-config</c> command to set the list of hosts. Here is an example of
134     some hosts that might be in your list:
135 swift 1.10 </p>
136    
137     <pre caption="Examples of host definitions">
138 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
139     192.168.0.1/2 192.168.0.2 192.168.0.3/10
140     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
141     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
142 swift 1.17 <comment>(There are also several other methods of setting up hosts. See the
143     distcc manpage for more details.)</comment>
144 swift 1.8 </pre>
145 swift 1.10
146     <p>
147     It may all look complicated, but in most cases a variant of line 1 or 2 will
148 swift 1.17 work.
149 swift 1.10 </p>
150    
151 swift 1.17 <ul>
152     <li>Line 1 is just a space-delimited list of hosts</li>
153     <li>
154     Line 2 is a list of hosts that specifies the maximum amount of jobs (by
155     defining the <c>/N</c> setting) to send that host at any given time
156     </li>
157     </ul>
158    
159 swift 1.10 <p>
160 swift 1.17 Since most people won't be using lines 3 or 4, I'll <uri
161     link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc
162     docs for more information.
163     </p>
164    
165     <p>
166     For instance, to set the first line in the previous example:
167 swift 1.10 </p>
168    
169     <pre caption="Sample command to set the hosts">
170 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
171 swift 1.8 </pre>
172 swift 1.1
173 swift 1.10 <p>
174 swift 1.17 Now start the distcc daemon on all the participating computers:
175 swift 1.10 </p>
176    
177 swift 1.17 <pre caption="Starting the distcc daemon">
178     <comment>(Add distccd to the default runlevel)</comment>
179     # <i>rc-update add distccd default</i>
180     <comment>(Start the distcc daemon)</comment>
181     # <i>/etc/init.d/distccd start</i>
182 swift 1.8 </pre>
183 swift 1.10
184     </body>
185     </section>
186     <section>
187     <title>Setting up Distcc to Work With Automake</title>
188     <body>
189    
190     <p>
191     This is in some cases easier than the Portage setup. What you have to do is
192 swift 1.17 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
193     in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
194 swift 1.10 However, there is a caveat. If you use ccache you have to put distcc after
195 swift 1.17 the ccache part:
196 swift 1.10 </p>
197    
198     <pre caption="Setting your path">
199 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
200 swift 1.8 </pre>
201 swift 1.10
202     <p>
203     Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
204 swift 1.17 (where N is an integer). The value of N depends on your network and the types
205     of computers you are using to compile. For me, I have a dual processor P3 and
206     a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon
207     Thunderbird) and I use -j5. Test your own settings to find the most
208     performant number.
209 swift 1.10 </p>
210    
211     </body>
212     </section>
213 blubber 1.2 </chapter>
214 swift 1.1
215 blubber 1.2 <chapter>
216 swift 1.17 <title>Cross-Compiling</title>
217     <section>
218     <title>A Note on Cross-Compiling</title>
219     <body>
220    
221     <p>
222     Cross-compiling is using one architecture to build programs for another
223     architecture. This can be as simple as using an Athlon (i686) to build a
224     program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
225     </p>
226    
227     </body>
228     </section>
229 swift 1.10 <section>
230 swift 1.17 <title>A Personal Note on Cross-compiling</title>
231 swift 1.10 <body>
232    
233     <p>
234 swift 1.17 I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
235     don't have any non-x86 machines. I can create things that work <e>in
236     theory</e>, but I have to rely on other people to test what I write.
237     That works to an extent, but it's tough. ;-)
238 swift 1.10 </p>
239    
240 swift 1.17 <p>
241     In the near future I hope to acquire a Sparc of some type to put Gentoo on so
242     I can play around in-house.
243     </p>
244    
245     </body>
246     </section>
247     </chapter>
248    
249     <chapter id="bootstrapping">
250     <title>Using Distcc to Bootstrap</title>
251     <section>
252     <title>Step 1: Configure Portage </title>
253     <body>
254 swift 1.10
255     <p>
256 swift 1.17 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
257     link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
258     up until the bootstrapping part. Then configure Portage to use distcc:
259 swift 1.10 </p>
260    
261 swift 1.17 <pre caption="Preliminary Setup">
262     # <i>nano -w /etc/make.conf</i>
263     <comment>(Add distcc to the FEATURES variable)</comment>
264     FEATURES="distcc"
265     <comment>(Set distcc's directory for temporary space)</comment>
266     DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
267     <comment>(Modify MAKEOPTS to include -jN with N an appropriate integer)</comment>
268     MAKEOPTS="-jN"
269 swift 1.8 </pre>
270 swift 1.10
271     </body>
272     </section>
273     <section>
274     <title>Step 2: Getting Distcc</title>
275     <body>
276    
277     <p>
278 swift 1.17 Install distcc:
279 swift 1.10 </p>
280    
281     <pre caption="Getting Distcc on the new box">
282 swift 1.11 # <i>USE='-*' emerge --nodeps distcc</i>
283 swift 1.8 </pre>
284 swift 1.10
285     </body>
286     </section>
287     <section>
288 swift 1.17 <title>Step 3: Setting Up Distcc</title>
289 swift 1.10 <body>
290    
291     <p>
292 swift 1.18 Add a "distcc" user to the <path>/etc/passwd</path> file and run
293     <c>distcc-config</c> to setup distcc:
294 swift 1.10 </p>
295    
296     <pre caption="Final distcc setup">
297 swift 1.18 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt; /etc/passwd</i>
298 swift 1.17 <comment>(Substitute host1, host2, ... with the IP number(s) of the participating hosts)</comment>
299 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
300 swift 1.17 </pre>
301 swift 1.10
302     <p>
303 swift 1.17 distcc is now set up to bootstrap! Continue the official installation
304     instructions and <e>don't forget</e> to re-emerge distcc after <c>emerge
305     system</c>. This is to make sure that all of the dependencies you want are
306     installed as well.
307 swift 1.10 </p>
308    
309 swift 1.12 <note>
310 swift 1.17 During bootstrap and <c>emerge system</c> Distcc may not appear to be used.
311     This is expected as some ebuilds do not work well with distcc, so they
312     intentionally disable it.
313 swift 1.12 </note>
314    
315 swift 1.10 </body>
316     </section>
317 blubber 1.2 </chapter>
318 swift 1.1
319 swift 1.5 <chapter>
320 swift 1.10 <title>Troubleshooting</title>
321     <section>
322     <title>Mozilla and Xfree</title>
323     <body>
324    
325     <p>
326 swift 1.17 As you emerge various packages, you'll notice that some of them aren't being
327     distributed (and aren't being built in parallel). This is because the
328     developers of the Mozilla and Xfree ebuilds intentionally disable parallel
329     building because it is known to cause problems.
330 swift 1.10 </p>
331    
332     <p>
333 swift 1.17 Sometimes distcc might cause a package to fail to compile. If this happens
334     for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
335 swift 1.10 </p>
336    
337     </body>
338     </section>
339     <section>
340     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
341     <body>
342    
343     <p>
344     With such a long title any explanation here is almost irrelevent. However, if
345     you plan to use distcc across hosts that have the <uri
346 swift 1.17 link="/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do
347     not, you will run into problems.
348 swift 1.10 </p>
349    
350     <p>
351     The solution requires a little foresight on your part; you have to run
352     <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
353     enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
354     which are a good thing to do since for the most part the protections offered
355     by both packages is a good thing and is transparent to the user.
356     </p>
357    
358     </body>
359     </section>
360     <section>
361     <title>Mixed GCC Versions</title>
362     <body>
363    
364     <p>
365 swift 1.17 If you have different GCC versions on your hosts, there will likely be very
366 swift 1.10 weird problems. The solution is to make certain all hosts have the same GCC
367     version.
368     </p>
369    
370     </body>
371     </section>
372 swift 1.5 </chapter>
373 swift 1.1
374     <chapter>
375 swift 1.10 <title>Distcc Extras</title>
376     <section>
377     <title>Distcc Monitors</title>
378     <body>
379    
380     <p>
381 swift 1.17 Distcc ships with two monitors. The text-based one is always built and is
382 swift 1.10 called <c>distccmon-text</c>. Running it for the first time can be a bit
383     confusing, but it is really quite easy to use. If you run the program with no
384     parameter it will run once. However, if you pass it a number it will update
385     every N seconds, where N is the argument you passed.
386     </p>
387    
388     <p>
389     The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
390 swift 1.17 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
391 swift 1.10 and it is quite lovely.
392     </p>
393    
394     <pre caption="Starting the monitors">
395 swift 1.17 # <i>distccmon-text N</i>
396     <comment>(Or ...)</comment>
397     # <i>distccmon-gnome</i>
398 swift 1.8 </pre>
399 swift 1.10
400     </body>
401     </section>
402 swift 1.1 </chapter>
403    
404     <chapter>
405 swift 1.10 <title>Future Plans for Distcc and Gentoo</title>
406     <section>
407     <title>distcc-subnetscan</title>
408     <body>
409    
410     <p>
411     <c>distcc-subnetscan</c> is a perl program in development that will scan a
412     subnet for hosts that have a participating distcc daemon. Even this will be
413     enhanced to test if the daemon on a remote host will conform to a specific
414     <c>CHOST</c> setting to make cross-compiling easier.
415     </p>
416    
417     <p>
418 swift 1.17 The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">lisa's
419     dev page</uri> until a more formal home can be found.
420 swift 1.10 </p>
421    
422     </body>
423     </section>
424 swift 1.1 </chapter>
425     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20