/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.12 - (hide annotations) (download) (as text)
Fri Jan 9 20:48:14 2004 UTC (10 years, 10 months ago) by swift
Branch: MAIN
Changes since 1.11: +16 -13 lines
File MIME type: application/xml
Fix #36386

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 swift 1.12 <!-- $Header: /home/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.11 2003/11/29 09:45:07 swift Exp $ -->
4 blubber 1.2
5 swift 1.1 <guide link="/doc/en/distcc.xml">
6    
7     <title>Gentoo Distcc Documentation</title>
8    
9     <author title="Author">
10 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 erwin 1.4 </author>
12     <author title="Editor">
13 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 erwin 1.4 </author>
15     <author title="Editor">
16 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
17 blubber 1.2 </author>
18     <author title="Reviewer">
19 swift 1.10 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 swift 1.1 </author>
21     <author title="Reviewer">
22 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
23 swift 1.1 </author>
24    
25 swift 1.6 <license/>
26    
27 swift 1.12 <version>1.3</version>
28     <date>January 9, 2004</date>
29 swift 1.1
30 swift 1.10 <abstract>
31     This document serves as a HOWTO for using distcc with Gentoo.
32     </abstract>
33 swift 1.1
34     <chapter>
35 swift 1.10 <title>Introduction</title>
36     <section>
37     <title>What is distcc?</title>
38     <body>
39    
40     <p>
41     Distcc is a program designed to distribute compiling tasks across a network to
42     participating hosts. It is comprised of a server, <c>distccd</c> and a client
43     program, <c>distcc</c>. Distcc can work transparently with <uri
44     link="http://ccache.samba.org">ccache</uri> and Portage with a little set up.
45     </p>
46    
47     </body>
48     </section>
49 swift 1.8 </chapter>
50    
51     <chapter>
52 swift 1.10 <title>Setup</title>
53     <section>
54     <title>Dependencies</title>
55     <body>
56    
57     <pre caption="Distcc dependencies (2.8 through 2.10)">
58 swift 1.1 &gt;=sys-apps/portage-2.0.46-r11
59 swift 1.8 <codenote>Distcc-2.11 and on require &gt;=sys-apps/portage-2.0.49-r6</codenote>
60 swift 1.1 &gt;=sys-devel/gcc-config-1.3.1
61 erwin 1.4 sys-apps/shadow
62 swift 1.8 <codenote>(As of version 2.8 and up until 2.11) And the following optional dependencies when you have <i>gtk</i> in your <i>USE</i> flags</codenote>
63 swift 1.1 &gt;=x11-libs/gtk+-2.2.1
64 swift 1.8 </pre>
65 swift 1.10
66     <pre caption="Distcc Dependencies (post 2.11.1)">
67 swift 1.8 &gt;=sys-apps/portage-2.0.49-r6
68     &gt;=sys-devel/gcc-config-1.3.1
69     sys-apps/shadow
70     <codenote>In this revision you may chose between a Gnome and GTK GUI monitor, they have the following added dependencies</codenote>
71     <codenote>For GTK:</codenote>
72     &gt;=x11-libs/gtk+-2.0.0
73     &gt;=gnome-base/libglade-2.0.0
74     x11-libs/pango
75     <codenote>For Gnome:</codenote>
76     &gt;=x11-libs/gtk+-2.0.0
77     &gt;=gnome-base/libglade-2.0.0
78     x11-libs/pango
79     &gt;=gnome-base/libgnomeui-2.0.0.0
80     &gt;=gnome-base/libgnome-2.0.0
81     </pre>
82 swift 1.10
83     </body>
84     </section>
85     <section>
86     <title>Installing Distcc</title>
87     <body>
88    
89     <p>
90     Installing Distcc is very easy. Simply set your USE flags and <c>emerge
91     distcc</c>. But, there's a couple of options you should know about.
92     </p>
93    
94     <p>
95     Distcc ships with a graphical monitor to monitor tasks that your computer is
96     sending away for compilation. If you use Gnome then put 'gnome' in your USE
97     flags. However, if you don't use Gnome and would still like to have the
98     monitor then you should put 'gtk' in your USE flags.
99     </p>
100    
101     </body>
102     </section>
103     <section>
104     <title>Setting up Portage to use Distcc</title>
105     <body>
106    
107     <p>
108     Setting up distcc is very easy to do with Portage. Follow these simple steps on
109     each computer you want to use distcc on:
110     </p>
111    
112     <pre caption="Integrating Distcc and Portage">
113 swift 1.1 # <i>emerge distcc</i>
114     # <i>nano -w /etc/make.conf</i>
115 swift 1.5 </pre>
116 swift 1.1
117 swift 1.10 <impo>
118     If you use distcc-2.11.1 or greater with &gt;=portage-2.0.49-r6, then all you
119     need to do as far as setup is to set your hosts (see next code block), modify
120     MAKEOPTS in <path>/etc/make.conf</path>, and add 'distcc' to FEATURES in
121     <path>/etc/make.conf</path>.
122     </impo>
123    
124     <note>
125     Edit your FEATURES flags to include &quot;distcc&quot;
126     </note>
127    
128     <p>
129     Next you have to specify what hosts you want to use. To do this you can use
130     the <c>distcc-config</c> command to set the list of hosts. Here is an example
131     of some hosts that might be in your list:
132     </p>
133    
134     <pre caption="Examples of host definitions">
135 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
136     192.168.0.1/2 192.168.0.2 192.168.0.3/10
137     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 swift 1.12 <codenote>There are also several other methods of setting up hosts. </codenote>
140     <codenote>See the distcc manpage for more details.</codenote>
141 swift 1.8 </pre>
142 swift 1.10
143     <p>
144     It may all look complicated, but in most cases a variant of line 1 or 2 will
145     work. An explanation of each line is: Line 1 is just a space-delimited list
146     of hosts that will use default everything. Line 2 is a list of hosts that
147     specifies the maximum number of jobs (by use of the /N) to send that host at
148     any given time (specified with the <c>/n</c>). Since most people won't be
149     using lines 3 or 4, I'll <uri link="http://distcc.samba.org/man/distcc_1.html">
150     point you to</uri> the distcc docs for more information.
151     </p>
152    
153     <p>
154     A sample command to set the hosts (for line 1) is:
155     </p>
156    
157     <pre caption="Sample command to set the hosts">
158 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
159 swift 1.8 </pre>
160 swift 1.1
161 swift 1.10 <p>
162     The final step to integrating distcc into Portage is to re-open your
163     <path>/etc/make.conf</path> and edit <c>MAKEOPTS</c> to include <c>-jN</c>
164     (where N is an integer). Typically you will want to set this to the total
165     number of processors in your network plus one.
166     </p>
167    
168     <pre caption="Final steps in make.conf">
169 swift 1.1 # <i>nano -w /etc/make.conf</i>
170 swift 1.5 MAKEOPTS=-jN
171 swift 1.8 </pre>
172 swift 1.10
173     <impo>
174     Don't forget to start the distcc daemon! <c>/etc/init.d/distccd start</c>
175     </impo>
176    
177 swift 1.12 <note>
178     You will likely have to find a suitable 'N' for your particular setup. Another
179     strategy is 2 x #number of CPUs + 1.
180     </note>
181    
182 swift 1.10 </body>
183     </section>
184     <section>
185     <title>Setting up Distcc to Work With Automake</title>
186     <body>
187    
188     <p>
189     This is in some cases easier than the Portage setup. What you have to do is
190     update your <c>PATH</c> environmental variable to include
191     <path>/usr/lib/distcc/bin</path> before where gcc (<path>/usr/bin</path>).
192     However, there is a caveat. If you use ccache you have to put distcc after
193     the ccache part. Like so:
194     </p>
195    
196     <pre caption="Setting your path">
197 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
198 swift 1.8 </pre>
199 swift 1.10
200     <p>
201     Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
202     (where N is an integer). A safe number to use for N varies on your network
203     and the types of computers you are using to compile. For me, I have a dual
204     processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz
205     Athlon Thunderbird) and I use -j5. Test your own settings to find the right
206     number.
207     </p>
208    
209     </body>
210     </section>
211 blubber 1.2 </chapter>
212 swift 1.1
213 blubber 1.2 <chapter>
214 swift 1.10 <title>Cross-Compiling</title>
215     <section>
216     <title>A Note on Cross-Compiling</title>
217     <body>
218    
219     <p>
220     Cross-compiling is using one architecture to build programs for another
221     architecture. This can be as simple as using an Athlon (i686) to build a
222     program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
223     </p>
224    
225     </body>
226     </section>
227     <section>
228     <title>A Personal Note on Cross-compiling</title>
229     <body>
230    
231     <p>
232     I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
233     don't have any non-x86 machines. I can create things that work <e>in
234     theory</e>, but I have to rely on other people to test what I write. That
235     works to an extent, but it's tough. ;-)
236     </p>
237    
238     <p>
239     In the near future I hope to acquire a Sparc of some type to put Gentoo on so
240     I can play around in-house.
241     </p>
242    
243     </body>
244     </section>
245 swift 1.1 </chapter>
246    
247     <chapter>
248 swift 1.10 <title>Using Distcc to Bootstrap</title>
249     <section>
250     <title>Step 1: Setup on New Box</title>
251     <body>
252    
253     <p>
254     Next, you have to boot your new box with a Gentoo Linux LiveCD and follow all
255     of the steps up until the bootstrapping. Then proceed with a little preliminary
256     setup on the new box.
257     </p>
258    
259     <pre caption="Preliminary Setup">
260 blubber 1.2 <codenote>Here we add distcc to FEATURES</codenote>
261 swift 1.8 # <i>nano -w /etc/make.conf</i>
262     <codenote>Add &quot;distcc&quot; to FEATURES</codenote>
263     <codenote>Modify MAKEOPTS in <path>/etc/make.conf to include -jN (as described above)</path></codenote>
264     </pre>
265 swift 1.10
266     <p>
267     Then add the distcc user to your <path>/etc/passwd</path>:
268     </p>
269    
270     <pre caption="Add the distcc user to /etc/passwd">
271 swift 1.5 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
272 erwin 1.4 <codenote>Do not forget the `<i>&gt;&gt;</i>'</codenote>
273 swift 1.8 </pre>
274 swift 1.10
275     </body>
276     </section>
277     <section>
278     <title>Step 2: Getting Distcc</title>
279     <body>
280    
281     <p>
282     Next you will want to issue the following command inside the chroot on the new
283     box.
284     </p>
285    
286     <pre caption="Getting Distcc on the new box">
287 swift 1.11 # <i>USE='-*' emerge --nodeps distcc</i>
288 swift 1.8 </pre>
289 swift 1.10
290 swift 1.11 <note>
291     <uri link="http://bugs.gentoo.org/show_bug.cgi?id=34262">Bug 34262</uri>
292     mentions a problem with pkg-config (dev-util/pkgconfig), so it may be necessary
293     to also issue <c>emerge --nodeps pkgconfig</c> prior to the above emerge.
294     </note>
295    
296 swift 1.10 </body>
297     </section>
298     <section>
299     <title>Step 3: Setting Up Distcc Itself</title>
300     <body>
301    
302     <p>
303     Now set up distcc itself...
304     </p>
305    
306     <pre caption="Final distcc setup">
307 erwin 1.4 # <i>/usr/bin/distcc-config --install</i>
308     # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
309 swift 1.8 </pre>
310 swift 1.10
311     <p>
312     distcc should be set up now to bootstrap! Continue the official install guide,
313     and don't forget to re-emerge distcc after <c>emerge system</c>.
314     </p>
315    
316 swift 1.12 <note>
317     It is important to note that during bootstrap and <c>emerge system</c> distcc
318     may not appear to be being used. This is not to be unexpected. Some ebuilds or
319     packages do not work well with distcc, so they intentionally disable distcc.
320     </note>
321    
322 swift 1.10 </body>
323     </section>
324 blubber 1.2 </chapter>
325 swift 1.1
326 swift 1.5 <chapter>
327 swift 1.10 <title>Troubleshooting</title>
328     <section>
329     <title>Mozilla and Xfree</title>
330     <body>
331    
332     <p>
333     As you emerge various packages, you'll notice that they aren't being distributed
334     (and indeed aren't being built in parallel). This is because the developers of
335     Mozilla and Xfree .ebuilds disabled parallel building because it is known to
336     cause problems. This isn't necessarily a distcc problem.
337     </p>
338    
339     <p>
340     That isn't to say that sometimes distcc will cause a package to fail to compile.
341     </p>
342    
343     </body>
344     </section>
345     <section>
346     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
347     <body>
348    
349     <p>
350     With such a long title any explanation here is almost irrelevent. However, if
351     you plan to use distcc across hosts that have the <uri
352     link="http://www.gentoo.org/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri>
353     and some that do not, you will run into problems.
354     </p>
355    
356     <p>
357     The solution requires a little foresight on your part; you have to run
358     <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
359     enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
360     which are a good thing to do since for the most part the protections offered
361     by both packages is a good thing and is transparent to the user.
362     </p>
363    
364     </body>
365     </section>
366     <section>
367     <title>Mixed GCC Versions</title>
368     <body>
369    
370     <p>
371     If you have differing versions of GCC on your hosts, there will likely be very
372     weird problems. The solution is to make certain all hosts have the same GCC
373     version.
374     </p>
375    
376     </body>
377     </section>
378 swift 1.5 </chapter>
379 swift 1.1
380     <chapter>
381 swift 1.10 <title>Distcc Extras</title>
382     <section>
383     <title>Distcc Monitors</title>
384     <body>
385    
386     <p>
387     Distcc ships with two monitors. The text-based one is always built, it is
388     called <c>distccmon-text</c>. Running it for the first time can be a bit
389     confusing, but it is really quite easy to use. If you run the program with no
390     parameter it will run once. However, if you pass it a number it will update
391     every N seconds, where N is the argument you passed.
392     </p>
393    
394     <p>
395     The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
396     in your <c>USE</c> flags. This one is GTK+ based and runs in an X environment,
397     and it is quite lovely.
398     </p>
399    
400     <p>
401     There is a caveat to using these programs. If you want to monitor any emerge,
402     you have to start the monitor like so:
403     </p>
404    
405     <pre caption="Starting the monitors">
406 swift 1.8 # <i>DISTCC_DIR=/path/to/distccdir distccmon-text N</i>
407 swift 1.1 <codenote>Or you can start the graphical monitor...</codenote>
408 swift 1.8 # <i>DISTCC_DIR=/path/to/distccdir distccmon-gnome</i>
409     </pre>
410 swift 1.10
411     <impo>
412     If you are using Portage and used the above example, this will be
413     <c>DISTCC_DIR=/var/tmp/portage/.distcc</c>.
414     </impo>
415    
416     <note>
417     If you aren't using Portage you do not need to specify DISTCC_DIR (i.e., just
418     type distccmon-text N).
419     </note>
420    
421     </body>
422     </section>
423 swift 1.1 </chapter>
424    
425     <chapter>
426 swift 1.10 <title>Future Plans for Distcc and Gentoo</title>
427     <section>
428     <title>distcc-subnetscan</title>
429     <body>
430    
431     <p>
432     <c>distcc-subnetscan</c> is a perl program in development that will scan a
433     subnet for hosts that have a participating distcc daemon. Even this will be
434     enhanced to test if the daemon on a remote host will conform to a specific
435     <c>CHOST</c> setting to make cross-compiling easier.
436     </p>
437    
438     <p>
439     The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">
440     right here</uri> until a more formal home can be found.
441     </p>
442    
443     </body>
444     </section>
445     <section>
446     <title>distcc-config</title>
447     <body>
448    
449     <p>
450     <c>distcc-config</c>, the userland configuration tool for distcc is out of date.
451 swift 1.11 It has been rewritten and is present in <path>distcc-2.11.1-r1</path> and up.
452 swift 1.10 </p>
453    
454     </body>
455     </section>
456 swift 1.1 </chapter>
457     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20