/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.13 - (hide annotations) (download) (as text)
Sun Jan 18 22:08:53 2004 UTC (10 years, 11 months ago) by neysx
Branch: MAIN
Changes since 1.12: +5 -15 lines
File MIME type: application/xml
Bug #34351

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 neysx 1.13 <!-- $Header: /home/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.12 2004/01/09 20:48:14 swift Exp $ -->
4 blubber 1.2
5 swift 1.1 <guide link="/doc/en/distcc.xml">
6    
7     <title>Gentoo Distcc Documentation</title>
8    
9     <author title="Author">
10 swift 1.10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 erwin 1.4 </author>
12     <author title="Editor">
13 swift 1.10 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 erwin 1.4 </author>
15     <author title="Editor">
16 swift 1.10 <mail link="erwin@gentoo.org">Erwin</mail>
17 blubber 1.2 </author>
18     <author title="Reviewer">
19 swift 1.10 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 swift 1.1 </author>
21     <author title="Reviewer">
22 swift 1.10 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
23 swift 1.1 </author>
24    
25 swift 1.6 <license/>
26    
27 swift 1.12 <version>1.3</version>
28     <date>January 9, 2004</date>
29 swift 1.1
30 swift 1.10 <abstract>
31     This document serves as a HOWTO for using distcc with Gentoo.
32     </abstract>
33 swift 1.1
34     <chapter>
35 swift 1.10 <title>Introduction</title>
36     <section>
37     <title>What is distcc?</title>
38     <body>
39    
40     <p>
41     Distcc is a program designed to distribute compiling tasks across a network to
42     participating hosts. It is comprised of a server, <c>distccd</c> and a client
43     program, <c>distcc</c>. Distcc can work transparently with <uri
44     link="http://ccache.samba.org">ccache</uri> and Portage with a little set up.
45     </p>
46    
47     </body>
48     </section>
49 swift 1.8 </chapter>
50    
51     <chapter>
52 swift 1.10 <title>Setup</title>
53     <section>
54     <title>Dependencies</title>
55     <body>
56    
57     <pre caption="Distcc dependencies (2.8 through 2.10)">
58 swift 1.1 &gt;=sys-apps/portage-2.0.46-r11
59 swift 1.8 <codenote>Distcc-2.11 and on require &gt;=sys-apps/portage-2.0.49-r6</codenote>
60 swift 1.1 &gt;=sys-devel/gcc-config-1.3.1
61 erwin 1.4 sys-apps/shadow
62 swift 1.8 <codenote>(As of version 2.8 and up until 2.11) And the following optional dependencies when you have <i>gtk</i> in your <i>USE</i> flags</codenote>
63 swift 1.1 &gt;=x11-libs/gtk+-2.2.1
64 swift 1.8 </pre>
65 swift 1.10
66     <pre caption="Distcc Dependencies (post 2.11.1)">
67 swift 1.8 &gt;=sys-apps/portage-2.0.49-r6
68     &gt;=sys-devel/gcc-config-1.3.1
69     sys-apps/shadow
70     <codenote>In this revision you may chose between a Gnome and GTK GUI monitor, they have the following added dependencies</codenote>
71     <codenote>For GTK:</codenote>
72     &gt;=x11-libs/gtk+-2.0.0
73     &gt;=gnome-base/libglade-2.0.0
74     x11-libs/pango
75     <codenote>For Gnome:</codenote>
76     &gt;=x11-libs/gtk+-2.0.0
77     &gt;=gnome-base/libglade-2.0.0
78     x11-libs/pango
79     &gt;=gnome-base/libgnomeui-2.0.0.0
80     &gt;=gnome-base/libgnome-2.0.0
81     </pre>
82 swift 1.10
83     </body>
84     </section>
85     <section>
86     <title>Installing Distcc</title>
87     <body>
88    
89     <p>
90     Installing Distcc is very easy. Simply set your USE flags and <c>emerge
91     distcc</c>. But, there's a couple of options you should know about.
92     </p>
93    
94     <p>
95     Distcc ships with a graphical monitor to monitor tasks that your computer is
96     sending away for compilation. If you use Gnome then put 'gnome' in your USE
97     flags. However, if you don't use Gnome and would still like to have the
98     monitor then you should put 'gtk' in your USE flags.
99     </p>
100    
101     </body>
102     </section>
103     <section>
104     <title>Setting up Portage to use Distcc</title>
105     <body>
106    
107     <p>
108     Setting up distcc is very easy to do with Portage. Follow these simple steps on
109     each computer you want to use distcc on:
110     </p>
111    
112     <pre caption="Integrating Distcc and Portage">
113 swift 1.1 # <i>emerge distcc</i>
114     # <i>nano -w /etc/make.conf</i>
115 swift 1.5 </pre>
116 swift 1.1
117 swift 1.10 <impo>
118     If you use distcc-2.11.1 or greater with &gt;=portage-2.0.49-r6, then all you
119     need to do as far as setup is to set your hosts (see next code block), modify
120     MAKEOPTS in <path>/etc/make.conf</path>, and add 'distcc' to FEATURES in
121     <path>/etc/make.conf</path>.
122     </impo>
123    
124     <note>
125     Edit your FEATURES flags to include &quot;distcc&quot;
126     </note>
127    
128     <p>
129     Next you have to specify what hosts you want to use. To do this you can use
130     the <c>distcc-config</c> command to set the list of hosts. Here is an example
131     of some hosts that might be in your list:
132     </p>
133    
134     <pre caption="Examples of host definitions">
135 swift 1.8 192.168.0.1 192.168.0.2 192.168.0.3
136     192.168.0.1/2 192.168.0.2 192.168.0.3/10
137     192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
138     @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
139 swift 1.12 <codenote>There are also several other methods of setting up hosts. </codenote>
140     <codenote>See the distcc manpage for more details.</codenote>
141 swift 1.8 </pre>
142 swift 1.10
143     <p>
144     It may all look complicated, but in most cases a variant of line 1 or 2 will
145     work. An explanation of each line is: Line 1 is just a space-delimited list
146     of hosts that will use default everything. Line 2 is a list of hosts that
147     specifies the maximum number of jobs (by use of the /N) to send that host at
148     any given time (specified with the <c>/n</c>). Since most people won't be
149     using lines 3 or 4, I'll <uri link="http://distcc.samba.org/man/distcc_1.html">
150     point you to</uri> the distcc docs for more information.
151     </p>
152    
153     <p>
154     A sample command to set the hosts (for line 1) is:
155     </p>
156    
157     <pre caption="Sample command to set the hosts">
158 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
159 swift 1.8 </pre>
160 swift 1.1
161 swift 1.10 <p>
162     The final step to integrating distcc into Portage is to re-open your
163     <path>/etc/make.conf</path> and edit <c>MAKEOPTS</c> to include <c>-jN</c>
164     (where N is an integer). Typically you will want to set this to the total
165     number of processors in your network plus one.
166     </p>
167    
168     <pre caption="Final steps in make.conf">
169 swift 1.1 # <i>nano -w /etc/make.conf</i>
170 swift 1.5 MAKEOPTS=-jN
171 swift 1.8 </pre>
172 swift 1.10
173     <impo>
174     Don't forget to start the distcc daemon! <c>/etc/init.d/distccd start</c>
175     </impo>
176    
177 swift 1.12 <note>
178     You will likely have to find a suitable 'N' for your particular setup. Another
179     strategy is 2 x #number of CPUs + 1.
180     </note>
181    
182 swift 1.10 </body>
183     </section>
184     <section>
185     <title>Setting up Distcc to Work With Automake</title>
186     <body>
187    
188     <p>
189     This is in some cases easier than the Portage setup. What you have to do is
190     update your <c>PATH</c> environmental variable to include
191     <path>/usr/lib/distcc/bin</path> before where gcc (<path>/usr/bin</path>).
192     However, there is a caveat. If you use ccache you have to put distcc after
193     the ccache part. Like so:
194     </p>
195    
196     <pre caption="Setting your path">
197 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
198 swift 1.8 </pre>
199 swift 1.10
200     <p>
201     Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
202     (where N is an integer). A safe number to use for N varies on your network
203     and the types of computers you are using to compile. For me, I have a dual
204     processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz
205     Athlon Thunderbird) and I use -j5. Test your own settings to find the right
206     number.
207     </p>
208    
209     </body>
210     </section>
211 blubber 1.2 </chapter>
212 swift 1.1
213 blubber 1.2 <chapter>
214 swift 1.10 <title>Cross-Compiling</title>
215     <section>
216     <title>A Note on Cross-Compiling</title>
217     <body>
218    
219     <p>
220     Cross-compiling is using one architecture to build programs for another
221     architecture. This can be as simple as using an Athlon (i686) to build a
222     program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
223     </p>
224    
225     </body>
226     </section>
227     <section>
228     <title>A Personal Note on Cross-compiling</title>
229     <body>
230    
231     <p>
232     I'd love to be able to help out on the Gentoo Cross-compiling effort, but I
233     don't have any non-x86 machines. I can create things that work <e>in
234     theory</e>, but I have to rely on other people to test what I write. That
235     works to an extent, but it's tough. ;-)
236     </p>
237    
238     <p>
239     In the near future I hope to acquire a Sparc of some type to put Gentoo on so
240     I can play around in-house.
241     </p>
242    
243     </body>
244     </section>
245 swift 1.1 </chapter>
246    
247     <chapter>
248 swift 1.10 <title>Using Distcc to Bootstrap</title>
249     <section>
250     <title>Step 1: Setup on New Box</title>
251     <body>
252    
253     <p>
254     Next, you have to boot your new box with a Gentoo Linux LiveCD and follow all
255     of the steps up until the bootstrapping. Then proceed with a little preliminary
256     setup on the new box.
257     </p>
258    
259     <pre caption="Preliminary Setup">
260 blubber 1.2 <codenote>Here we add distcc to FEATURES</codenote>
261 swift 1.8 # <i>nano -w /etc/make.conf</i>
262     <codenote>Add &quot;distcc&quot; to FEATURES</codenote>
263     <codenote>Modify MAKEOPTS in <path>/etc/make.conf to include -jN (as described above)</path></codenote>
264     </pre>
265 swift 1.10
266     <p>
267     Then add the distcc user to your <path>/etc/passwd</path>:
268     </p>
269    
270     <pre caption="Add the distcc user to /etc/passwd">
271 swift 1.5 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
272 erwin 1.4 <codenote>Do not forget the `<i>&gt;&gt;</i>'</codenote>
273 swift 1.8 </pre>
274 swift 1.10
275     </body>
276     </section>
277     <section>
278     <title>Step 2: Getting Distcc</title>
279     <body>
280    
281     <p>
282     Next you will want to issue the following command inside the chroot on the new
283     box.
284     </p>
285    
286     <pre caption="Getting Distcc on the new box">
287 swift 1.11 # <i>USE='-*' emerge --nodeps distcc</i>
288 swift 1.8 </pre>
289 swift 1.10
290 swift 1.11 <note>
291 neysx 1.13 According to <uri link="http://bugs.gentoo.org/show_bug.cgi?id=34262">bug
292     34262</uri>, if the command mentioned above fails with an error like
293     'pkg-config: command not found', you should issue <c>emerge --nodeps
294     pkgconfig</c> and then try the above emerge again.
295 swift 1.11 </note>
296    
297 swift 1.10 </body>
298     </section>
299     <section>
300     <title>Step 3: Setting Up Distcc Itself</title>
301     <body>
302    
303     <p>
304     Now set up distcc itself...
305     </p>
306    
307     <pre caption="Final distcc setup">
308 erwin 1.4 # <i>/usr/bin/distcc-config --install</i>
309     # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
310 swift 1.8 </pre>
311 swift 1.10
312     <p>
313     distcc should be set up now to bootstrap! Continue the official install guide,
314     and don't forget to re-emerge distcc after <c>emerge system</c>.
315     </p>
316    
317 swift 1.12 <note>
318     It is important to note that during bootstrap and <c>emerge system</c> distcc
319     may not appear to be being used. This is not to be unexpected. Some ebuilds or
320     packages do not work well with distcc, so they intentionally disable distcc.
321     </note>
322    
323 swift 1.10 </body>
324     </section>
325 blubber 1.2 </chapter>
326 swift 1.1
327 swift 1.5 <chapter>
328 swift 1.10 <title>Troubleshooting</title>
329     <section>
330     <title>Mozilla and Xfree</title>
331     <body>
332    
333     <p>
334     As you emerge various packages, you'll notice that they aren't being distributed
335     (and indeed aren't being built in parallel). This is because the developers of
336     Mozilla and Xfree .ebuilds disabled parallel building because it is known to
337     cause problems. This isn't necessarily a distcc problem.
338     </p>
339    
340     <p>
341     That isn't to say that sometimes distcc will cause a package to fail to compile.
342     </p>
343    
344     </body>
345     </section>
346     <section>
347     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
348     <body>
349    
350     <p>
351     With such a long title any explanation here is almost irrelevent. However, if
352     you plan to use distcc across hosts that have the <uri
353     link="http://www.gentoo.org/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri>
354     and some that do not, you will run into problems.
355     </p>
356    
357     <p>
358     The solution requires a little foresight on your part; you have to run
359     <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to
360     enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of
361     which are a good thing to do since for the most part the protections offered
362     by both packages is a good thing and is transparent to the user.
363     </p>
364    
365     </body>
366     </section>
367     <section>
368     <title>Mixed GCC Versions</title>
369     <body>
370    
371     <p>
372     If you have differing versions of GCC on your hosts, there will likely be very
373     weird problems. The solution is to make certain all hosts have the same GCC
374     version.
375     </p>
376    
377     </body>
378     </section>
379 swift 1.5 </chapter>
380 swift 1.1
381     <chapter>
382 swift 1.10 <title>Distcc Extras</title>
383     <section>
384     <title>Distcc Monitors</title>
385     <body>
386    
387     <p>
388     Distcc ships with two monitors. The text-based one is always built, it is
389     called <c>distccmon-text</c>. Running it for the first time can be a bit
390     confusing, but it is really quite easy to use. If you run the program with no
391     parameter it will run once. However, if you pass it a number it will update
392     every N seconds, where N is the argument you passed.
393     </p>
394    
395     <p>
396     The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
397     in your <c>USE</c> flags. This one is GTK+ based and runs in an X environment,
398     and it is quite lovely.
399     </p>
400    
401     <p>
402     There is a caveat to using these programs. If you want to monitor any emerge,
403     you have to start the monitor like so:
404     </p>
405    
406     <pre caption="Starting the monitors">
407 swift 1.8 # <i>DISTCC_DIR=/path/to/distccdir distccmon-text N</i>
408 swift 1.1 <codenote>Or you can start the graphical monitor...</codenote>
409 swift 1.8 # <i>DISTCC_DIR=/path/to/distccdir distccmon-gnome</i>
410     </pre>
411 swift 1.10
412     <impo>
413     If you are using Portage and used the above example, this will be
414     <c>DISTCC_DIR=/var/tmp/portage/.distcc</c>.
415     </impo>
416    
417     <note>
418     If you aren't using Portage you do not need to specify DISTCC_DIR (i.e., just
419     type distccmon-text N).
420     </note>
421    
422     </body>
423     </section>
424 swift 1.1 </chapter>
425    
426     <chapter>
427 swift 1.10 <title>Future Plans for Distcc and Gentoo</title>
428     <section>
429     <title>distcc-subnetscan</title>
430     <body>
431    
432     <p>
433     <c>distcc-subnetscan</c> is a perl program in development that will scan a
434     subnet for hosts that have a participating distcc daemon. Even this will be
435     enhanced to test if the daemon on a remote host will conform to a specific
436     <c>CHOST</c> setting to make cross-compiling easier.
437     </p>
438    
439     <p>
440     The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">
441     right here</uri> until a more formal home can be found.
442     </p>
443    
444     </body>
445     </section>
446 swift 1.1 </chapter>
447     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20