/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.42 - (show annotations) (download) (as text)
Mon Apr 7 20:56:59 2008 UTC (6 years, 4 months ago) by jkt
Branch: MAIN
Changes since 1.41: +18 -3 lines
File MIME type: application/xml
#216677, -march=native doesn't work with distcc

1 <?xml version='1.0' encoding="UTF-8"?>
2 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 <!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.41 2008/01/11 07:05:02 nightmorph Exp $ -->
4
5 <guide link="/doc/en/distcc.xml">
6
7 <title>Gentoo Distcc Documentation</title>
8
9 <author title="Author">
10 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11 </author>
12 <author title="Editor">
13 <mail link="vapier@gentoo.org">Mike Frysinger</mail>
14 </author>
15 <author title="Editor">
16 <mail link="erwin@gentoo.org">Erwin</mail>
17 </author>
18 <author title="Editor">
19 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
20 </author>
21 <author title="Editor">
22 <mail link="pylon@gentoo.org">Lars Weiler</mail>
23 </author>
24 <author title="Reviewer">
25 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
26 </author>
27
28 <abstract>
29 This document serves as a HOWTO for using distcc with Gentoo.
30 </abstract>
31
32 <!-- The content of this document is licensed under the CC-BY-SA license -->
33 <!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
34 <license/>
35
36 <version>1.21</version>
37 <date>2008-04-07</date>
38
39 <chapter>
40 <title>Introduction</title>
41 <section>
42 <title>What is distcc?</title>
43 <body>
44
45 <p>
46 Distcc is a program designed to distribute compiling tasks across a network to
47 participating hosts. It is comprised of a server, <c>distccd</c>, and a client
48 program, <c>distcc</c>. Distcc can work transparently with <uri
49 link="http://ccache.samba.org">ccache</uri>, Portage, and Automake with a
50 little setup.
51 </p>
52
53 </body>
54 </section>
55 <section>
56 <title>Using distcc to bootstrap</title>
57 <body>
58
59 <p>
60 If you are planning on using distcc to help you bootstrap a Gentoo
61 installation, make sure you read the section <uri link="#bootstrapping">Using
62 distcc to Bootstrap</uri>, which is situated further down in this document.
63 </p>
64
65 </body>
66 </section>
67 </chapter>
68
69 <chapter>
70 <title>Setup</title>
71 <section>
72 <title>Dependencies</title>
73 <body>
74
75 <p>
76 In order to use Distcc, all of the computers on your network need to have the
77 same GCC versions. For example, mixing 3.3.x (where the x varies) is okay, but
78 mixing 3.3.x with 3.2.x <b>may</b> result in compilation errors or runtime
79 errors.
80 </p>
81
82 </body>
83 </section>
84 <section>
85 <title>Installing Distcc</title>
86 <body>
87
88 <p>
89 There are a couple of options you should be aware of before you start
90 installing distcc.
91 </p>
92
93 <p>
94 Distcc ships with a graphical monitor to monitor tasks that your computer is
95 sending away for compilation. If you use Gnome then put 'gnome' in your USE
96 flags. However, if you don't use Gnome and would still like to have the
97 monitor then you should put 'gtk' in your USE flags.
98 </p>
99
100 <pre caption="Installing distcc">
101 # <i>emerge distcc</i>
102 </pre>
103
104 <impo>
105 Remember, you must be sure to install distcc on all of your participating
106 machines.
107 </impo>
108
109 </body>
110 </section>
111 <section>
112 <title>Setting up Portage to use Distcc</title>
113 <body>
114
115 <p>
116 Setting up Portage to use distcc is easy. Execute the following steps on
117 each system that should participate in the distributed compiling:
118 </p>
119
120 <pre caption="Integrating Distcc and Portage">
121 # <i>emerge distcc</i>
122 # <i>nano -w /etc/make.conf</i>
123 <comment>(Set N to a suitable number for your particular setup)</comment>
124 <comment>(A common strategy is setting N as twice the number of total CPUs + 1 available)</comment>
125 MAKEOPTS="-jN"
126 <comment>(Add distcc to your FEATURES)</comment>
127 FEATURES="distcc"
128 </pre>
129
130 </body>
131 </section>
132 <section>
133 <title>Specifying Participating Hosts</title>
134 <body>
135
136 <p>
137 Use the <c>distcc-config</c> command to set the list of hosts. Here is an
138 example of some hosts that might be in your list:
139 </p>
140
141 <pre caption="Examples of host definitions">
142 192.168.0.1 192.168.0.2 192.168.0.3
143 192.168.0.1/2 192.168.0.2 192.168.0.3/10
144 192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
145 @192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
146 <comment>(There are also several other methods of setting up hosts. See the
147 distcc manpage for more details.)</comment>
148 <comment>If you wish to compile on the local machine you should put 'localhost'
149 in the hosts list. Conversely if you do not wish to use the local machine to
150 compile (which is often the case) omit it from the hosts list. On a slow
151 machine using localhost may actually slow things down. Make sure to test your
152 settings for performance.</comment>
153 </pre>
154
155 <p>
156 It may all look complicated, but in most cases a variant of line 1 or 2 will
157 work.
158 </p>
159
160 <p>
161 Since most people won't be using lines 3 or 4, I'll <uri
162 link="http://distcc.samba.org/man/distcc_1.html">refer to</uri> the distcc
163 docs (man distcc) for more information.
164 </p>
165
166 <p>
167 For instance, to set the first line in the previous example:
168 </p>
169
170 <pre caption="Sample command to set the hosts">
171 # <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"</i>
172 </pre>
173
174 <p>
175 Edit <path>/etc/conf.d/distccd</path> to your needs and be sure to set the
176 <c>--allow</c> directive to allow only hosts you trust. For added security,
177 you should also use the <c>--listen</c> directive to tell the distcc daemon
178 what IP to listen on (for multi-homed systems). More information on distcc
179 security can be found at <uri
180 link="http://distcc.samba.org/security.html">Distcc Security Design</uri>.
181 </p>
182
183 <impo>
184 It is important to use --allow and --listen. Please read the distccd manpage
185 or the above security document for more information.
186 </impo>
187
188
189 <p>
190 Now start the distcc daemon on all the participating computers:
191 </p>
192
193 <pre caption="Starting the distcc daemon">
194 <comment>(Add distccd to the default runlevel)</comment>
195 # <i>rc-update add distccd default</i>
196 <comment>(Start the distcc daemon)</comment>
197 # <i>/etc/init.d/distccd start</i>
198 </pre>
199
200 </body>
201 </section>
202 <section>
203 <title>Setting up Distcc to Work With Automake</title>
204 <body>
205
206 <p>
207 This is, in some cases, easier than the Portage setup. What you have to do is
208 update your <c>PATH</c> variable to include <path>/usr/lib/distcc/bin</path>
209 in front of the directory that contains <c>gcc</c> (<path>/usr/bin</path>).
210 However, there is a caveat. If you use ccache you have to put distcc after
211 the ccache part:
212 </p>
213
214 <pre caption="Setting your path">
215 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
216 <comment>(You can put this in your ~/.bashrc or equivalent file to have the PATH
217 set every time you log in)</comment>
218 </pre>
219
220 <p>
221 Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
222 (where N is an integer). The value of N depends on your network and the types
223 of computers you are using to compile. Test your own settings to find the
224 number that yields the best performance.
225 </p>
226
227 </body>
228 </section>
229 </chapter>
230
231 <chapter>
232 <title>Cross-Compiling</title>
233 <section>
234 <body>
235
236 <p>
237 Cross-compiling is using one architecture to build programs for another
238 architecture. This can be as simple as using an Athlon (i686) to build a
239 program for a K6-2 (i586), or using a Sparc to build a program for a ppc. This
240 is documented in our <uri link="/doc/en/cross-compiling-distcc.xml">DistCC
241 Cross-compiling Guide</uri>.
242 </p>
243
244 </body>
245 </section>
246 </chapter>
247
248 <chapter id="bootstrapping">
249 <title>Using Distcc to Bootstrap</title>
250 <section>
251 <title>Step 1: Configure Portage</title>
252 <body>
253 <!--
254 Note that this will need to become a link to the old 2005.1 networked handbooks
255 once the 2006.0 handbooks are released. Until then, the link can stay.
256 -->
257 <p>
258 Boot your new box with a Gentoo Linux LiveCD and follow the <uri
259 link="/doc/en/handbook/handbook-x86.xml?part=1">installation instructions</uri>
260 up until the bootstrapping part. (See the <uri
261 link="/doc/en/faq.xml#stage12">Gentoo FAQ</uri> for more information about
262 bootstrapping.) Then configure Portage to use distcc:
263 </p>
264
265 <pre caption="Preliminary Setup">
266 # <i>nano -w /etc/make.conf</i>
267 <comment>(Add distcc to the FEATURES</comment>
268 FEATURES="distcc"
269 <comment>(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs
270 +1 available)</comment>
271 MAKEOPTS="-jN"
272 </pre>
273
274 <pre caption="Setting your path">
275 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
276 </pre>
277
278 </body>
279 </section>
280 <section>
281 <title>Step 2: Getting Distcc</title>
282 <body>
283
284 <p>
285 Install distcc:
286 </p>
287
288 <pre caption="Getting Distcc on the new box">
289 # <i>USE='-*' emerge --nodeps sys-devel/distcc</i>
290 </pre>
291
292 </body>
293 </section>
294 <section>
295 <title>Step 3: Setting Up Distcc</title>
296 <body>
297
298 <p>
299 Run <c>distcc-config --install</c> to setup distcc:
300 </p>
301
302 <pre caption="Final distcc setup">
303 <comment>(Substitute host1, host2, ... with the IP number(s) of the
304 participating hosts)</comment>
305 # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
306 <comment>An example: <i>/usr/bin/distcc-config --set-hosts "localhost
307 192.168.0.4 192.168.0.6"</i></comment>
308 </pre>
309
310 <p>
311 Distcc is now set up to bootstrap! Continue with the official installation
312 instructions and <e>do not forget</e> to re-emerge distcc after <c>emerge
313 system</c>. This is to make sure that all of the dependencies you want are
314 installed as well.
315 </p>
316
317 <note>
318 During bootstrap and <c>emerge system</c> distcc may not appear to be used.
319 This is expected as some ebuilds do not work well with distcc, so they
320 intentionally disable it.
321 </note>
322
323 </body>
324 </section>
325 </chapter>
326
327 <chapter>
328 <title>Troubleshooting</title>
329 <section>
330 <title>Some Packages Don't Use Distcc</title>
331 <body>
332
333 <p>
334 As you emerge various packages, you'll notice that some of them aren't being
335 distributed (and aren't being built in parallel). This may happen because the
336 package's Makefile doesn't support parallel operations or the maintainer of the
337 ebuild has explicitly disabled parallel operations due to a known problem.
338 </p>
339
340 <p>
341 Sometimes distcc might cause a package to fail to compile. If this happens
342 for you, please <uri link="http://bugs.gentoo.org">report</uri> it to us.
343 </p>
344
345 </body>
346 </section>
347 <section>
348 <title>Mixed GCC Versions</title>
349 <body>
350
351 <p>
352 If you have different GCC versions on your hosts, there will likely be very
353 weird problems. The solution is to make certain all hosts have the same GCC
354 version.
355 </p>
356
357 <p>
358 Recent Portage updates have made Portage use <c>${CHOST}-gcc</c> instead of
359 <c>gcc</c>. This means that if you're mixing i686 machines with other types
360 (i386, i586) you will run into problems. A workaround for this may be to
361 <c>export CC='gcc' CXX='c++'</c> or to put it in <path>/etc/make.conf</path>.
362 </p>
363
364 <impo>
365 Doing this explicitly redefines some behaviour of Portage and may have some
366 weird results in the future. Only do this if you're mixing CHOSTs.
367 </impo>
368
369
370 </body>
371 </section>
372 <section>
373 <title>-march=native</title>
374 <body>
375
376 <p>
377 Starting with GCC 4.3.0, the compiler supports the <c>-march=native</c> switch
378 which turns on CPU autodetection and optimizations that are worth being enabled
379 on the processor the GCC is running at. This is a problem with <c>distcc</c> as
380 it allows mixing of code optimized for different processors (like AMD Athlon and
381 Intel Pentium). <e>Don't</e> use <c>-march=native</c> or <c>-mtune=native</c> in
382 your <c>CFLAGS</c> or <c>CXXFLAGS</c> when compiling with <c>distcc</c>.
383 </p>
384
385 </body>
386 </section>
387 </chapter>
388
389 <chapter>
390 <title>Distcc Extras</title>
391 <section>
392 <title>Distcc Monitors</title>
393 <body>
394
395 <p>
396 Distcc ships with two monitors. The text-based one is always built and is
397 called <c>distccmon-text</c>. Running it for the first time can be a bit
398 confusing, but it is really quite easy to use. If you run the program with no
399 parameter it will run once. However, if you pass it a number it will update
400 every N seconds, where N is the argument you passed.
401 </p>
402
403 <p>
404 The other monitor is only turned on if you enabled <c>gtk</c> or <c>gnome</c>
405 in your <c>USE</c> flags. This one is GTK+ based, runs in an X environment
406 and it is quite lovely. For Gentoo the GUI monitor has been called
407 <c>distccmon-gui</c> for less confusion. Elsewhere it may be referred to as
408 <c>distccmon-gnome</c>.
409 </p>
410
411 <pre caption="Starting the monitors">
412 # <i>distccmon-text N</i>
413 <comment>(Or)</comment>
414 # <i>distccmon-gui</i>
415 <comment>To monitor Portage's distcc usage you can use:</comment>
416 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text N</i>
417 <comment>(Or)</comment>
418 # <i>DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-gui</i>
419 </pre>
420
421 <impo>
422 If your distcc directory is elsewhere, change the DISTCC_DIR variable
423 accordingly.
424 </impo>
425
426 </body>
427 </section>
428 </chapter>
429 </guide>

  ViewVC Help
Powered by ViewVC 1.1.20