/[gentoo]/xml/htdocs/doc/en/distcc.xml
Gentoo

Diff of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

Revision 1.1 Revision 1.2
1<?xml version='1.0' encoding="UTF-8"?> 1<?xml version='1.0' encoding="UTF-8"?>
2<!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> 2<!DOCTYPE guide SYSTEM "htdocs/dtd/guide.dtd">
3<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.2 2003/09/08 07:04:10 blubber Exp $ -->
4
3<guide link="/doc/en/distcc.xml"> 5<guide link="/doc/en/distcc.xml">
4 6
5<title>Gentoo Distcc Documentation</title> 7<title>Gentoo Distcc Documentation</title>
6 8
7<author title="Author"> 9<author title="Author">
8 <mail link="lisa@gentoo.org">Lisa M. Seelye</mail> 10 <mail link="lisa@gentoo.org">lisa@gentoo.org</mail>
9</author> 11</author>
12
10<author title="Reviewer"> 13<author title="Reviewer">
11 <mail link="swift@gentoo.org">Sven Vermeulen</mail> 14 <mail link="swift@gentoo.org">swift@gentoo.org</mail>
12</author> 15</author>
16
17<author title="Reviewer">
18 <mail link="blubber@gentoo.org">blubber@gentoo.org</mail>
19</author>
20
13<version>1.0</version> 21<version>1.1</version>
14<date>5th of August 2003</date> 22<date>September 08, 2003</date>
15 23
16<license/>
17
18<abstract>
19This document serves as a HOWTO for using distcc with gentoo. 24<abstract>This document serves as a HOWTO for using distcc with Gentoo.</abstract>
20</abstract>
21 25
22<chapter> 26<chapter>
23<title>Introduction</title> 27 <title>Introduction</title>
24<section> 28 <section>
25<title>What is distcc?</title> 29 <title>What is distcc?</title>
26<body> 30 <body>
27<p> 31 <p>Distcc is a program designed to distribute compiling tasks across a network to participating hosts. It is comprised of a server, <c>distccd</c> and a client program, <c>distcc</c>. Distcc can work transparently with <uri link="http://ccache.samba.org">ccache</uri> and Portage with a little set up.</p>
28Distcc is a program designed to distribute compiling tasks across a
29network to participating hosts. It is comprised of a server, <c>distccd</c>
30and a client program, <c>distcc</c>. Distcc can work transparently with
31<uri link="http://ccache.samba.org">ccache</uri> and Portage with a little
32set up.
33</p>
34
35</body> 32 </body>
36</section> 33 </section>
37 34
38<section> 35 <section>
39<title>Dependancies</title> 36 <title>Dependancies</title>
40<body> 37 <body>
41<pre caption="Distcc Dependancies"> 38 <pre caption="Distcc Dependancies">
42&gt;=sys-apps/portage-2.0.46-r11 39&gt;=sys-apps/portage-2.0.46-r11
43&gt;=sys-devel/gcc-config-1.3.1 40&gt;=sys-devel/gcc-config-1.3.1
44&gt;sys-apps/shadow 41&gt;sys-apps/shadow
45<codenote>(As of version 2.8) And the following optional dependancies when you have <c>gtk</c> in your <c>USE</c> flags</codenote> 42<codenote>(As of version 2.8) And the following optional dependancies when you have <c>gtk</c> in your <c>USE</c> flags</codenote>
46&gt;=x11-libs/gtk+-2.2.1 43&gt;=x11-libs/gtk+-2.2.1
47</pre> 44 </pre>
48
49</body> 45 </body>
50</section> 46 </section>
51</chapter>
52
53<chapter> 47</chapter>
48
49<chapter>
54<title>Setup</title> 50 <title>Setup</title>
55<section> 51 <section>
56<title>Setting up Portage to use Distcc</title> 52 <title>Setting up Portage to use Distcc</title>
57<body> 53 <body>
58 54 <p>Setting up distcc is very easy to do with Portage. Follow these simple steps on each computer you want to use distcc on:</p>
59<p>
60Setting up distcc is very easy to do with Portage. Follow these simple
61steps on each computer you want to use distcc on:
62</p>
63
64<pre caption="Integrating Distcc and Portage"> 55 <pre caption="Integrating Distcc and Portage">
65# <i>emerge distcc</i> 56# <i>emerge distcc</i>
66# <i>nano -w /etc/make.conf</i> 57# <i>nano -w /etc/make.conf</i>
67<codenote>Edit your FEATURES flags to include &quot;distcc&quot;</codenote> 58<codenote>Edit your FEATURES flags to include &quot;distcc&quot;</codenote>
68</pre> 59 </pre>
69 60
70<p> 61 <p>Next you have to specify what hosts you want to use. To do this you can use the <i>distcc-config</i> command to set that list of hosts. Here is an example of some hosts that might be in your list:</p>
71Next you have to specify what hosts you want to use. To do this you can
72use the <i>distcc-config</i> command to set that list of hosts. Here is an
73example of some hosts that might be in your list:
74</p>
75
76<pre caption="Examples of host definitions"> 62 <pre caption="Examples of host definitions">
77192.168.0.1 192.168.0.2 192.168.0.3 63192.168.0.1 192.168.0.2 192.168.0.3
78192.168.0.1/2 192.168.0.2 192.168.0.3/10 64192.168.0.1/2 192.168.0.2 192.168.0.3/10
79192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4 65192.168.0.1:4000/2 192.168.0.2/1 192.168.0.3:3632/4
80@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3 66@192.168.0.1 @192.168.0.2:/usr/bin/distccd 192.168.0.3
81</pre> 67 </pre>
68 <p>It may all look complicated, but in most cases a variant of line 1 or 2 will work. An explanation of each line is: Line 1 is just a space-delimited list of hosts that will use default everything. Line 2 is a list of hosts that specifies the maxmimum number of jobs (by use of the /N) to send that host at any given time (specified with the <c>/n</c>). Since most people won't be using lines 3 or 4, I'll <uri link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc docs for more information.</p>
69 <p>A sample command to set the hosts (for line 1) is:</p>
70 <pre caption="Sample command to set the hosts">
71# <i>/usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3</i>
72 </pre>
82 73
83<p> 74 <p>The final step to integrating distcc into Portage is to re-open your <path>/etc/make.conf</path> and edit <c>MAKE_OPTS</c> to include <c>-jN</c> (where N is an integer). Typically you will want to set this to the total number of processors in your network plus one.</p>
84It may all look complicated, but in most cases a variant of line 1
85or 2 will work. An explanation of each line is: Line 1 is just a
86space-delimited list of hosts that will use default everything.
87Line 2 is a list of hosts that specifies the maximum number of jobs (by
88use of the /N) to send that host at any given time. Since most people
89won't be using lines 3 or 4, I'll point you to the <uri
90link="http://distcc.samba.org/man/distcc_1.html">distcc docs</uri> for
91more information.
92</p>
93
94<p>
95The final step to integrating distcc into Portage is to re-open your
96<path>/etc/make.conf</path> and edit <c>MAKE_OPTS</c> to include <c>-jN</c>
97(where N is an integer). Typically you will want to set this to the total
98number of processors in your network plus one.
99</p>
100
101<pre caption="Final steps in make.conf"> 75 <pre caption="Final steps in make.conf">
102# <i>nano -w /etc/make.conf</i> 76# <i>nano -w /etc/make.conf</i>
103MAKE_OPTS=-jN 77MAKE_OPTS=-jN
104</pre> 78 </pre>
105
106</body> 79 </body>
107</section> 80 </section>
108
109<section> 81 <section>
110<title>Setting up Distcc to Work With Automake</title> 82 <title>Setting up Distcc to Work With Automake</title>
111<body> 83 <body>
112 84 <p>This is in some cases easier than the Portage setup. What you have to do is update your <c>PATH</c> envvar to include <path>/usr/lib/distcc/bin</path> before where gcc (<path>/usr/bin</path>). However, there is a caveat. If you use ccache you have to put distcc after the ccache part. Like so:</p>
113<p>
114This is in some cases easier than the Portage setup. What you have to do
115is update your <c>PATH</c> environment variable to include
116<path>/usr/lib/distcc/bin</path> before the directory where gcc is
117situated (<path>/usr/bin</path>). However, there is a caveat. If you use
118ccache you have to put distcc after the ccache part. Like so:
119</p>
120
121<pre caption="Setting your path"> 85 <pre caption="Setting your path">
122# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i> 86# <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
123</pre> 87 </pre>
124 88 <p>Then, as you would normally type <c>make</c>, you would type <c>make -jN</c> (where N is an integer). A safe number to use for N varries on your network and the types of computers you are using to compile. For me, I have a dual processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz Athlon Thunderbird) and I use -j5. Test your own settings to find the right number.</p>
125<p>
126Then, as you would normally type <c>make</c>, you would type <c>make -jN</c>
127(where N is an integer). A safe number to use for N varies on your network
128and the types of computers you are using to compile. For me, I have a dual
129processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz
130Athlon Thunderbird) and I use -j5. Test your own settings to find the right
131number.
132</p>
133
134</body> 89 </body>
135</section> 90 </section>
136</chapter>
137
138<chapter> 91</chapter>
92
93<chapter>
139<title>Cross-Compiling</title> 94 <title>Cross-Compiling</title>
140<section> 95 <section>
141<title>A Note on Cross-Compiling</title> 96 <title>A Note on Cross-Compiling</title>
142<body> 97 <body>
143 98 <p>Cross-compiling is using one architecture to build programs for another architecture. This can be as simple as using an Athlon (i686) to build a program for a K6-2 (i586), or using a Sparc to build a program for a ppc.</p>
144<p>
145Cross-compiling is using one architecture to build programs for another
146architecture. This can be as simple as using an Athlon (i686) to build a
147program for a K6-2 (i586), or using a Sparc to build a program for a ppc.
148</p>
149
150</body> 99 </body>
151</section> 100 </section>
152
153<section> 101 <section>
154<title>A Personal Note on Cross-compiling</title> 102 <title>A Personal Note on Cross-compiling</title>
155<body> 103 <body>
156 104 <p>I'd love to be able to help out on the Gentoo Cross-compiling effort, but I don't have any non-x86 machines. I can create things that work <e>in theory</e>, but I have to rely on other people to test what I write. That works to an extent, but it's tough. ;-)</p>
157<p>
158I'd love to be able to help out on the Gentoo Cross-compiling effort, but
159I don't have any non-x86 machines. I can create things that work
160<e>in theory</e>, but I have to rely on other people to test what I write.
161That works to an extent, but it's tough. ;-)
162</p>
163
164<p>
165In the near future I hope to acquire a Sparc of some type to put Gentoo 105 <p>In the near future I hope to acquire a Sparc of some type to put Gentoo on so I can play around in-house.</p>
166on so I can play around in-house.
167</p>
168
169</body> 106 </body>
170</section> 107 </section>
171</chapter>
172
173<chapter> 108</chapter>
109
110<chapter>
174<title>Known Problems With Distcc</title> 111 <title>Known Problems With Distcc</title>
175
176<section> 112 <section>
177<title>Portage Doesn't Work With Distcc</title> 113 <title>Portage Doesn't Work With Distcc</title>
178<body> 114 <body>
179 115 <p>This heading is a misnomer, really. The only thing that doens't work is the monitoring programs (<c>distccmon-text</c> and <c>distccmon-gnome</c>) with the standard distcc source. The reason for this is that distcc relies on the <c>TMPDIR</c> envvar, but Portage reassigns that.</p>
180<p> 116 <p>The solution to this is a patch that changes the dependancy to <c>DISTCC_TMPDIR</c>. The patch is automatically applied to distcc. If you don't want to use <path>/tmp</path> (the default) you can reassign <c>DISTCC_TMPDIR</c>.</p>
181This heading is a misnomer, really. The only thing that doens't work is 117 <p>You may also notice that Distcc does not work with some packages. This can be a result of several things, such as an error in the .ebuild (Using <c>make</c> instead of <c>emake</c>) or an error in the program's Makefile, which the next section addresses.</p>
182the monitoring programs (<c>distccmon-text</c> and <c>distccmon-gnome</c>)
183with the standard distcc source. The reason for this is that distcc relies
184on the <c>TMPDIR</c> environment variable, but Portage reassigns that.
185</p>
186
187<p>
188The solution to this is a patch that changes the dependancy to
189<c>DISTCC_TMPDIR</c>. The patch is automatically applied to distcc.
190If you don't want to use <path>/tmp</path> (the default) you can reassign
191<c>DISTCC_TMPDIR</c>.
192</p>
193
194<p>
195You may also notice that distcc does not work with some packages.
196This can be a result of several things, such as an error in the .ebuild
197(Using <c>make</c> instead of <c>emake</c>) or an error in the program's
198Makefile, which the next section addresses.
199</p>
200
201</body> 118 </body>
202</section> 119 </section>
203<section> 120 <section>
204<title>Mozilla and XFree</title> 121 <title>Mozilla and Xfree</title>
205<body> 122 <body>
206 123 <p>As you emerge various packages, you'll notice that they aren't being distributed (and indeed aren't being built in parallel). This is because the developers of Mozilla and Xfree .ebuilds disabled parallel building because it is known to cause problems. This isn't necessarily a distcc problem.</p>
207<p>
208As you emerge various packages, you'll notice that they aren't being
209distributed (and indeed aren't being built in parallel). This is because
210the developers of Mozilla and XFree .ebuilds disabled parallel building
211because it is known to cause problems. This isn't necessarily a distcc
212problem.
213</p>
214
215<p>
216That isn't to say that sometimes distcc will cause a package to fail 124 <p>That isn't to say that sometimes distcc will cause a package to fail to compile.</p>
217to compile.
218</p>
219
220</body> 125 </body>
221</section> 126 </section>
222</chapter>
223
224<chapter> 127</chapter>
128
129<chapter>
130 <title>Using Distcc to Bootstrap</title>
131 <section>
132 <title>Step 1: Build the Tarball</title>
133 <body>
134 <p>This section is helpful if you want to use Distcc during the Gentoo installation. Surprisingly it isn't that hard. What you'll need is another computer running Gentoo and a way to transfer a tarball to your new system (I use scp).</p>
135 <p>First you have to build a tarball of Distcc and include a library (<c>/usr/lib/libpopt.so.*</c>).</p>
136 <pre caption="Building the tarball">
137<i># USE='-gtk -selinux' emerge --buildpkg distcc
138# mkdir -p /tmp/distcc/usr/lib
139# cp /usr/portage/packages/sys-devel/distcc-DISTCC_VERSION.tbz2 /tmp/distcc/
140# cp /usr/lib/libpopt.so.* /tmp/distcc/usr/lib
141# cd /tmp/distcc
142# tar cfzv distcc.tar.gz *</i>
143<codenote>Be sure you replace DISTCC_VERSION with the version (and Gentoo revision) of Distcc, <c>like 2.10-r1</c></codenote>
144 </pre>
145 </body>
146 </section>
147 <section>
148 <title>Step 2: Setup on New Box</title>
149 <body>
150 <impo>Do not forget to start sshd and reset the root password for the LiveCD!</impo>
151 <p>Next, you have to boot your new box with a Gentoo Linux LiveCD and follow all of the steps up until the bootstrapping. Then procede with a little preliminary setup on the new box.</p>
152 <pre caption="Preliminary Setup">
153<codenote>Here we add distcc to FEATURES</codenote>
154<i># echo "FEATURES=\"\${FEATURES} distcc\"" >> /etc/make.conf</i>
155<codenote>You can also use <c>nano</c> to edit <c>/etc/make.conf</c> and manually add distcc to FEATURES.</codenote>
156<i># echo "MAKEOPTS=\"\${MAKEOPTS} -jN\"" >> /etc/make.conf</i>
157<codenote>Just like above, you can use <c>nano</c> to edit <c>/etc/make.conf</c> and manually change MAKEOPTS.</codenote>
158 </pre>
159 <p>Then add the distcc user to your <c>/etc/passwd</c>:</p>
160 <pre caption="Add the distcc user to /etc/passwd">
161<i># echo "distcc:x:240:2:distccd:/dev/null:/bin/false" >>/etc/passwd</i>
162<codenote>Do not forget the `<c>&gt;&gt;</c>'</codenote>
163 </pre>
164 </body>
165 </section>
166 <section>
167 <title>Step 3: Copy the Tarball</title>
168 <body>
169 <p>Next you will want to copy the distcc tarball that you made.</p>
170 <pre caption="Copy the tarball to the new box">
171<codenote>Execute this on the "old" box</codenote>
172<i># scp /tmp/distcc/distcc.tar.gz root@ip.of.new.box:/mnt/gentoo/</i>
173<codenote>Substitute <c>ip.of.new.box</c> for your new box's IP</codenote>
174 </pre>
175 </body>
176 </section>
177
178 <section>
179 <title>Step 4: Unpacking the Tarball</title>
180 <body>
181 <pre caption="Unpack the tarball">
182<i># tar xvfz distcc.tar.gz
183# tar xvfjp distcc-DISTCC_VERSION.tbz2</i>
184<codenote>Be sure you replace DISTCC_VERSION with the version (and Gentoo revision) of Distcc, <c>like 2.10-r1</c></codenote>
185 </pre>
186 </body>
187 </section>
188
189 <section>
190 <title>Step 5: Setting Up Distcc Itself</title>
191 <body>
192 <p>Now set up distcc itself...</p>
193 <pre caption="Final distcc setup">
194<i># /usr/bin/distcc-config --install
195# /usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
196 </pre>
197
198 <p>distcc should be set up now to bootstrap! Continue the official install guide, and dont forget to re-emerge distcc after <c>emerge system</c>.</p>
199 </body>
200 </section>
201</chapter>
202
203
204<chapter>
225<title>Distcc Extras</title> 205 <title>Distcc Extras</title>
226<section> 206 <section>
227<title>Distcc Monitors</title> 207 <title>Distcc Monitors</title>
228<body> 208 <body>
229 209 <p>Distcc ships with two monitors. The text-based one is always built, it is called <c>distccmon-text</c>. Running it for the first time can be a bit confusing, but it is really quite easy to use. If you run the program with no parameter it will run once. However, if you pass it a number it will update every N seconds, where N is the argument you passed.</p>
230<p> 210 <p>The other monitor is only turned on if you enabled <c>gtk</c> in your <c>USE</c> flags. This one is GTK+ based and runs in an X environment, and it is quite lovely.</p>
231Distcc ships with two monitors. The text-based one is always built and 211 <p>There is a caveat to using these programs. If you want to monitor any emerge, you have to start the monitor like so:</p>
232is called <c>distccmon-text</c>. Running it for the first time can be a
233bit confusing, but it is really quite easy to use. If you run the program
234with no parameter it will run once. However, if you pass it a number it
235will update every N seconds, where N is the argument you passed.
236</p>
237
238<p>
239The other monitor is only turned on if you enabled <c>gtk</c> in your
240<c>USE</c> flags. This one is GTK+ based and runs in an X environment, and
241it is quite lovely.
242</p>
243
244<p>
245There is a caveat to using these programs. If you want to monitor any
246emerge, you have to start the monitor like so:
247</p>
248
249<pre caption="Starting the monitors"> 212 <pre caption="Starting the monitors">
250# <i>sudo -u portage distccmon-text N</i> 213<i>sudo -u portage distccmon-text N</i>
251<codenote>Or you can start the graphical monitor...</codenote> 214<codenote>Or you can start the graphical monitor...</codenote>
252# <i>sudo -u portage distccmon-gnome</i> 215<i>sudo -u portage distccmon-gnome</i>
216<note>If you aren't using <c>userpriv</c> in your FEATURES, you should not use the '-u portage' part.</note>
253</pre> 217 </pre>
254
255</body> 218 </body>
256</section> 219 </section>
257</chapter>
258
259<chapter> 220</chapter>
221
222<chapter>
260<title>Future Plans for Distcc and Gentoo</title> 223 <title>Future Plans for Distcc and Gentoo</title>
261<section> 224 <section>
262<title>distcc-subnetscan</title> 225 <title>distcc-subnetscan</title>
263<body> 226 <body>
264 227 <p><c>distcc-subnetscan</c> is a perl program in development that will scan a subnet for hosts that have a participating distcc daemon. Even this will be enhanced to test if the daemon on a remote host will conform to a specific <c>CHOST</c> setting to make cross-compiling easier.</p>
265<p>
266<c>distcc-subnetscan</c> is a perl program in development that will scan a
267subnet for hosts that have a participating distcc daemon. Even this will
268be enhanced to test if the daemon on a remote host will conform to a
269specific <c>CHOST</c> setting to make cross-compiling easier.
270</p>
271
272<p>
273The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">right here</uri> 228 <p>The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">right here</uri> until a more formal home can be found.</p>
274until a more formal home can be found.
275</p>
276
277</body> 229 </body>
278</section> 230 </section>
279
280<section> 231 <section>
281<title>distcc-config</title> 232 <title>distcc-config</title>
282<body> 233 <body>
283 234 <p><c>distcc-config</c>, the userland configuration tool for distcc is out of date. Soon it will be rewritten to be brought up the current version of distcc.</p>
284<p>
285<c>distcc-config</c>, the userland configuration tool for distcc is out of
286date. Soon it will be rewritten to be brought up with the current version
287of distcc.
288</p>
289
290</body> 235 </body>
291</section> 236 </section>
292</chapter> 237</chapter>
293 238
294</guide> 239</guide>

Legend:
Removed from v.1.1  
changed lines
  Added in v.1.2

  ViewVC Help
Powered by ViewVC 1.1.20