Contents of /xml/htdocs/doc/en/distcc.xml

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.6 - (hide annotations) (download) (as text)
Tue Sep 30 08:52:17 2003 UTC (15 years ago) by swift
Branch: MAIN
Changes since 1.5: +5 -3 lines
File MIME type: application/xml
Add license information... again...

1 swift 1.1 <?xml version='1.0' encoding="UTF-8"?>
2 blubber 1.3 <!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
3 swift 1.6 <!-- $Header: /home/cvsroot/gentoo/xml/htdocs/doc/en/distcc.xml,v 1.5 2003/09/18 09:33:41 swift Exp $ -->
4 blubber 1.2
5 swift 1.1 <guide link="/doc/en/distcc.xml">
7     <title>Gentoo Distcc Documentation</title>
9     <author title="Author">
10 erwin 1.4 <mail link="lisa@gentoo.org">Lisa Seelye</mail>
11     </author>
13     <author title="Editor">
14     <mail link="vapier@gentoo.org">Mike Frysinger</mail>
15     </author>
17     <author title="Editor">
18     <mail link="erwin@gentoo.org">Erwin</mail>
19 blubber 1.2 </author>
21     <author title="Reviewer">
22 erwin 1.4 <mail link="swift@gentoo.org">Sven Vermeulen</mail>
23 swift 1.1 </author>
24 blubber 1.2
25 swift 1.1 <author title="Reviewer">
26 erwin 1.4 <mail link="blubber@gentoo.org">Tiemo Kieft</mail>
27 swift 1.1 </author>
29 swift 1.6 <license/>
31     <version>1.2.1</version>
32     <date>September 30, 2003</date>
33 swift 1.1
34 blubber 1.2 <abstract>This document serves as a HOWTO for using distcc with Gentoo.</abstract>
35 swift 1.1
36     <chapter>
37 blubber 1.2 <title>Introduction</title>
38     <section>
39     <title>What is distcc?</title>
40     <body>
41     <p>Distcc is a program designed to distribute compiling tasks across a network to participating hosts. It is comprised of a server, <c>distccd</c> and a client program, <c>distcc</c>. Distcc can work transparently with <uri link="http://ccache.samba.org">ccache</uri> and Portage with a little set up.</p>
42     </body>
43     </section>
45     <section>
46 erwin 1.4 <title>Dependencies</title>
47 blubber 1.2 <body>
48 erwin 1.4 <pre caption="Distcc dependencies">
49 swift 1.1 &gt;=sys-apps/portage-2.0.46-r11
50     &gt;=sys-devel/gcc-config-1.3.1
51 erwin 1.4 sys-apps/shadow
52 swift 1.5 <codenote>Distcc has the following optional dependencies when you have gtk in your USE flags (As of version 2.8)</codenote>
53 swift 1.1 &gt;=x11-libs/gtk+-2.2.1
54 blubber 1.2 </pre>
55     </body>
56     </section>
57 swift 1.1 </chapter>
59     <chapter>
60 blubber 1.2 <title>Setup</title>
61     <section>
62     <title>Setting up Portage to use Distcc</title>
63     <body>
64     <p>Setting up distcc is very easy to do with Portage. Follow these simple steps on each computer you want to use distcc on:</p>
65     <pre caption="Integrating Distcc and Portage">
66 swift 1.1 # <i>emerge distcc</i>
67     # <i>nano -w /etc/make.conf</i>
68 swift 1.5 </pre>
69     <note>Edit your FEATURES flags to include &quot;distcc&quot;</note>
70     <note>Set <c>DISTCC_DIR=${PORTAGE_TMPDIR}/portage/.distcc</c></note>
71     <note>You should also uncomment the DISTCC_TMPDIR line.</note>
73 swift 1.1
74 erwin 1.4 <p>Next you have to specify what hosts you want to use. To do this you can use the <c>distcc-config</c> command to set the list of hosts. Here is an example of some hosts that might be in your list:</p>
75 blubber 1.2 <pre caption="Examples of host definitions">
76 swift 1.1
79     @ @
80 blubber 1.2 </pre>
81 erwin 1.4 <p>It may all look complicated, but in most cases a variant of line 1 or 2 will work. An explanation of each line is: Line 1 is just a space-delimited list of hosts that will use default everything. Line 2 is a list of hosts that specifies the maximum number of jobs (by use of the /N) to send that host at any given time (specified with the <c>/n</c>). Since most people won't be using lines 3 or 4, I'll <uri link="http://distcc.samba.org/man/distcc_1.html">point you to</uri> the distcc docs for more information.</p>
82 blubber 1.2 <p>A sample command to set the hosts (for line 1) is:</p>
83     <pre caption="Sample command to set the hosts">
84 erwin 1.4 # <i>/usr/bin/distcc-config --set-hosts ""</i>
85 blubber 1.2 </pre>
86 swift 1.1
87 swift 1.5 <p>The final step to integrating distcc into Portage is to re-open your <path>/etc/make.conf</path> and edit <c>MAKEOPTS</c> to include <c>-jN</c> (where N is an integer). Typically you will want to set this to the total number of processors in your network plus one.</p>
88 blubber 1.2 <pre caption="Final steps in make.conf">
89 swift 1.1 # <i>nano -w /etc/make.conf</i>
90 swift 1.5 MAKEOPTS=-jN
91 blubber 1.2 </pre>
92 swift 1.5 <impo>Don't forget to start the distcc daemon! <c>/etc/init.d/distccd start</c></impo>
93 blubber 1.2 </body>
94     </section>
95     <section>
96     <title>Setting up Distcc to Work With Automake</title>
97     <body>
98 swift 1.5 <p>This is in some cases easier than the Portage setup. What you have to do is update your <c>PATH</c> environment variable to include <path>/usr/lib/distcc/bin</path> before where gcc (<path>/usr/bin</path>). However, there is a caveat. If you use ccache you have to put distcc after the ccache part. Like so:</p>
99 blubber 1.2 <pre caption="Setting your path">
100 swift 1.1 # <i>export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"</i>
101 blubber 1.2 </pre>
102     <p>Then, as you would normally type <c>make</c>, you would type <c>make -jN</c> (where N is an integer). A safe number to use for N varries on your network and the types of computers you are using to compile. For me, I have a dual processor P3 and a K6-2/400 that compile everything for my desktop (a 1200MHz Athlon Thunderbird) and I use -j5. Test your own settings to find the right number.</p>
103     </body>
104     </section>
105     </chapter>
106 swift 1.1
107 blubber 1.2 <chapter>
108     <title>Cross-Compiling</title>
109     <section>
110     <title>A Note on Cross-Compiling</title>
111     <body>
112     <p>Cross-compiling is using one architecture to build programs for another architecture. This can be as simple as using an Athlon (i686) to build a program for a K6-2 (i586), or using a Sparc to build a program for a ppc.</p>
113     </body>
114     </section>
115     <section>
116     <title>A Personal Note on Cross-compiling</title>
117     <body>
118     <p>I'd love to be able to help out on the Gentoo Cross-compiling effort, but I don't have any non-x86 machines. I can create things that work <e>in theory</e>, but I have to rely on other people to test what I write. That works to an extent, but it's tough. ;-)</p>
119     <p>In the near future I hope to acquire a Sparc of some type to put Gentoo on so I can play around in-house.</p>
120     </body>
121     </section>
122 swift 1.1 </chapter>
124     <chapter>
125 blubber 1.2 <title>Using Distcc to Bootstrap</title>
126     <section>
127     <title>Step 1: Build the Tarball</title>
128     <body>
129     <p>This section is helpful if you want to use Distcc during the Gentoo installation. Surprisingly it isn't that hard. What you'll need is another computer running Gentoo and a way to transfer a tarball to your new system (I use scp).</p>
130 erwin 1.4 <p>First you have to build a tarball of Distcc and include a library (<path>/usr/lib/libpopt.so.*</path>).</p>
131 blubber 1.2 <pre caption="Building the tarball">
132 erwin 1.4 # <i>USE='-gtk -selinux' emerge --buildpkg distcc</i>
133     # <i>mkdir -p /tmp/distcc/usr/lib</i>
134     # <i>cp /usr/portage/packages/sys-devel/distcc-DISTCC_VERSION.tbz2 /tmp/distcc/</i>
135     # <i>cp /usr/lib/libpopt.so.* /tmp/distcc/usr/lib</i>
136     # <i>cd /tmp/distcc</i>
137     # <i>tar cfzv distcc.tar.gz *</i>
138     <codenote>Be sure you replace DISTCC_VERSION with the version (and Gentoo revision) of Distcc, <i>like 2.10-r1</i></codenote>
139 blubber 1.2 </pre>
140     </body>
141     </section>
142     <section>
143     <title>Step 2: Setup on New Box</title>
144     <body>
145     <impo>Do not forget to start sshd and reset the root password for the LiveCD!</impo>
146 erwin 1.4 <p>Next, you have to boot your new box with a Gentoo Linux LiveCD and follow all of the steps up until the bootstrapping. Then proceed with a little preliminary setup on the new box.</p>
147 blubber 1.2 <pre caption="Preliminary Setup">
148     <codenote>Here we add distcc to FEATURES</codenote>
149 swift 1.5 # <i>echo "FEATURES=\"\${FEATURES} distcc\"" &gt;&gt; /etc/make.conf</i>
150 erwin 1.4 <codenote>You can also use <i>nano</i> to edit <path>/etc/make.conf</path> and manually add distcc to FEATURES.</codenote>
151 swift 1.5 # <i>echo "MAKEOPTS=\"\${MAKEOPTS} -jN\"" &gt;&gt; /etc/make.conf</i>
152 erwin 1.4 <codenote>Just like above, you can use <i>nano</i> to edit <path>/etc/make.conf</path> and manually change MAKEOPTS.</codenote>
153 blubber 1.2 </pre>
154 erwin 1.4 <p>Then add the distcc user to your <path>/etc/passwd</path>:</p>
155 blubber 1.2 <pre caption="Add the distcc user to /etc/passwd">
156 swift 1.5 # <i>echo "distcc:x:240:2:distccd:/dev/null:/bin/false" &gt;&gt;/etc/passwd</i>
157 erwin 1.4 <codenote>Do not forget the `<i>&gt;&gt;</i>'</codenote>
158 blubber 1.2 </pre>
159     </body>
160     </section>
161     <section>
162     <title>Step 3: Copy the Tarball</title>
163     <body>
164     <p>Next you will want to copy the distcc tarball that you made.</p>
165     <pre caption="Copy the tarball to the new box">
166     <codenote>Execute this on the "old" box</codenote>
167 erwin 1.4 # <i>scp /tmp/distcc/distcc.tar.gz root@ip.of.new.box:/mnt/gentoo/</i>
168     <codenote>Substitute <i>ip.of.new.box</i> for your new box's IP</codenote>
169 blubber 1.2 </pre>
170     </body>
171     </section>
173     <section>
174     <title>Step 4: Unpacking the Tarball</title>
175     <body>
176     <pre caption="Unpack the tarball">
177 erwin 1.4 # <i>tar xvfz distcc.tar.gz</i>
178     # <i>tar xvfjp distcc-DISTCC_VERSION.tbz2</i>
179     <codenote>Be sure you replace DISTCC_VERSION with the version (and Gentoo revision) of Distcc, <i>like 2.10-r1</i></codenote>
180 blubber 1.2 </pre>
181     </body>
182     </section>
184     <section>
185     <title>Step 5: Setting Up Distcc Itself</title>
186     <body>
187     <p>Now set up distcc itself...</p>
188     <pre caption="Final distcc setup">
189 erwin 1.4 # <i>/usr/bin/distcc-config --install</i>
190     # <i>/usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."</i>
191 swift 1.5 </pre>
192 erwin 1.4 <p>distcc should be set up now to bootstrap! Continue the official install guide, and don't forget to re-emerge distcc after <c>emerge system</c>.</p>
193 blubber 1.2 </body>
194     </section>
195     </chapter>
196 swift 1.1
197 swift 1.5 <chapter>
198     <title>Known Problems With Distcc</title>
199     <section>
200     <title>Mozilla and Xfree</title>
201     <body>
202     <p>As you emerge various packages, you'll notice that they aren't being distributed (and indeed aren't being built in parallel). This is because the developers of Mozilla and Xfree .ebuilds disabled parallel building because it is known to cause problems. This isn't necessarily a distcc problem.</p>
203     <p>That isn't to say that sometimes distcc will cause a package to fail to compile.</p>
204     </body>
205     </section>
206     <section>
207     <title>A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty</title>
208     <body>
209     <p>With such a long title any explanation here is almost irrelevent. However, if you plan to use distcc across hosts that have the <uri link="http://www.gentoo.org/proj/en/hardened/etdyn-ssp.xml">PaX/hardened-gcc</uri> and some that do not, you will run into problems.</p>
210     <p>The solution requires a little foresight on your part; you have to run <c>hardened-gcc -R</c> on the host that has PaX/hardened-gcc, or you have to enable PaX protections in your kernel and <c>emerge hardened-gcc</c>. Both of which are a good thing to do since for the most part the protections offered by both packages is a good thing and is transparent to the user.</p>
211     </body>
212     </section>
213     <section>
214     <title>Mixed GCC Versions</title>
215     <body>
216     <p>If you have differing versions of GCC on your hosts, there will likely be very weird problems. The solution is to make certain all hosts have the same GCC version.</p>
217     </body>
218     </section>
219     </chapter>
220 swift 1.1
221     <chapter>
222 blubber 1.2 <title>Distcc Extras</title>
223     <section>
224     <title>Distcc Monitors</title>
225     <body>
226     <p>Distcc ships with two monitors. The text-based one is always built, it is called <c>distccmon-text</c>. Running it for the first time can be a bit confusing, but it is really quite easy to use. If you run the program with no parameter it will run once. However, if you pass it a number it will update every N seconds, where N is the argument you passed.</p>
227     <p>The other monitor is only turned on if you enabled <c>gtk</c> in your <c>USE</c> flags. This one is GTK+ based and runs in an X environment, and it is quite lovely.</p>
228     <p>There is a caveat to using these programs. If you want to monitor any emerge, you have to start the monitor like so:</p>
229     <pre caption="Starting the monitors">
230 swift 1.5 # <i>DISTCC_DIR=/path/to/distccdir portage distccmon-text N</i>
231 swift 1.1 <codenote>Or you can start the graphical monitor...</codenote>
232 swift 1.5 # <i>DISTCC_DIR=/path/to/distccdir portage distccmon-gnome</i>
233 blubber 1.2 </pre>
234 swift 1.5 <note>If you aren't using Portage you do not need to specify DISTCC_DIR (i.e., just type distccmon-text N).</note>
235     <impo>If you are using Portage and used the above example, this will be <c>DISTCC_DIR=/var/tmp/portage/.distcc</c>.</impo>
236 blubber 1.2 </body>
237     </section>
238 swift 1.1 </chapter>
240     <chapter>
241 blubber 1.2 <title>Future Plans for Distcc and Gentoo</title>
242     <section>
243     <title>distcc-subnetscan</title>
244     <body>
245     <p><c>distcc-subnetscan</c> is a perl program in development that will scan a subnet for hosts that have a participating distcc daemon. Even this will be enhanced to test if the daemon on a remote host will conform to a specific <c>CHOST</c> setting to make cross-compiling easier.</p>
246     <p>The perl script is being kept at <uri link="http://dev.gentoo.org/~lisa/distcc/distcc-subnetscan/distcc-subnetscan.pl">right here</uri> until a more formal home can be found.</p>
247     </body>
248     </section>
249     <section>
250     <title>distcc-config</title>
251     <body>
252     <p><c>distcc-config</c>, the userland configuration tool for distcc is out of date. Soon it will be rewritten to be brought up the current version of distcc.</p>
253     </body>
254     </section>
255 swift 1.1 </chapter>
257     </guide>

  ViewVC Help
Powered by ViewVC 1.1.20