doc/en/source_mirrors.xml

<?xml version='1.0' encoding="UTF-8"?>
<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/source_mirrors.xml,v 1.37 2010/04/28 21:22:01 robbat2 Exp $ -->
<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">

<guide link="/doc/en/source_mirrors.xml">

<title>Gentoo Linux Source Mirrors Policy</title>

<author title="Author">
  <mail link="klieber@gentoo.org">Kurt Lieber</mail>
</author>
<author title="Editor">
  <mail link="cshields@gentoo.org">Corey Shields</mail>
</author>
<author title="Editor">
  <mail link="bennyc@gentoo.org">Benny Chuang</mail>
</author>
<author title="Editor">
  <mail link="jforman@gentoo.org">Jeffrey Forman</mail>
</author>
<author title="Editor">
  <mail link="robbat2@gentoo.org">Robin H. Johnson</mail> 
</author>
<author title="Editor">
  <mail link="fox2mike@gentoo.org">Shyam Mani</mail>
</author>

<abstract>
This document explains the requirements for source mirrors.
</abstract>

<!-- The content of this document is licensed under the CC-BY-SA license -->
<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
<license/>
<version>1.14</version>
<date>2011-05-13</date>

<chapter>
<title>Requirements</title>
<section>
<title>Minimum Bandwidth</title>
<body>

<p>
To properly host a source mirror, you should have a minimum of 5Mbps upstream
bandwidth. ADSL and cable connections do not generally make adequate
connections for distfile hosts due to their limited upstream performance.
</p>

</body>
</section>
<section>
<title>Minimum Disk Space</title>
<body>

<p>
The space requirements depend on whether you choose to run a "Full" mirror or a
"Distfiles only" mirror. Full mirrors require 240 GB of disk space.  Distfiles
only mirrors require 150 GB of space.
</p>

</body>
</section>
<section>
<title>Minimum User Count</title>
<body>

<p>
We ask that you support a minimum of 15 and preferrably 30 or more concurrent
user connections.
</p>

</body>
</section>
<section>
<title>Update Frequency</title>
<body>

<p>
All mirrors must be synchronized every four hours, starting at midnight in your
local time zone.  This helps distribute the load out over a period of time,
rather than having all mirrors synchronize at the same time.
</p>

</body>
</section>
<section>
<title>Mirror Needs</title>
<body>

<p>
Your mirror may not be added immediately to our listing and given access to the
private distribution point. We add mirrors on an as-needed basis so as not to
clutter up the listing, and to provide an adequate amount of high speed mirrors.
You are more than welcome to peek in <uri
link="irc://irc.gentoo.org/gentoo-mirrors">#gentoo-mirrors</uri> on IRC and chat
up to see the status of your mirror being added to our official network.
</p>

</body>
</section>
</chapter>

<chapter>
<title>Implementation details</title>
<section>
<body>

<p>
For new source mirror requests, the procedure will be as follows:
</p>

<p>
Set up your server to rsync with ftp.ussg.iu.edu::gentoo-distfiles. Once you
have a complete mirror of that server, please file a bug report on <uri
link="http://bugs.gentoo.org/enter_bug.cgi?product=Mirrors">bugs.gentoo.org</uri>
under the <c>Mirrors</c> category stating that you have a new mirror and wish
it to be added to the list of official mirrors.  Please specify whether you
wish to be a "Full" mirror or "Distfiles only" mirror. Distfiles
only mirrors will be designated as such on the website.
</p>

<p>
Full mirrors must carry the following directories (plus any directories that
get added in the future):
</p>

<ul>
  <li>/releases (45-55G)</li>
  <li>/experimental (20-30G)</li>
  <li>/distfiles (140G)</li>
  <li>/snapshots (600M)</li>
</ul>

<p>
Distfiles only mirrors need to carry the following directories:
</p>

<ul>
  <li>/distfiles (110G)</li>
  <li>/snapshots (600M)</li>
</ul>

<p>
We will then verify that the mirror exists and we can connect to it properly.
Then, in the next 2 weeks, we will determine if your mirror is syncing
correctly and in accordance with this policy.
</p>

<p>
If all steps went smoothly, we will then add you to our list of <uri
link="/main/en/mirrors2.xml">download mirrors</uri> on the main web site.  We
will also send you the login information for our private master distfile mirror
which is only available to official Gentoo Linux source mirrors.
</p>

<p>
Additionally, you, the mirror admin, will be requested to subscribe to the
gentoo-mirrors mailing list (low traffic) so that you can follow all issues
associated with source mirrors.
</p>

<note>
Thanks for helping out Gentoo Linux users and developers! :) For any mirror
administration issues or problems, please contact <mail
link="mirror-admin@gentoo.org">mirror-admin@gentoo.org</mail>.
</note>

<warn>
Do NOT email mirror-admin@gentoo.org for mirror requests, please file a bug. It
makes mirror tracking easier on us.
</warn>

</body>
</section>
</chapter>

<chapter>
<title>Short FAQ</title>
<section>
<title>
Q: Who should I contact regarding new mirrors or to get my mirror added to the
Gentoo web site?
</title>
<body>

<p>
A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
</p>

</body>
</section>
<section>
<title>Q: Who should I contact regarding mirror issues and maintenance?</title>
<body>

<p>
A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
</p>

</body>
</section>
<section>
<title>
Q: I run a private source mirror for my company.  Can I still access the
private master source mirror?
</title>
<body>

<p>
A: Because our resources are limited, we need to ensure we allocate them in
such a way to provide the maximum amount of benefit to our users.  As such, we
limit connections to our master rsync and distfile mirrors to public mirrors
only.  Users are welcome to use our regular mirror system to establish a
private source mirror.
</p>

</body>
</section>
<section>
<title>Q: How do I find the mirror nearest to me?</title>
<body>

<p>
A: mirrorselect was designed to do this for you. If you haven't already run
<c>emerge mirrorselect</c>, then please do so.  Then run: <c>mirrorselect</c>.
An ncurses-based menu will display allowing you to select a mirror close to
you.
</p>

</body>
</section>
<section>
<title>
Q: I don't have that much disk space.  Can I mirror part of the sources?
</title>
<body>

<p>
A: Yes, if you are unable to mirror our full sources, you can elect to mirror
the following two directories only:
</p>

<ul>
  <li>/distfiles</li>
  <li>/snapshots</li>
</ul>

<p>
These two directories will require approximately 50GB of space and you will be
listed on our site as a distfiles-only mirror.
</p>

</body>
</section>
<section>
<title>
Q: Do I have to turn on Directory Indexing for directories served up via HTTP?
</title>
<body>

<p>
A: Directory Indexing is required for the /releases and /experimental
directories, but can be turned off for all other directories (including
/distfiles) if you wish.
</p>

</body>
</section>

<section>
<title>Q: Do I have to mirror historical? I mean, it is old releases.</title>
<body>

<p>
A: Of course not, it is quite a bit of data. In your rsync command, add
<path>--exclude=/releases/historical</path> to your rsync options.  This will
exclude historical and save you quite a bit of space.
</p>

</body>
</section>
</chapter>
</guide>
1	klieber	1.1	<?xml version='1.0' encoding="UTF-8"?>
2	darkside	1.38	<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/source_mirrors.xml,v 1.37 2010/04/28 21:22:01 robbat2 Exp $ -->
3	klieber	1.1	<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
4
5	klieber	1.3	<guide link="/doc/en/source_mirrors.xml">
6	neysx	1.29
7	klieber	1.1	<title>Gentoo Linux Source Mirrors Policy</title>
8	neysx	1.29
9	neysx	1.19	<author title="Author">
10			<mail link="klieber@gentoo.org">Kurt Lieber</mail>
11			</author>
12			<author title="Editor">
13			<mail link="cshields@gentoo.org">Corey Shields</mail>
14			</author>
15			<author title="Editor">
16			<mail link="bennyc@gentoo.org">Benny Chuang</mail>
17			</author>
18			<author title="Editor">
19			<mail link="jforman@gentoo.org">Jeffrey Forman</mail>
20			</author>
21	robbat2	1.33	<author title="Editor">
22			<mail link="robbat2@gentoo.org">Robin H. Johnson</mail>
23			</author>
24			<author title="Editor">
25			<mail link="fox2mike@gentoo.org">Shyam Mani</mail>
26			</author>
27	klieber	1.1
28	swift	1.15	<abstract>
29			This document explains the requirements for source mirrors.
30			</abstract>
31	neysx	1.19
32	neysx	1.30	<!-- The content of this document is licensed under the CC-BY-SA license -->
33			<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
34	swift	1.15	<license/>
35	darkside	1.38	<version>1.14</version>
36			<date>2011-05-13</date>
37	swift	1.13
38	klieber	1.1	<chapter>
39			<title>Requirements</title>
40			<section>
41			<title>Minimum Bandwidth</title>
42			<body>
43	neysx	1.19
44			<p>
45			To properly host a source mirror, you should have a minimum of 5Mbps upstream
46	neysx	1.29	bandwidth. ADSL and cable connections do not generally make adequate
47	neysx	1.19	connections for distfile hosts due to their limited upstream performance.
48			</p>
49
50	klieber	1.1	</body>
51			</section>
52			<section>
53	cshields	1.16	<title>Minimum Disk Space</title>
54			<body>
55	neysx	1.19
56			<p>
57			The space requirements depend on whether you choose to run a "Full" mirror or a
58	darkside	1.38	"Distfiles only" mirror. Full mirrors require 240 GB of disk space. Distfiles
59			only mirrors require 150 GB of space.
60	neysx	1.19	</p>
61
62	cshields	1.16	</body>
63			</section>
64			<section>
65	klieber	1.1	<title>Minimum User Count</title>
66			<body>
67	neysx	1.19
68			<p>
69	robbat2	1.36	We ask that you support a minimum of 15 and preferrably 30 or more concurrent
70			user connections.
71	neysx	1.19	</p>
72
73	klieber	1.1	</body>
74			</section>
75			<section>
76			<title>Update Frequency</title>
77			<body>
78	neysx	1.19
79			<p>
80			All mirrors must be synchronized every four hours, starting at midnight in your
81			local time zone. This helps distribute the load out over a period of time,
82			rather than having all mirrors synchronize at the same time.
83			</p>
84	neysx	1.29
85	klieber	1.1	</body>
86			</section>
87	jforman	1.26	<section>
88			<title>Mirror Needs</title>
89			<body>
90	neysx	1.29
91	jforman	1.26	<p>
92	neysx	1.29	Your mirror may not be added immediately to our listing and given access to the
93			private distribution point. We add mirrors on an as-needed basis so as not to
94	nightmorph	1.32	clutter up the listing, and to provide an adequate amount of high speed mirrors.
95			You are more than welcome to peek in <uri
96			link="irc://irc.gentoo.org/gentoo-mirrors">#gentoo-mirrors</uri> on IRC and chat
97			up to see the status of your mirror being added to our official network.
98	jforman	1.26	</p>
99	neysx	1.29
100	jforman	1.26	</body>
101			</section>
102	klieber	1.1	</chapter>
103	neysx	1.19
104	klieber	1.1	<chapter>
105			<title>Implementation details</title>
106			<section>
107			<body>
108	neysx	1.19
109			<p>
110			For new source mirror requests, the procedure will be as follows:
111			</p>
112
113			<p>
114	neysx	1.30	Set up your server to rsync with ftp.ussg.iu.edu::gentoo-distfiles. Once you
115			have a complete mirror of that server, please file a bug report on <uri
116			link="http://bugs.gentoo.org/enter_bug.cgi?product=Mirrors">bugs.gentoo.org</uri>
117			under the <c>Mirrors</c> category stating that you have a new mirror and wish
118			it to be added to the list of official mirrors. Please specify whether you
119			wish to be a "Full" mirror or "Distfiles only" mirror. Distfiles
120	neysx	1.23	only mirrors will be designated as such on the website.
121	neysx	1.19	</p>
122
123			<p>
124			Full mirrors must carry the following directories (plus any directories that
125			get added in the future):
126			</p>
127
128	cshields	1.16	<ul>
129	darkside	1.35	<li>/releases (45-55G)</li>
130	robbat2	1.33	<li>/experimental (20-30G)</li>
131	darkside	1.38	<li>/distfiles (140G)</li>
132	darkside	1.35	<li>/snapshots (600M)</li>
133	cshields	1.16	</ul>
134	neysx	1.19
135			<p>
136			Distfiles only mirrors need to carry the following directories:
137			</p>
138
139	cshields	1.16	<ul>
140	darkside	1.35	<li>/distfiles (110G)</li>
141			<li>/snapshots (600M)</li>
142	cshields	1.16	</ul>
143
144	neysx	1.19	<p>
145			We will then verify that the mirror exists and we can connect to it properly.
146	robbat2	1.34	Then, in the next 2 weeks, we will determine if your mirror is syncing
147	neysx	1.19	correctly and in accordance with this policy.
148			</p>
149	klieber	1.1
150	neysx	1.19	<p>
151			If all steps went smoothly, we will then add you to our list of <uri
152	robbat2	1.34	link="/main/en/mirrors2.xml">download mirrors</uri> on the main web site. We
153	neysx	1.23	will also send you the login information for our private master distfile mirror
154			which is only available to official Gentoo Linux source mirrors.
155	neysx	1.29	</p>
156	neysx	1.19
157			<p>
158			Additionally, you, the mirror admin, will be requested to subscribe to the
159			gentoo-mirrors mailing list (low traffic) so that you can follow all issues
160			associated with source mirrors.
161			</p>
162	klieber	1.1
163			<note>
164	neysx	1.19	Thanks for helping out Gentoo Linux users and developers! :) For any mirror
165			administration issues or problems, please contact <mail
166			link="mirror-admin@gentoo.org">mirror-admin@gentoo.org</mail>.
167	klieber	1.1	</note>
168	neysx	1.19
169	jforman	1.21	<warn>
170	neysx	1.29	Do NOT email mirror-admin@gentoo.org for mirror requests, please file a bug. It
171			makes mirror tracking easier on us.
172	jforman	1.21	</warn>
173
174	klieber	1.1	</body>
175			</section>
176			</chapter>
177	neysx	1.19
178	klieber	1.1	<chapter>
179			<title>Short FAQ</title>
180			<section>
181	neysx	1.29	<title>
182			Q: Who should I contact regarding new mirrors or to get my mirror added to the
183			Gentoo web site?
184			</title>
185	klieber	1.1	<body>
186	neysx	1.19
187			<p>
188	neysx	1.30	A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
189	neysx	1.19	</p>
190
191	klieber	1.1	</body>
192			</section>
193			<section>
194			<title>Q: Who should I contact regarding mirror issues and maintenance?</title>
195			<body>
196	neysx	1.19
197			<p>
198	neysx	1.30	A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
199	neysx	1.29	</p>
200	neysx	1.19
201	klieber	1.1	</body>
202			</section>
203	klieber	1.9	<section>
204	neysx	1.29	<title>
205			Q: I run a private source mirror for my company. Can I still access the
206			private master source mirror?
207			</title>
208	klieber	1.9	<body>
209	neysx	1.19
210			<p>
211			A: Because our resources are limited, we need to ensure we allocate them in
212			such a way to provide the maximum amount of benefit to our users. As such, we
213			limit connections to our master rsync and distfile mirrors to public mirrors
214			only. Users are welcome to use our regular mirror system to establish a
215			private source mirror.
216	klieber	1.9	</p>
217	neysx	1.19
218	klieber	1.9	</body>
219			</section>
220	klieber	1.1	<section>
221	klieber	1.9	<title>Q: How do I find the mirror nearest to me?</title>
222	klieber	1.1	<body>
223	neysx	1.19
224			<p>
225			A: mirrorselect was designed to do this for you. If you haven't already run
226			<c>emerge mirrorselect</c>, then please do so. Then run: <c>mirrorselect</c>.
227			An ncurses-based menu will display allowing you to select a mirror close to
228			you.
229	klieber	1.1	</p>
230	neysx	1.19
231	klieber	1.1	</body>
232			</section>
233	klieber	1.5	<section>
234	neysx	1.29	<title>
235			Q: I don't have that much disk space. Can I mirror part of the sources?
236			</title>
237	klieber	1.11	<body>
238	neysx	1.19
239			<p>
240			A: Yes, if you are unable to mirror our full sources, you can elect to mirror
241			the following two directories only:
242			</p>
243
244	klieber	1.11	<ul>
245	neysx	1.28	<li>/distfiles</li>
246			<li>/snapshots</li>
247	klieber	1.11	</ul>
248	neysx	1.19
249			<p>
250	neysx	1.31	These two directories will require approximately 50GB of space and you will be
251	neysx	1.19	listed on our site as a distfiles-only mirror.
252			</p>
253
254	klieber	1.5	</body>
255			</section>
256	klieber	1.8	<section>
257	neysx	1.29	<title>
258			Q: Do I have to turn on Directory Indexing for directories served up via HTTP?
259			</title>
260	klieber	1.8	<body>
261	neysx	1.19
262			<p>
263			A: Directory Indexing is required for the /releases and /experimental
264			directories, but can be turned off for all other directories (including
265	neysx	1.29	/distfiles) if you wish.
266	klieber	1.8	</p>
267	neysx	1.29
268	jforman	1.24	</body>
269			</section>
270
271			<section>
272			<title>Q: Do I have to mirror historical? I mean, it is old releases.</title>
273			<body>
274	neysx	1.19
275	jforman	1.24	<p>
276	neysx	1.25	A: Of course not, it is quite a bit of data. In your rsync command, add
277			<path>--exclude=/releases/historical</path> to your rsync options. This will
278			exclude historical and save you quite a bit of space.
279	jforman	1.24	</p>
280	neysx	1.25
281	klieber	1.8	</body>
282			</section>
283	klieber	1.1	</chapter>
284			</guide>