doc/en/source_mirrors.xml

<?xml version='1.0' encoding="UTF-8"?>
<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/source_mirrors.xml,v 1.34 2009/11/19 18:08:04 robbat2 Exp $ -->
<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">

<guide link="/doc/en/source_mirrors.xml">

<title>Gentoo Linux Source Mirrors Policy</title>

<author title="Author">
  <mail link="klieber@gentoo.org">Kurt Lieber</mail>
</author>
<author title="Editor">
  <mail link="cshields@gentoo.org">Corey Shields</mail>
</author>
<author title="Editor">
  <mail link="bennyc@gentoo.org">Benny Chuang</mail>
</author>
<author title="Editor">
  <mail link="jforman@gentoo.org">Jeffrey Forman</mail>
</author>
<author title="Editor">
  <mail link="robbat2@gentoo.org">Robin H. Johnson</mail> 
</author>
<author title="Editor">
  <mail link="fox2mike@gentoo.org">Shyam Mani</mail>
</author>

<abstract>
This document explains the requirements for source mirrors.
</abstract>

<!-- The content of this document is licensed under the CC-BY-SA license -->
<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
<license/>
<version>1.12</version>
<date>2010-04-23</date>

<chapter>
<title>Requirements</title>
<section>
<title>Minimum Bandwidth</title>
<body>

<p>
To properly host a source mirror, you should have a minimum of 5Mbps upstream
bandwidth. ADSL and cable connections do not generally make adequate
connections for distfile hosts due to their limited upstream performance.
</p>

</body>
</section>
<section>
<title>Minimum Disk Space</title>
<body>

<p>
The space requirements depend on whether you choose to run a "Full" mirror or a
"Distfiles only" mirror. Full mirrors require 180 GB of disk space.  Distfiles
only mirrors require 100 GB of space.
</p>

</body>
</section>
<section>
<title>Minimum User Count</title>
<body>

<p>
We ask that you support a minimum of 15 and preferrably 30 concurrent user
connections.
</p>

</body>
</section>
<section>
<title>Update Frequency</title>
<body>

<p>
All mirrors must be synchronized every four hours, starting at midnight in your
local time zone.  This helps distribute the load out over a period of time,
rather than having all mirrors synchronize at the same time.
</p>

</body>
</section>
<section>
<title>Mirror Needs</title>
<body>

<p>
Your mirror may not be added immediately to our listing and given access to the
private distribution point. We add mirrors on an as-needed basis so as not to
clutter up the listing, and to provide an adequate amount of high speed mirrors.
You are more than welcome to peek in <uri
link="irc://irc.gentoo.org/gentoo-mirrors">#gentoo-mirrors</uri> on IRC and chat
up to see the status of your mirror being added to our official network.
</p>

</body>
</section>
</chapter>

<chapter>
<title>Implementation details</title>
<section>
<body>

<p>
For new source mirror requests, the procedure will be as follows:
</p>

<p>
Set up your server to rsync with ftp.ussg.iu.edu::gentoo-distfiles. Once you
have a complete mirror of that server, please file a bug report on <uri
link="http://bugs.gentoo.org/enter_bug.cgi?product=Mirrors">bugs.gentoo.org</uri>
under the <c>Mirrors</c> category stating that you have a new mirror and wish
it to be added to the list of official mirrors.  Please specify whether you
wish to be a "Full" mirror or "Distfiles only" mirror. Distfiles
only mirrors will be designated as such on the website.
</p>

<p>
Full mirrors must carry the following directories (plus any directories that
get added in the future):
</p>

<ul>
  <li>/releases (45-55G)</li>
  <li>/experimental (20-30G)</li>
  <li>/distfiles (110G)</li>
  <li>/snapshots (600M)</li>
  <li>/grp (0M, obsolete)</li>
</ul>

<p>
Distfiles only mirrors need to carry the following directories:
</p>

<ul>
  <li>/distfiles (110G)</li>
  <li>/snapshots (600M)</li>
</ul>

<p>
We will then verify that the mirror exists and we can connect to it properly.
Then, in the next 2 weeks, we will determine if your mirror is syncing
correctly and in accordance with this policy.
</p>

<p>
If all steps went smoothly, we will then add you to our list of <uri
link="/main/en/mirrors2.xml">download mirrors</uri> on the main web site.  We
will also send you the login information for our private master distfile mirror
which is only available to official Gentoo Linux source mirrors.
</p>

<p>
Additionally, you, the mirror admin, will be requested to subscribe to the
gentoo-mirrors mailing list (low traffic) so that you can follow all issues
associated with source mirrors.
</p>

<note>
Thanks for helping out Gentoo Linux users and developers! :) For any mirror
administration issues or problems, please contact <mail
link="mirror-admin@gentoo.org">mirror-admin@gentoo.org</mail>.
</note>

<warn>
Do NOT email mirror-admin@gentoo.org for mirror requests, please file a bug. It
makes mirror tracking easier on us.
</warn>

</body>
</section>
</chapter>

<chapter>
<title>Short FAQ</title>
<section>
<title>
Q: Who should I contact regarding new mirrors or to get my mirror added to the
Gentoo web site?
</title>
<body>

<p>
A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
</p>

</body>
</section>
<section>
<title>Q: Who should I contact regarding mirror issues and maintenance?</title>
<body>

<p>
A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
</p>

</body>
</section>
<section>
<title>
Q: I run a private source mirror for my company.  Can I still access the
private master source mirror?
</title>
<body>

<p>
A: Because our resources are limited, we need to ensure we allocate them in
such a way to provide the maximum amount of benefit to our users.  As such, we
limit connections to our master rsync and distfile mirrors to public mirrors
only.  Users are welcome to use our regular mirror system to establish a
private source mirror.
</p>

</body>
</section>
<section>
<title>Q: How do I find the mirror nearest to me?</title>
<body>

<p>
A: mirrorselect was designed to do this for you. If you haven't already run
<c>emerge mirrorselect</c>, then please do so.  Then run: <c>mirrorselect</c>.
An ncurses-based menu will display allowing you to select a mirror close to
you.
</p>

</body>
</section>
<section>
<title>
Q: I don't have that much disk space.  Can I mirror part of the sources?
</title>
<body>

<p>
A: Yes, if you are unable to mirror our full sources, you can elect to mirror
the following two directories only:
</p>

<ul>
  <li>/distfiles</li>
  <li>/snapshots</li>
</ul>

<p>
These two directories will require approximately 50GB of space and you will be
listed on our site as a distfiles-only mirror.
</p>

</body>
</section>
<section>
<title>
Q: Do I have to turn on Directory Indexing for directories served up via HTTP?
</title>
<body>

<p>
A: Directory Indexing is required for the /releases and /experimental
directories, but can be turned off for all other directories (including
/distfiles) if you wish.
</p>

</body>
</section>

<section>
<title>Q: Do I have to mirror historical? I mean, it is old releases.</title>
<body>

<p>
A: Of course not, it is quite a bit of data. In your rsync command, add
<path>--exclude=/releases/historical</path> to your rsync options.  This will
exclude historical and save you quite a bit of space.
</p>

</body>
</section>
</chapter>
</guide>
1	klieber	1.1	<?xml version='1.0' encoding="UTF-8"?>
2	darkside	1.35	<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/source_mirrors.xml,v 1.34 2009/11/19 18:08:04 robbat2 Exp $ -->
3	klieber	1.1	<!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
4
5	klieber	1.3	<guide link="/doc/en/source_mirrors.xml">
6	neysx	1.29
7	klieber	1.1	<title>Gentoo Linux Source Mirrors Policy</title>
8	neysx	1.29
9	neysx	1.19	<author title="Author">
10			<mail link="klieber@gentoo.org">Kurt Lieber</mail>
11			</author>
12			<author title="Editor">
13			<mail link="cshields@gentoo.org">Corey Shields</mail>
14			</author>
15			<author title="Editor">
16			<mail link="bennyc@gentoo.org">Benny Chuang</mail>
17			</author>
18			<author title="Editor">
19			<mail link="jforman@gentoo.org">Jeffrey Forman</mail>
20			</author>
21	robbat2	1.33	<author title="Editor">
22			<mail link="robbat2@gentoo.org">Robin H. Johnson</mail>
23			</author>
24			<author title="Editor">
25			<mail link="fox2mike@gentoo.org">Shyam Mani</mail>
26			</author>
27	klieber	1.1
28	swift	1.15	<abstract>
29			This document explains the requirements for source mirrors.
30			</abstract>
31	neysx	1.19
32	neysx	1.30	<!-- The content of this document is licensed under the CC-BY-SA license -->
33			<!-- See http://creativecommons.org/licenses/by-sa/2.5 -->
34	swift	1.15	<license/>
35	darkside	1.35	<version>1.12</version>
36			<date>2010-04-23</date>
37	swift	1.13
38	klieber	1.1	<chapter>
39			<title>Requirements</title>
40			<section>
41			<title>Minimum Bandwidth</title>
42			<body>
43	neysx	1.19
44			<p>
45			To properly host a source mirror, you should have a minimum of 5Mbps upstream
46	neysx	1.29	bandwidth. ADSL and cable connections do not generally make adequate
47	neysx	1.19	connections for distfile hosts due to their limited upstream performance.
48			</p>
49
50	klieber	1.1	</body>
51			</section>
52			<section>
53	cshields	1.16	<title>Minimum Disk Space</title>
54			<body>
55	neysx	1.19
56			<p>
57			The space requirements depend on whether you choose to run a "Full" mirror or a
58	robbat2	1.34	"Distfiles only" mirror. Full mirrors require 180 GB of disk space. Distfiles
59			only mirrors require 100 GB of space.
60	neysx	1.19	</p>
61
62	cshields	1.16	</body>
63			</section>
64			<section>
65	klieber	1.1	<title>Minimum User Count</title>
66			<body>
67	neysx	1.19
68			<p>
69			We ask that you support a minimum of 15 and preferrably 30 concurrent user
70			connections.
71			</p>
72
73	klieber	1.1	</body>
74			</section>
75			<section>
76			<title>Update Frequency</title>
77			<body>
78	neysx	1.19
79			<p>
80			All mirrors must be synchronized every four hours, starting at midnight in your
81			local time zone. This helps distribute the load out over a period of time,
82			rather than having all mirrors synchronize at the same time.
83			</p>
84	neysx	1.29
85	klieber	1.1	</body>
86			</section>
87	jforman	1.26	<section>
88			<title>Mirror Needs</title>
89			<body>
90	neysx	1.29
91	jforman	1.26	<p>
92	neysx	1.29	Your mirror may not be added immediately to our listing and given access to the
93			private distribution point. We add mirrors on an as-needed basis so as not to
94	nightmorph	1.32	clutter up the listing, and to provide an adequate amount of high speed mirrors.
95			You are more than welcome to peek in <uri
96			link="irc://irc.gentoo.org/gentoo-mirrors">#gentoo-mirrors</uri> on IRC and chat
97			up to see the status of your mirror being added to our official network.
98	jforman	1.26	</p>
99	neysx	1.29
100	jforman	1.26	</body>
101			</section>
102	klieber	1.1	</chapter>
103	neysx	1.19
104	klieber	1.1	<chapter>
105			<title>Implementation details</title>
106			<section>
107			<body>
108	neysx	1.19
109			<p>
110			For new source mirror requests, the procedure will be as follows:
111			</p>
112
113			<p>
114	neysx	1.30	Set up your server to rsync with ftp.ussg.iu.edu::gentoo-distfiles. Once you
115			have a complete mirror of that server, please file a bug report on <uri
116			link="http://bugs.gentoo.org/enter_bug.cgi?product=Mirrors">bugs.gentoo.org</uri>
117			under the <c>Mirrors</c> category stating that you have a new mirror and wish
118			it to be added to the list of official mirrors. Please specify whether you
119			wish to be a "Full" mirror or "Distfiles only" mirror. Distfiles
120	neysx	1.23	only mirrors will be designated as such on the website.
121	neysx	1.19	</p>
122
123			<p>
124			Full mirrors must carry the following directories (plus any directories that
125			get added in the future):
126			</p>
127
128	cshields	1.16	<ul>
129	darkside	1.35	<li>/releases (45-55G)</li>
130	robbat2	1.33	<li>/experimental (20-30G)</li>
131	darkside	1.35	<li>/distfiles (110G)</li>
132			<li>/snapshots (600M)</li>
133			<li>/grp (0M, obsolete)</li>
134	cshields	1.16	</ul>
135	neysx	1.19
136			<p>
137			Distfiles only mirrors need to carry the following directories:
138			</p>
139
140	cshields	1.16	<ul>
141	darkside	1.35	<li>/distfiles (110G)</li>
142			<li>/snapshots (600M)</li>
143	cshields	1.16	</ul>
144
145	neysx	1.19	<p>
146			We will then verify that the mirror exists and we can connect to it properly.
147	robbat2	1.34	Then, in the next 2 weeks, we will determine if your mirror is syncing
148	neysx	1.19	correctly and in accordance with this policy.
149			</p>
150	klieber	1.1
151	neysx	1.19	<p>
152			If all steps went smoothly, we will then add you to our list of <uri
153	robbat2	1.34	link="/main/en/mirrors2.xml">download mirrors</uri> on the main web site. We
154	neysx	1.23	will also send you the login information for our private master distfile mirror
155			which is only available to official Gentoo Linux source mirrors.
156	neysx	1.29	</p>
157	neysx	1.19
158			<p>
159			Additionally, you, the mirror admin, will be requested to subscribe to the
160			gentoo-mirrors mailing list (low traffic) so that you can follow all issues
161			associated with source mirrors.
162			</p>
163	klieber	1.1
164			<note>
165	neysx	1.19	Thanks for helping out Gentoo Linux users and developers! :) For any mirror
166			administration issues or problems, please contact <mail
167			link="mirror-admin@gentoo.org">mirror-admin@gentoo.org</mail>.
168	klieber	1.1	</note>
169	neysx	1.19
170	jforman	1.21	<warn>
171	neysx	1.29	Do NOT email mirror-admin@gentoo.org for mirror requests, please file a bug. It
172			makes mirror tracking easier on us.
173	jforman	1.21	</warn>
174
175	klieber	1.1	</body>
176			</section>
177			</chapter>
178	neysx	1.19
179	klieber	1.1	<chapter>
180			<title>Short FAQ</title>
181			<section>
182	neysx	1.29	<title>
183			Q: Who should I contact regarding new mirrors or to get my mirror added to the
184			Gentoo web site?
185			</title>
186	klieber	1.1	<body>
187	neysx	1.19
188			<p>
189	neysx	1.30	A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
190	neysx	1.19	</p>
191
192	klieber	1.1	</body>
193			</section>
194			<section>
195			<title>Q: Who should I contact regarding mirror issues and maintenance?</title>
196			<body>
197	neysx	1.19
198			<p>
199	neysx	1.30	A: Please file a bug on bugs.gentoo.org under the <c>Mirrors</c> category.
200	neysx	1.29	</p>
201	neysx	1.19
202	klieber	1.1	</body>
203			</section>
204	klieber	1.9	<section>
205	neysx	1.29	<title>
206			Q: I run a private source mirror for my company. Can I still access the
207			private master source mirror?
208			</title>
209	klieber	1.9	<body>
210	neysx	1.19
211			<p>
212			A: Because our resources are limited, we need to ensure we allocate them in
213			such a way to provide the maximum amount of benefit to our users. As such, we
214			limit connections to our master rsync and distfile mirrors to public mirrors
215			only. Users are welcome to use our regular mirror system to establish a
216			private source mirror.
217	klieber	1.9	</p>
218	neysx	1.19
219	klieber	1.9	</body>
220			</section>
221	klieber	1.1	<section>
222	klieber	1.9	<title>Q: How do I find the mirror nearest to me?</title>
223	klieber	1.1	<body>
224	neysx	1.19
225			<p>
226			A: mirrorselect was designed to do this for you. If you haven't already run
227			<c>emerge mirrorselect</c>, then please do so. Then run: <c>mirrorselect</c>.
228			An ncurses-based menu will display allowing you to select a mirror close to
229			you.
230	klieber	1.1	</p>
231	neysx	1.19
232	klieber	1.1	</body>
233			</section>
234	klieber	1.5	<section>
235	neysx	1.29	<title>
236			Q: I don't have that much disk space. Can I mirror part of the sources?
237			</title>
238	klieber	1.11	<body>
239	neysx	1.19
240			<p>
241			A: Yes, if you are unable to mirror our full sources, you can elect to mirror
242			the following two directories only:
243			</p>
244
245	klieber	1.11	<ul>
246	neysx	1.28	<li>/distfiles</li>
247			<li>/snapshots</li>
248	klieber	1.11	</ul>
249	neysx	1.19
250			<p>
251	neysx	1.31	These two directories will require approximately 50GB of space and you will be
252	neysx	1.19	listed on our site as a distfiles-only mirror.
253			</p>
254
255	klieber	1.5	</body>
256			</section>
257	klieber	1.8	<section>
258	neysx	1.29	<title>
259			Q: Do I have to turn on Directory Indexing for directories served up via HTTP?
260			</title>
261	klieber	1.8	<body>
262	neysx	1.19
263			<p>
264			A: Directory Indexing is required for the /releases and /experimental
265			directories, but can be turned off for all other directories (including
266	neysx	1.29	/distfiles) if you wish.
267	klieber	1.8	</p>
268	neysx	1.29
269	jforman	1.24	</body>
270			</section>
271
272			<section>
273			<title>Q: Do I have to mirror historical? I mean, it is old releases.</title>
274			<body>
275	neysx	1.19
276	jforman	1.24	<p>
277	neysx	1.25	A: Of course not, it is quite a bit of data. In your rsync command, add
278			<path>--exclude=/releases/historical</path> to your rsync options. This will
279			exclude historical and save you quite a bit of space.
280	jforman	1.24	</p>
281	neysx	1.25
282	klieber	1.8	</body>
283			</section>
284	klieber	1.1	</chapter>
285			</guide>