| 1 |
neysx |
1.1 |
<?xml version='1.0' encoding="UTF-8"?> |
| 2 |
fox2mike |
1.19 |
<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/utf-8.xml,v 1.18 2005/06/02 18:51:06 swift Exp $ --> |
| 3 |
neysx |
1.1 |
<!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> |
| 4 |
|
|
|
| 5 |
|
|
<guide link="/doc/en/utf-8.xml"> |
| 6 |
|
|
<title>Using UTF-8 with Gentoo</title> |
| 7 |
|
|
|
| 8 |
|
|
<author title="Author"> |
| 9 |
|
|
<mail link="slarti@gentoo.org">Thomas Martin</mail> |
| 10 |
|
|
</author> |
| 11 |
|
|
<author title="Contributor"> |
| 12 |
|
|
<mail link="devil@gentoo.org.ua">Alexander Simonov</mail> |
| 13 |
|
|
</author> |
| 14 |
|
|
|
| 15 |
|
|
<abstract> |
| 16 |
|
|
This guide shows you how to set up and use the UTF-8 Unicode character set with |
| 17 |
|
|
your Gentoo Linux system, after explaining the benefits of Unicode and more |
| 18 |
|
|
specifically UTF-8. |
| 19 |
|
|
</abstract> |
| 20 |
|
|
|
| 21 |
|
|
<license /> |
| 22 |
|
|
|
| 23 |
swift |
1.18 |
<version>2.2</version> |
| 24 |
fox2mike |
1.19 |
<date>2005-06-18</date> |
| 25 |
neysx |
1.1 |
|
| 26 |
|
|
<chapter> |
| 27 |
|
|
<title>Character Encodings</title> |
| 28 |
|
|
<section> |
| 29 |
|
|
<title>What is a Character Encoding?</title> |
| 30 |
|
|
<body> |
| 31 |
|
|
|
| 32 |
|
|
<p> |
| 33 |
|
|
Computers do not understand text themselves. Instead, every character is |
| 34 |
|
|
represented by a number. Traditionally, each set of numbers used to represent |
| 35 |
|
|
alphabets and characters (known as a coding system, encoding or character set) |
| 36 |
|
|
was limited in size due to limitations in computer hardware. |
| 37 |
|
|
</p> |
| 38 |
|
|
|
| 39 |
|
|
</body> |
| 40 |
|
|
</section> |
| 41 |
|
|
<section> |
| 42 |
|
|
<title>The History of Character Encodings</title> |
| 43 |
|
|
<body> |
| 44 |
|
|
|
| 45 |
|
|
<p> |
| 46 |
|
|
The most common (or at least the most widely accepted) character set is |
| 47 |
|
|
<b>ASCII</b> (American Standard Code for Information Interchange). It is widely |
| 48 |
|
|
held that ASCII is the most successful software standard ever. Modern ASCII |
| 49 |
|
|
was standardised in 1986 (ANSI X3.4, RFC 20, ISO/IEC 646:1991, ECMA-6) by the |
| 50 |
|
|
American National Standards Institute. |
| 51 |
|
|
</p> |
| 52 |
|
|
|
| 53 |
|
|
<p> |
| 54 |
|
|
ASCII is strictly seven-bit, meaning that it uses bit patterns representable |
| 55 |
|
|
with seven binary digits, which provides a range of 0 to 127 in decimal. These |
| 56 |
|
|
include 32 non-visible control characters, most between 0 and 31, with the |
| 57 |
|
|
final control character, DEL or delete at 127. Characters 32 to 126 are |
| 58 |
|
|
visible characters: a space, punctuation marks, Latin letters and numbers. |
| 59 |
|
|
</p> |
| 60 |
|
|
|
| 61 |
|
|
<p> |
| 62 |
|
|
The eighth bit in ASCII was originally used as a parity bit for error checking. |
| 63 |
|
|
If this is not desired, it is left as 0. This means that, with ASCII, each |
| 64 |
|
|
character is represented by a single byte. |
| 65 |
|
|
</p> |
| 66 |
|
|
|
| 67 |
|
|
<p> |
| 68 |
|
|
Although ASCII was enough for communication in modern English, in other |
| 69 |
|
|
European languages that include accented characters, things were not so easy. |
| 70 |
|
|
The ISO 8859 standards were developed to meet these needs. They were backwards |
| 71 |
|
|
compatible with ASCII, but instead of leaving the eighth bit blank, they used |
| 72 |
|
|
it to allow another 127 characters in each encoding. ISO 8859's limitations |
| 73 |
|
|
soon came to light, and there are currently 15 variants of the ISO 8859 |
| 74 |
|
|
standard (8859-1 through to 8859-15). Outside of the ASCII-compatible byte |
| 75 |
|
|
range of these character sets, there is often conflict between the letters |
| 76 |
|
|
represented by each byte. To complicate interoperability between character |
| 77 |
|
|
encodings further, Windows-1252 is used in some versions of Microsoft Windows |
| 78 |
|
|
instead for Western European languages. This is a superset of ISO 8859-1, |
| 79 |
|
|
however it is different in several ways. These sets do all retain ASCII |
| 80 |
|
|
compatibility, however. |
| 81 |
|
|
</p> |
| 82 |
|
|
|
| 83 |
|
|
<p> |
| 84 |
|
|
The necessary development of completely different single-byte encodings for |
| 85 |
|
|
non-Latin alphabets, such as EUC (Extended Unix Coding) which is used for |
| 86 |
|
|
Japanese and Korean (and to a lesser extent Chinese) created more confusion, |
| 87 |
|
|
while other operating systems still used different character sets for the same |
| 88 |
|
|
languages, for example, Shift-JIS and ISO-2022-JP. Users wishing to view |
| 89 |
|
|
cyrillic glyphs had to choose between KOI8-R for Russian and Bulgarian or |
| 90 |
|
|
KOI8-U for Ukrainian, as well as all the other cyrillic encodings such as the |
| 91 |
|
|
unsuccessful ISO 8859-5, and the common Windows-1251 set. All of these |
| 92 |
|
|
character sets broke most compatibility with ASCII (although KOI8 encodings |
| 93 |
|
|
place cyrillic characters in Latin order, so in case the eighth bit is |
| 94 |
|
|
stripped, text is still decipherable on an ASCII terminal through case-reversed |
| 95 |
|
|
transliteration.) |
| 96 |
|
|
</p> |
| 97 |
|
|
|
| 98 |
|
|
<p> |
| 99 |
|
|
This has led to confusion, and also to an almost total inability for |
| 100 |
|
|
multilingual communication, especially across different alphabets. Enter |
| 101 |
|
|
Unicode. |
| 102 |
|
|
</p> |
| 103 |
|
|
|
| 104 |
|
|
</body> |
| 105 |
|
|
</section> |
| 106 |
|
|
<section> |
| 107 |
|
|
<title>What is Unicode?</title> |
| 108 |
|
|
<body> |
| 109 |
|
|
|
| 110 |
|
|
<p> |
| 111 |
bennyc |
1.11 |
Unicode throws away the traditional single-byte limit of character sets. It |
| 112 |
|
|
uses 17 "planes" of 65,536 code points to describe a maximum of 1,114,112 |
| 113 |
|
|
characters. As the first plane, aka. "Basic Multilingual Plane" or BMP, |
| 114 |
|
|
contains almost everything you will ever use, many have made the wrong |
| 115 |
|
|
assumption that Unicode was a 16-bit character set. |
| 116 |
neysx |
1.1 |
</p> |
| 117 |
|
|
|
| 118 |
|
|
<p> |
| 119 |
|
|
Unicode has been mapped in many different ways, but the two most common are |
| 120 |
|
|
<b>UTF</b> (Unicode Transformation Format) and <b>UCS</b> (Universal Character |
| 121 |
|
|
Set). A number after UTF indicates the number of bits in one unit, while the |
| 122 |
|
|
number after UCS indicates the number of bytes. UTF-8 has become the most |
| 123 |
|
|
widespread means for the interchange of Unicode text as a result of its |
| 124 |
|
|
eight-bit clean nature, and it is the subject of this document. |
| 125 |
|
|
</p> |
| 126 |
|
|
|
| 127 |
|
|
</body> |
| 128 |
|
|
</section> |
| 129 |
|
|
<section> |
| 130 |
|
|
<title>UTF-8</title> |
| 131 |
|
|
<body> |
| 132 |
|
|
|
| 133 |
|
|
<p> |
| 134 |
|
|
UTF-8 is a variable-length character encoding, which in this instance means |
| 135 |
|
|
that it uses 1 to 4 bytes per symbol. So, the first UTF-8 byte is used for |
| 136 |
|
|
encoding ASCII, giving the character set full backwards compatibility with |
| 137 |
|
|
ASCII. UTF-8 means that ASCII and Latin characters are interchangeable with |
| 138 |
|
|
little increase in the size of the data, because only the first bit is used. |
| 139 |
|
|
Users of Eastern alphabets such as Japanese, who have been assigned a higher |
| 140 |
|
|
byte range are unhappy, as this results in as much as a 50% redundancy in their |
| 141 |
|
|
data. |
| 142 |
|
|
</p> |
| 143 |
|
|
|
| 144 |
|
|
</body> |
| 145 |
|
|
</section> |
| 146 |
|
|
<section> |
| 147 |
|
|
<title>What UTF-8 Can Do for You</title> |
| 148 |
|
|
<body> |
| 149 |
|
|
|
| 150 |
|
|
<p> |
| 151 |
|
|
UTF-8 allows you to work in a standards-compliant and internationally accepted |
| 152 |
bennyc |
1.11 |
multilingual environment, with a comparatively low data redundancy. UTF-8 is |
| 153 |
neysx |
1.1 |
the preferred way for transmitting non-ASCII characters over the Internet, |
| 154 |
|
|
through Email, IRC or almost any other medium. Despite this, many people regard |
| 155 |
|
|
UTF-8 in online communication as abusive. It is always best to be aware of the |
| 156 |
|
|
attitude towards UTF-8 in a specific channel, mailing list or Usenet group |
| 157 |
|
|
before using <e>non-ASCII</e> UTF-8. |
| 158 |
|
|
</p> |
| 159 |
|
|
|
| 160 |
|
|
</body> |
| 161 |
|
|
</section> |
| 162 |
|
|
</chapter> |
| 163 |
|
|
|
| 164 |
|
|
<chapter> |
| 165 |
|
|
<title>Setting up UTF-8 with Gentoo Linux</title> |
| 166 |
|
|
<section> |
| 167 |
|
|
<title>Finding or Creating UTF-8 Locales</title> |
| 168 |
|
|
<body> |
| 169 |
|
|
|
| 170 |
|
|
<p> |
| 171 |
|
|
Now that you understand the principles behind Unicode, you're ready to start |
| 172 |
|
|
using UTF-8 with your system. |
| 173 |
|
|
</p> |
| 174 |
|
|
|
| 175 |
|
|
<p> |
| 176 |
|
|
The preliminary requirement for UTF-8 is to have a version of glibc installed |
| 177 |
|
|
that has national language support. The recommend means to do this is the |
| 178 |
|
|
<path>/etc/locales.build</path> file in combination with the <c>userlocales</c> |
| 179 |
|
|
USE flag. It is beyond the scope of this document to explain the usage of this |
| 180 |
|
|
file though, luckily, the usage of this file is well documented in the comments |
| 181 |
|
|
within it. It is also explained in the <uri |
| 182 |
|
|
link="/doc/en/guide-localization.xml#doc_chap3_sect3"> Gentoo Localisation |
| 183 |
|
|
Guide</uri>. |
| 184 |
|
|
</p> |
| 185 |
|
|
|
| 186 |
|
|
<p> |
| 187 |
|
|
Next, we'll need to decide whether a UTF-8 locale is already available for our |
| 188 |
|
|
language, or whether we need to create one. |
| 189 |
|
|
</p> |
| 190 |
|
|
|
| 191 |
|
|
<pre caption="Checking for an existing UTF-8 locale"> |
| 192 |
|
|
<comment>(Replace "en_GB" with your desired locale setting)</comment> |
| 193 |
|
|
# <i>locale -a | grep 'en_GB'</i> |
| 194 |
|
|
en_GB |
| 195 |
bennyc |
1.12 |
en_GB.UTF-8 |
| 196 |
neysx |
1.1 |
</pre> |
| 197 |
|
|
|
| 198 |
|
|
<p> |
| 199 |
|
|
From the output of this command line, we need to take the result with a suffix |
| 200 |
bennyc |
1.12 |
similar to <c>.UTF-8</c>. If there is no result with a suffix similar to |
| 201 |
|
|
<c>.UTF-8</c>, we need to create a UTF-8 compatible locale. |
| 202 |
neysx |
1.1 |
</p> |
| 203 |
|
|
|
| 204 |
|
|
<note> |
| 205 |
|
|
Only execute the following code listing if you do not have a UTF-8 locale |
| 206 |
|
|
available for your language. |
| 207 |
|
|
</note> |
| 208 |
|
|
|
| 209 |
|
|
<pre caption="Creating a UTF-8 locale"> |
| 210 |
|
|
<comment>(Replace "en_GB" with your desired locale setting)</comment> |
| 211 |
bennyc |
1.12 |
# <i>localedef -i en_GB -f UTF-8 en_GB.UTF-8</i> |
| 212 |
neysx |
1.1 |
</pre> |
| 213 |
|
|
|
| 214 |
bennyc |
1.11 |
<p> |
| 215 |
|
|
Another way to include a UTF-8 locale is to add it to the |
| 216 |
|
|
<path>/etc/locales.build</path> file and rebuild <c>glibc</c> with the |
| 217 |
|
|
<c>userlocales</c> USE flag set. |
| 218 |
|
|
</p> |
| 219 |
|
|
|
| 220 |
|
|
<pre caption="Line in /etc/locales.build"> |
| 221 |
|
|
en_GB.UTF-8/UTF-8 |
| 222 |
|
|
</pre> |
| 223 |
|
|
|
| 224 |
neysx |
1.1 |
</body> |
| 225 |
|
|
</section> |
| 226 |
|
|
<section> |
| 227 |
|
|
<title>Setting the Locale</title> |
| 228 |
|
|
<body> |
| 229 |
|
|
|
| 230 |
|
|
<p> |
| 231 |
fox2mike |
1.19 |
There is one environment variable that needs to be set in order to use |
| 232 |
swift |
1.17 |
our new UTF-8 locales: <c>LC_ALL</c> (this variable overrides the <c>LANG</c> |
| 233 |
|
|
setting as well). There are also many different ways to set it; some people |
| 234 |
|
|
prefer to only have a UTF-8 environment for a specific user, in which case |
| 235 |
swift |
1.18 |
they set them in their <path>~/.profile</path> (if you use <c>/bin/sh</c>), |
| 236 |
|
|
<path>~/.bash_profile</path> or <path>~/.bashrc</path> (if you use |
| 237 |
|
|
<c>/bin/bash</c>). |
| 238 |
|
|
</p> |
| 239 |
|
|
|
| 240 |
|
|
<p> |
| 241 |
swift |
1.17 |
Others prefer to set the locale globally. One specific circumstance where |
| 242 |
|
|
the author particularly recommends doing this is when |
| 243 |
|
|
<path>/etc/init.d/xdm</path> is in use, because |
| 244 |
bennyc |
1.12 |
this init script starts the display manager and desktop before any of the |
| 245 |
|
|
aforementioned shell startup files are sourced, and so before any of the |
| 246 |
|
|
variables are in the environment. |
| 247 |
neysx |
1.1 |
</p> |
| 248 |
|
|
|
| 249 |
bennyc |
1.12 |
<p> |
| 250 |
|
|
Setting the locale globally should be done using |
| 251 |
swift |
1.15 |
<path>/etc/env.d/02locale</path>. The file should look something like the |
| 252 |
bennyc |
1.12 |
following: |
| 253 |
|
|
</p> |
| 254 |
|
|
|
| 255 |
|
|
<pre caption="Demonstration /etc/env.d/02locale"> |
| 256 |
|
|
<comment>(As always, change "en_GB.UTF-8" to your locale)</comment> |
| 257 |
|
|
LC_ALL="en_GB.UTF-8" |
| 258 |
|
|
</pre> |
| 259 |
|
|
|
| 260 |
|
|
<p> |
| 261 |
|
|
Next, the environment must be updated with the change. |
| 262 |
|
|
</p> |
| 263 |
bennyc |
1.10 |
|
| 264 |
bennyc |
1.12 |
<pre caption="Updating the environment"> |
| 265 |
|
|
# <i>env-update</i> |
| 266 |
|
|
>>> Regenerating /etc/ld.so.cache... |
| 267 |
|
|
* Caching service dependencies ... |
| 268 |
swift |
1.13 |
# <i>source /etc/profile</i> |
| 269 |
bennyc |
1.10 |
</pre> |
| 270 |
|
|
|
| 271 |
bennyc |
1.12 |
<p> |
| 272 |
|
|
Now, run <c>locale</c> with no arguments to see if we have the correct |
| 273 |
|
|
variables in our environment: |
| 274 |
|
|
</p> |
| 275 |
|
|
|
| 276 |
|
|
<pre caption="Checking if our new locale is in the environment"> |
| 277 |
|
|
# <i>locale</i> |
| 278 |
swift |
1.16 |
LANG= |
| 279 |
bennyc |
1.12 |
LC_CTYPE="en_GB.UTF-8" |
| 280 |
|
|
LC_NUMERIC="en_GB.UTF-8" |
| 281 |
|
|
LC_TIME="en_GB.UTF-8" |
| 282 |
|
|
LC_COLLATE="en_GB.UTF-8" |
| 283 |
|
|
LC_MONETARY="en_GB.UTF-8" |
| 284 |
|
|
LC_MESSAGES="en_GB.UTF-8" |
| 285 |
|
|
LC_PAPER="en_GB.UTF-8" |
| 286 |
|
|
LC_NAME="en_GB.UTF-8" |
| 287 |
|
|
LC_ADDRESS="en_GB.UTF-8" |
| 288 |
|
|
LC_TELEPHONE="en_GB.UTF-8" |
| 289 |
|
|
LC_MEASUREMENT="en_GB.UTF-8" |
| 290 |
|
|
LC_IDENTIFICATION="en_GB.UTF-8" |
| 291 |
|
|
LC_ALL=en_GB.UTF-8 |
| 292 |
neysx |
1.1 |
</pre> |
| 293 |
|
|
|
| 294 |
bennyc |
1.10 |
<p> |
| 295 |
bennyc |
1.12 |
That's everything. You are now using UTF-8 locales, and the next hurdle is the |
| 296 |
|
|
configuration of the applications you use from day to day. |
| 297 |
neysx |
1.1 |
</p> |
| 298 |
|
|
|
| 299 |
|
|
</body> |
| 300 |
|
|
</section> |
| 301 |
|
|
</chapter> |
| 302 |
|
|
|
| 303 |
|
|
<chapter> |
| 304 |
|
|
<title>Application Support</title> |
| 305 |
|
|
<section> |
| 306 |
|
|
<body> |
| 307 |
|
|
|
| 308 |
|
|
<p> |
| 309 |
|
|
When Unicode first started gaining momentum in the software world, multibyte |
| 310 |
|
|
character sets were not well suited to languages like C, in which many of the |
| 311 |
|
|
day-to-day programs people use are written. Even today, some programs are not |
| 312 |
|
|
able to handle UTF-8 properly. Fortunately, most are! |
| 313 |
|
|
</p> |
| 314 |
|
|
|
| 315 |
|
|
</body> |
| 316 |
|
|
</section> |
| 317 |
|
|
<section> |
| 318 |
|
|
<title>Filenames, NTFS, and FAT</title> |
| 319 |
|
|
<body> |
| 320 |
|
|
|
| 321 |
|
|
<p> |
| 322 |
|
|
There are several NLS options in the Linux kernel configuration menu, but it is |
| 323 |
|
|
important to not become confused! For the most part, the only thing you need to |
| 324 |
|
|
do is to build UTF-8 NLS support into your kernel, and change the default NLS |
| 325 |
|
|
option to utf8. |
| 326 |
|
|
</p> |
| 327 |
|
|
|
| 328 |
|
|
<pre caption="Kernel configuration steps for UTF-8 NLS"> |
| 329 |
|
|
File Systems --> |
| 330 |
|
|
Native Language Support --> |
| 331 |
|
|
(utf8) Default NLS Option |
| 332 |
|
|
<*> NLS UTF8 |
| 333 |
|
|
<comment>(Also <*> other character sets that are in use in |
| 334 |
|
|
your FAT filesystems or Joilet CD-ROMs.)</comment> |
| 335 |
|
|
</pre> |
| 336 |
|
|
|
| 337 |
|
|
<p> |
| 338 |
|
|
If you plan on mounting NTFS partitions, you may need to specify an <c>nls=</c> |
| 339 |
|
|
option with mount. For more information, see <c>man mount</c>. |
| 340 |
|
|
</p> |
| 341 |
|
|
|
| 342 |
|
|
<p> |
| 343 |
|
|
For changing the encoding of filenames, <c>app-text/convmv</c> can be used. |
| 344 |
|
|
</p> |
| 345 |
|
|
|
| 346 |
|
|
<pre caption="Example usage of convmv"> |
| 347 |
|
|
# <i>emerge --ask app-text/convmv</i> |
| 348 |
|
|
# <i>convmv -f current-encoding -t utf-8 filename</i> |
| 349 |
|
|
</pre> |
| 350 |
|
|
|
| 351 |
|
|
<p> |
| 352 |
|
|
For changing the <e>contents</e> of files, use the <c>iconv</c> utility, |
| 353 |
|
|
bundled with <c>glibc</c>: |
| 354 |
|
|
</p> |
| 355 |
|
|
|
| 356 |
|
|
<pre caption="Example usage of iconv"> |
| 357 |
|
|
<comment>(substitute iso-8859-1 with the charset you are converting from)</comment> |
| 358 |
|
|
<comment>(Check the output is sane)</comment> |
| 359 |
|
|
# <i>iconv -f iso-8859-1 -t utf-8 filename</i> |
| 360 |
|
|
<comment>(Convert a file, you must create another file)</comment> |
| 361 |
|
|
# <i>iconv -f iso-8859-1 -t utf-8 filename > newfile</i> |
| 362 |
|
|
</pre> |
| 363 |
|
|
|
| 364 |
|
|
<p> |
| 365 |
|
|
<c>app-text/recode</c> can also be used for this purpose. |
| 366 |
|
|
</p> |
| 367 |
|
|
|
| 368 |
|
|
</body> |
| 369 |
|
|
</section> |
| 370 |
|
|
<section> |
| 371 |
|
|
<title>The System Console</title> |
| 372 |
|
|
<body> |
| 373 |
|
|
|
| 374 |
|
|
<impo> |
| 375 |
|
|
You need >=sys-apps/baselayout-1.11.9 for Unicode on the console. |
| 376 |
|
|
</impo> |
| 377 |
|
|
|
| 378 |
|
|
<p> |
| 379 |
|
|
To enable UTF-8 on the console, you should edit <path>/etc/rc.conf</path> and |
| 380 |
|
|
set <c>UNICODE="yes"</c>, and also read the comments in that file -- it is |
| 381 |
|
|
important to have a font that has a good range of characters if you plan on |
| 382 |
|
|
making the most of Unicode. |
| 383 |
|
|
</p> |
| 384 |
|
|
|
| 385 |
|
|
<p> |
| 386 |
|
|
The <c>KEYMAP</c> variable, set in <path>/etc/conf.d/keymaps</path>, should |
| 387 |
|
|
have a Unicode keymap specified. To do this, simply prepend the keymap already |
| 388 |
|
|
specified there with -u. |
| 389 |
|
|
</p> |
| 390 |
|
|
|
| 391 |
|
|
<pre caption="Example /etc/conf.d/keymaps snippet"> |
| 392 |
|
|
<comment>(Change "uk" to your local layout)</comment> |
| 393 |
bennyc |
1.11 |
KEYMAP="-u uk" |
| 394 |
neysx |
1.1 |
</pre> |
| 395 |
|
|
|
| 396 |
|
|
</body> |
| 397 |
|
|
</section> |
| 398 |
|
|
<section> |
| 399 |
|
|
<title>Ncurses and Slang</title> |
| 400 |
|
|
<body> |
| 401 |
|
|
|
| 402 |
|
|
<note> |
| 403 |
|
|
Ignore any mention of Slang in this section if you do not have it installed or |
| 404 |
|
|
do not use it. |
| 405 |
|
|
</note> |
| 406 |
|
|
|
| 407 |
|
|
<p> |
| 408 |
|
|
It is wise to add <c>unicode</c> to your global USE flags in |
| 409 |
|
|
<path>/etc/make.conf</path>, and then to remerge <c>sys-libs/ncurses</c> and |
| 410 |
|
|
also <c>sys-libs/slang</c> if appropriate: |
| 411 |
|
|
</p> |
| 412 |
|
|
|
| 413 |
|
|
<pre caption="Emerging ncurses and slang"> |
| 414 |
|
|
<comment>(We avoid putting these libraries in our world file with --oneshot)</comment> |
| 415 |
|
|
# <i>emerge --oneshot --verbose --ask sys-libs/ncurses sys-libs/slang</i> |
| 416 |
|
|
</pre> |
| 417 |
|
|
|
| 418 |
|
|
<p> |
| 419 |
|
|
We also need to rebuild packages that link to these, now the USE changes have |
| 420 |
bennyc |
1.11 |
been applied. The tool we use (<c>revdep-rebuild</c>) is part of the |
| 421 |
|
|
<c>gentoolkit</c> package. |
| 422 |
neysx |
1.1 |
</p> |
| 423 |
|
|
|
| 424 |
|
|
<pre caption="Rebuilding of programs that link to ncurses or slang"> |
| 425 |
|
|
# <i>revdep-rebuild --soname libncurses.so.5</i> |
| 426 |
|
|
# <i>revdep-rebuild --soname libslang.so.1</i> |
| 427 |
|
|
</pre> |
| 428 |
|
|
|
| 429 |
|
|
</body> |
| 430 |
|
|
</section> |
| 431 |
|
|
<section> |
| 432 |
|
|
<title>KDE, GNOME and Xfce</title> |
| 433 |
|
|
<body> |
| 434 |
|
|
|
| 435 |
|
|
<p> |
| 436 |
|
|
All of the major desktop environments have full Unicode support, and will |
| 437 |
|
|
require no further setup than what has already been covered in this guide. This |
| 438 |
|
|
is because the underlying graphical toolkits (Qt or GTK+2) are UTF-8 aware. |
| 439 |
|
|
Subsequently, all applications running on top of these toolkits should be |
| 440 |
|
|
UTF-8-aware out of the box. |
| 441 |
|
|
</p> |
| 442 |
|
|
|
| 443 |
|
|
<p> |
| 444 |
|
|
The exceptions to this rule come in Xlib and GTK+1. GTK+1 requires a |
| 445 |
|
|
iso-10646-1 FontSpec in the ~/.gtkrc, for example |
| 446 |
|
|
<c>-misc-fixed-*-*-*-*-*-*-*-*-*-*-iso10646-1</c>. Also, applications using |
| 447 |
|
|
Xlib or Xaw will need to be given a similar FontSpec, otherwise they will not |
| 448 |
|
|
work. |
| 449 |
|
|
</p> |
| 450 |
|
|
|
| 451 |
|
|
<note> |
| 452 |
|
|
If you have a version of the gnome1 control center around, use that instead. |
| 453 |
|
|
Pick any iso10646-1 font from there. |
| 454 |
|
|
</note> |
| 455 |
|
|
|
| 456 |
|
|
<pre caption="Example ~/.gtkrc (for GTK+1) that defines a Unicode compatible font"> |
| 457 |
|
|
style "user-font" |
| 458 |
|
|
{ |
| 459 |
|
|
fontset="-misc-fixed-*-*-*-*-*-*-*-*-*-*-iso10646-1" |
| 460 |
|
|
} |
| 461 |
|
|
widget_class "*" style "user-font" |
| 462 |
|
|
</pre> |
| 463 |
|
|
|
| 464 |
|
|
<p> |
| 465 |
|
|
If an application has support for both a Qt and GTK+2 GUI, the GTK+2 GUI will |
| 466 |
|
|
generally give better results with Unicode. |
| 467 |
|
|
</p> |
| 468 |
|
|
|
| 469 |
|
|
</body> |
| 470 |
|
|
</section> |
| 471 |
|
|
<section> |
| 472 |
|
|
<title>X11 and Fonts</title> |
| 473 |
|
|
<body> |
| 474 |
|
|
|
| 475 |
bennyc |
1.11 |
<impo> |
| 476 |
|
|
<c>x11-base/xorg-x11</c> has far better support for Unicode than XFree86 |
| 477 |
|
|
and is <e>highly</e> recommended. |
| 478 |
|
|
</impo> |
| 479 |
|
|
|
| 480 |
neysx |
1.1 |
<p> |
| 481 |
|
|
TrueType fonts have support for Unicode, and most of the fonts that ship with |
| 482 |
|
|
Xorg have impressive character support, although, obviously, not every single |
| 483 |
|
|
glyph available in Unicode has been created for that font. To build fonts |
| 484 |
|
|
(including the Bitstream Vera set) with support for East Asian letters with X, |
| 485 |
|
|
make sure you have the <c>cjk</c> USE flag set. Many other applications utilise |
| 486 |
|
|
this flag, so it may be worthwhile to add it as a permanent USE flag. |
| 487 |
|
|
</p> |
| 488 |
|
|
|
| 489 |
|
|
<p> |
| 490 |
|
|
Also, several font packages in Portage are Unicode aware. |
| 491 |
|
|
</p> |
| 492 |
|
|
|
| 493 |
|
|
<pre caption="Optional: Install some more Unicode-aware fonts"> |
| 494 |
|
|
# <i>emerge terminus-font intlfonts freefonts cronyx-fonts corefonts</i> |
| 495 |
|
|
</pre> |
| 496 |
|
|
|
| 497 |
|
|
</body> |
| 498 |
|
|
</section> |
| 499 |
|
|
<section> |
| 500 |
|
|
<title>Window Managers and Terminal Emulators</title> |
| 501 |
|
|
<body> |
| 502 |
|
|
|
| 503 |
|
|
<p> |
| 504 |
bennyc |
1.11 |
Window managers not built on GTK or Qt generally have very good Unicode |
| 505 |
|
|
support, as they often use the Xft library for handling fonts. If your window |
| 506 |
|
|
manager does not use Xft for fonts, you can still use the FontSpec mentioned in |
| 507 |
|
|
the previous section as a Unicode font. |
| 508 |
neysx |
1.1 |
</p> |
| 509 |
|
|
|
| 510 |
|
|
<p> |
| 511 |
|
|
Terminal emulators that use Xft and support Unicode are harder to come by. |
| 512 |
|
|
Aside from Konsole and gnome-terminal, the best options in Portage are |
| 513 |
|
|
<c>x11-terms/rxvt-unicode</c>, <c>xfce-extra/terminal</c>, |
| 514 |
cam |
1.5 |
<c>gnustep-apps/terminal</c>, <c>x11-terms/mlterm</c>, <c>x11-terms/mrxvt</c> or |
| 515 |
neysx |
1.1 |
plain <c>x11-terms/xterm</c> when built with the <c>unicode</c> USE flag and |
| 516 |
|
|
invoked as <c>uxterm</c>. <c>app-misc/screen</c> supports UTF-8 too, when |
| 517 |
bennyc |
1.11 |
invoked as <c>screen -u</c> or the following is put into the |
| 518 |
neysx |
1.1 |
<path>~/.screenrc</path>: |
| 519 |
|
|
</p> |
| 520 |
|
|
|
| 521 |
|
|
<pre caption="~/.screenrc for UTF-8"> |
| 522 |
|
|
defutf8 on |
| 523 |
|
|
</pre> |
| 524 |
|
|
|
| 525 |
|
|
</body> |
| 526 |
|
|
</section> |
| 527 |
|
|
<section> |
| 528 |
|
|
<title>Vim, Emacs, Xemacs and Nano</title> |
| 529 |
|
|
<body> |
| 530 |
|
|
|
| 531 |
|
|
<p> |
| 532 |
|
|
Vim, Emacs and Xemacs provide full UTF-8 support, and also have builtin |
| 533 |
|
|
detection of UTF-8 files. For further information in Vim, use <c>:help |
| 534 |
|
|
mbyte.txt</c>. |
| 535 |
|
|
</p> |
| 536 |
|
|
|
| 537 |
|
|
<p> |
| 538 |
|
|
Nano currently does not provide support for UTF-8, although it has been planned |
| 539 |
|
|
for a long time. With luck, this will change in future. At the time of writing, |
| 540 |
|
|
UTF-8 support is in Nano's CVS, and should be included in the next release. |
| 541 |
|
|
</p> |
| 542 |
|
|
|
| 543 |
|
|
</body> |
| 544 |
|
|
</section> |
| 545 |
|
|
<section> |
| 546 |
|
|
<title>Shells</title> |
| 547 |
|
|
<body> |
| 548 |
|
|
|
| 549 |
|
|
<p> |
| 550 |
|
|
Currently, <c>bash</c> provides full Unicode support through the GNU readline |
| 551 |
|
|
library. Z Shell users are in a somewhat worse position -- no parts of the |
| 552 |
|
|
shell have Unicode support, although there is a concerted effort to add |
| 553 |
|
|
multibyte character set support underway at the moment. |
| 554 |
|
|
</p> |
| 555 |
|
|
|
| 556 |
|
|
<p> |
| 557 |
|
|
The C shell, <c>tcsh</c> and <c>ksh</c> do not provide UTF-8 support at all. |
| 558 |
|
|
</p> |
| 559 |
|
|
|
| 560 |
|
|
</body> |
| 561 |
|
|
</section> |
| 562 |
|
|
<section> |
| 563 |
|
|
<title>Irssi</title> |
| 564 |
|
|
<body> |
| 565 |
|
|
|
| 566 |
|
|
<p> |
| 567 |
cam |
1.5 |
Since 0.8.10, Irssi has complete UTF-8 support, although it does require a user |
| 568 |
|
|
to set an option. |
| 569 |
neysx |
1.1 |
</p> |
| 570 |
|
|
|
| 571 |
|
|
<pre caption="Enabling UTF-8 in Irssi"> |
| 572 |
|
|
/set term_charset UTF-8 |
| 573 |
|
|
</pre> |
| 574 |
|
|
|
| 575 |
|
|
<p> |
| 576 |
|
|
For channels where non-ASCII characters are often exchanged in non-UTF-8 |
| 577 |
|
|
charsets, the <c>/recode</c> command may be used to convert the characters. |
| 578 |
|
|
Type <c>/help recode</c> for more information. |
| 579 |
|
|
</p> |
| 580 |
|
|
|
| 581 |
|
|
</body> |
| 582 |
|
|
</section> |
| 583 |
|
|
<section> |
| 584 |
|
|
<title>Mutt</title> |
| 585 |
|
|
<body> |
| 586 |
|
|
|
| 587 |
|
|
<p> |
| 588 |
|
|
The Mutt mail user agent has very good Unicode support. To use UTF-8 with Mutt, |
| 589 |
|
|
put the following in your <path>~/.muttrc</path>: |
| 590 |
|
|
</p> |
| 591 |
|
|
|
| 592 |
|
|
<pre caption="~/.muttrc for UTF-8"> |
| 593 |
|
|
set send_charset="utf8" <comment>(outgoing character set)</comment> |
| 594 |
|
|
set charset="utf8" <comment>(display character set)</comment> |
| 595 |
|
|
</pre> |
| 596 |
|
|
|
| 597 |
|
|
<note> |
| 598 |
|
|
You may still see '?' in mail you read with Mutt. This is a result of people |
| 599 |
bennyc |
1.11 |
using a mail client which does not indicate the used charset. You can't do much |
| 600 |
|
|
about this than to ask them to configure their client correctly. |
| 601 |
neysx |
1.1 |
</note> |
| 602 |
|
|
|
| 603 |
|
|
<p> |
| 604 |
|
|
Further information is available from the <uri |
| 605 |
|
|
link="http://wiki.mutt.org/index.cgi?MuttFaq/Charset"> Mutt WikiWiki</uri>. |
| 606 |
|
|
</p> |
| 607 |
|
|
|
| 608 |
|
|
</body> |
| 609 |
|
|
</section> |
| 610 |
|
|
<section> |
| 611 |
swift |
1.14 |
<title>Less</title> |
| 612 |
|
|
<body> |
| 613 |
|
|
|
| 614 |
|
|
<p> |
| 615 |
|
|
We all use a lot of <c>more</c> or <c>less</c> along with the <c>|</c> to be |
| 616 |
|
|
able to correctly see the output of a command, like for example |
| 617 |
|
|
<c>dmesg | less</c>. While <c>more</c> only needs the shell to be UTF-8 aware, |
| 618 |
|
|
<c>less</c> needs an environment variable set, <c>LESSCHARSET</c> to ensure |
| 619 |
|
|
that unicode characters are rendered correctly. This can be set in |
| 620 |
|
|
<path>/etc/profile</path> or <path>~/.bash_profile</path>. Fire up the editor |
| 621 |
|
|
of your choice and the add the following line to one of the files mentioned |
| 622 |
|
|
above. |
| 623 |
|
|
</p> |
| 624 |
|
|
|
| 625 |
|
|
<pre caption="Setting up the Environment variable for less"> |
| 626 |
|
|
LESSCHARSET=utf-8 |
| 627 |
|
|
</pre> |
| 628 |
|
|
|
| 629 |
|
|
</body> |
| 630 |
|
|
</section> |
| 631 |
|
|
<section> |
| 632 |
|
|
<title>Man</title> |
| 633 |
|
|
<body> |
| 634 |
|
|
|
| 635 |
|
|
<p> |
| 636 |
|
|
Man pages are an integral part of any Linux machine. To ensure that any |
| 637 |
|
|
unicode in your man pages render correctly, edit <path>/etc/man.conf</path> |
| 638 |
|
|
and replace a line as shown below. |
| 639 |
|
|
</p> |
| 640 |
|
|
|
| 641 |
|
|
<pre caption="man.conf changes for Unicode support"> |
| 642 |
|
|
<comment>(This is the old line)</comment> |
| 643 |
|
|
NROFF /usr/bin/nroff -Tascii -c -mandoc |
| 644 |
|
|
<comment>(Replace the one above with this)</comment> |
| 645 |
|
|
NROFF /usr/bin/nroff -mandoc -c |
| 646 |
|
|
</pre> |
| 647 |
|
|
|
| 648 |
|
|
</body> |
| 649 |
|
|
</section> |
| 650 |
|
|
<section> |
| 651 |
neysx |
1.1 |
<title>Testing it all out</title> |
| 652 |
|
|
<body> |
| 653 |
|
|
|
| 654 |
|
|
<p> |
| 655 |
|
|
There are numerous UTF-8 test websites around. <c>net-www/w3m</c>, |
| 656 |
|
|
<c>net-www/links</c>, <c>net-www/elinks</c>, <c>net-www/lynx</c> and all |
| 657 |
cam |
1.3 |
Mozilla based browsers (including Firefox) support UTF-8. Konqueror and Opera |
| 658 |
|
|
have full UTF-8 support too. |
| 659 |
neysx |
1.1 |
</p> |
| 660 |
|
|
|
| 661 |
|
|
<p> |
| 662 |
|
|
When using one of the text-only web browsers, make absolutely sure you are |
| 663 |
|
|
using a Unicode-aware terminal. |
| 664 |
|
|
</p> |
| 665 |
|
|
|
| 666 |
|
|
<p> |
| 667 |
|
|
If you see certain characters displayed as boxes with letters or numbers |
| 668 |
|
|
inside, this means that your font does not have a character for the symbol or |
| 669 |
|
|
glyph that the UTF-8 wants. Instead, it displays a box with the hex code of the |
| 670 |
|
|
UTF-8 symbol. |
| 671 |
|
|
</p> |
| 672 |
|
|
|
| 673 |
|
|
<ul> |
| 674 |
|
|
<li> |
| 675 |
|
|
<uri link="http://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html">A W3C |
| 676 |
|
|
UTF-8 Test Page</uri> |
| 677 |
|
|
</li> |
| 678 |
|
|
<li> |
| 679 |
|
|
<uri link="http://titus.uni-frankfurt.de/indexe.htm?/unicode/unitest.htm"> |
| 680 |
|
|
A UTF-8 test page provided by the University of Frankfurt</uri> |
| 681 |
|
|
</li> |
| 682 |
|
|
</ul> |
| 683 |
|
|
|
| 684 |
|
|
</body> |
| 685 |
|
|
</section> |
| 686 |
|
|
<section> |
| 687 |
|
|
<title>Input Methods</title> |
| 688 |
|
|
<body> |
| 689 |
|
|
|
| 690 |
|
|
<p> |
| 691 |
|
|
<e>Dead keys</e> may be used to input characters in X that are not included on |
| 692 |
|
|
your keyboard. These work by pressing your right Alt key (or in some countries, |
| 693 |
|
|
AltGr) and an optional key from the non-alphabetical section of the keyboard to |
| 694 |
|
|
the left of the return key at once, releasing them, and then pressing a letter. |
| 695 |
|
|
The dead key should modify it. Input can be further modified by using the Shift |
| 696 |
|
|
key at the same time as pressing the AltGr and modifier. |
| 697 |
|
|
</p> |
| 698 |
|
|
|
| 699 |
|
|
<p> |
| 700 |
|
|
To enable dead keys in X, you need a layout that supports it. Most European |
| 701 |
|
|
layouts already have dead keys with the default variant. However, this is not |
| 702 |
|
|
true of North American layouts. Although there is a degree of inconsistency |
| 703 |
|
|
between layouts, the easiest solution seems to be to use a layout in the form |
| 704 |
|
|
"en_US" rather than "us", for example. The layout is set in |
| 705 |
|
|
<path>/etc/X11/xorg.conf</path> like so: |
| 706 |
|
|
</p> |
| 707 |
|
|
|
| 708 |
|
|
<pre caption="/etc/X11/xorg.conf snippet"> |
| 709 |
|
|
Section "InputDevice" |
| 710 |
|
|
Identifier "Keyboard0" |
| 711 |
|
|
Driver "kbd" |
| 712 |
|
|
Option "XkbLayout" "en_US" <comment># Rather than just "us"</comment> |
| 713 |
|
|
<comment>(Other Xkb options here)</comment> |
| 714 |
|
|
EndSection |
| 715 |
|
|
</pre> |
| 716 |
|
|
|
| 717 |
|
|
<note> |
| 718 |
|
|
The preceding change only needs to be applied if you are using a North American |
| 719 |
|
|
layout, or another layout where dead keys do not seem to be working. European |
| 720 |
|
|
users should have working dead keys as is. |
| 721 |
|
|
</note> |
| 722 |
|
|
|
| 723 |
|
|
<p> |
| 724 |
bennyc |
1.11 |
This change will come into effect when your X server is restarted. To apply the |
| 725 |
neysx |
1.1 |
change now, use the <c>setxkbmap</c> tool, for example, <c>setxkbmap en_US</c>. |
| 726 |
|
|
</p> |
| 727 |
|
|
|
| 728 |
|
|
<p> |
| 729 |
|
|
It is probably easiest to describe dead keys with examples. Although the |
| 730 |
bennyc |
1.11 |
results are locale dependent, the concepts should remain the same regardless of |
| 731 |
neysx |
1.1 |
locale. The examples contain UTF-8, so to view them you need to either tell |
| 732 |
|
|
your browser to view the page as UTF-8, or have a UTF-8 locale already |
| 733 |
|
|
configured. |
| 734 |
|
|
</p> |
| 735 |
|
|
|
| 736 |
|
|
<p> |
| 737 |
|
|
When I press AltGr and [ at once, release them, and then press a, 'ä' is |
| 738 |
bennyc |
1.11 |
produced. When I press AltGr and [ at once, and then press e, 'ë' is produced. |
| 739 |
|
|
When I press AltGr and ; at once, 'á' is produced, and when I press AltGr and ; |
| 740 |
|
|
at once, release them, and then press e, 'é' is produced. |
| 741 |
neysx |
1.1 |
</p> |
| 742 |
|
|
|
| 743 |
|
|
<p> |
| 744 |
|
|
By pressing AltGr, Shift and [ at once, releasing them, and then pressing a, a |
| 745 |
|
|
Scandinavian 'å' is produced. Similarly, when I press AltGr, Shift and [ at |
| 746 |
|
|
once, release <e>only</e> the [, and then press it again, '˚' is produced. |
| 747 |
|
|
Although it looks like one, this (U+02DA) is not the same as a degree symbol |
| 748 |
|
|
(U+00B0). This works for other accents produced by dead keys — AltGr and [, |
| 749 |
|
|
releasing only the [, then pressing it again makes '¨'. |
| 750 |
|
|
</p> |
| 751 |
|
|
|
| 752 |
|
|
<p> |
| 753 |
|
|
AltGr can be used with alphabetical keys alone. For example, AltGr and m, a |
| 754 |
bennyc |
1.12 |
Greek lower-case letter mu is produced: 'µ'. AltGr and s produce a |
| 755 |
|
|
scharfes s or esszet: 'ß'. As many European users would expect (because |
| 756 |
|
|
it is marked on their keyboard), AltGr and 4 produces a Euro sign, '€'. |
| 757 |
neysx |
1.1 |
</p> |
| 758 |
|
|
|
| 759 |
|
|
</body> |
| 760 |
|
|
</section> |
| 761 |
|
|
<section> |
| 762 |
|
|
<title>Resources</title> |
| 763 |
|
|
<body> |
| 764 |
|
|
|
| 765 |
|
|
<ul> |
| 766 |
|
|
<li> |
| 767 |
|
|
<uri link="http://www.wikipedia.com/wiki/Unicode">The Wikipedia entry for |
| 768 |
|
|
Unicode</uri> |
| 769 |
|
|
</li> |
| 770 |
|
|
<li> |
| 771 |
|
|
<uri link="http://www.wikipedia.com/wiki/UTF-8">The Wikipedia entry for |
| 772 |
|
|
UTF-8</uri> |
| 773 |
|
|
</li> |
| 774 |
|
|
<li><uri link="http://www.unicode.org">Unicode.org</uri></li> |
| 775 |
|
|
<li><uri link="http://www.utf-8.com">UTF-8.com</uri></li> |
| 776 |
|
|
<li><uri link="http://www.ietf.org/rfc/rfc3629.txt">RFC 3629</uri></li> |
| 777 |
|
|
<li><uri link="http://www.ietf.org/rfc/rfc2277.txt">RFC 2277</uri></li> |
| 778 |
bennyc |
1.11 |
<li> |
| 779 |
|
|
<uri |
| 780 |
|
|
link="http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF">Characters vs. |
| 781 |
|
|
Bytes</uri> |
| 782 |
|
|
</li> |
| 783 |
neysx |
1.1 |
</ul> |
| 784 |
|
|
|
| 785 |
|
|
</body> |
| 786 |
|
|
</section> |
| 787 |
|
|
</chapter> |
| 788 |
|
|
</guide> |