Computers do not understand text themselves. Instead, every character is represented by a number. Traditionally, each set of numbers used to represent alphabets and characters (known as a coding system, encoding or character set) was limited in size due to limitations in computer hardware.
The most common (or at least the most widely accepted) character set is ASCII (American Standard Code for Information Interchange). It is widely held that ASCII is the most successful software standard ever. Modern ASCII was standardised in 1986 (ANSI X3.4, RFC 20, ISO/IEC 646:1991, ECMA-6) by the American National Standards Institute.
ASCII is strictly seven-bit, meaning that it uses bit patterns representable with seven binary digits, which provides a range of 0 to 127 in decimal. These include 32 non-visible control characters, most between 0 and 31, with the final control character, DEL or delete at 127. Characters 32 to 126 are visible characters: a space, punctuation marks, Latin letters and numbers.
The eighth bit in ASCII was originally used as a parity bit for error checking. If this is not desired, it is left as 0. This means that, with ASCII, each character is represented by a single byte.
Although ASCII was enough for communication in modern English, in other European languages that include accented characters, things were not so easy. The ISO 8859 standards were developed to meet these needs. They were backwards compatible with ASCII, but instead of leaving the eighth bit blank, they used it to allow another 127 characters in each encoding. ISO 8859's limitations soon came to light, and there are currently 15 variants of the ISO 8859 standard (8859-1 through to 8859-15). Outside of the ASCII-compatible byte range of these character sets, there is often conflict between the letters represented by each byte. To complicate interoperability between character encodings further, Windows-1252 is used in some versions of Microsoft Windows instead for Western European languages. This is a superset of ISO 8859-1, however it is different in several ways. These sets do all retain ASCII compatibility, however.
The necessary development of completely different single-byte encodings for non-Latin alphabets, such as EUC (Extended Unix Coding) which is used for Japanese and Korean (and to a lesser extent Chinese) created more confusion, while other operating systems still used different character sets for the same languages, for example, Shift-JIS and ISO-2022-JP. Users wishing to view cyrillic glyphs had to choose between KOI8-R for Russian and Bulgarian or KOI8-U for Ukrainian, as well as all the other cyrillic encodings such as the unsuccessful ISO 8859-5, and the common Windows-1251 set. All of these character sets broke most compatibility with ASCII (although KOI8 encodings place cyrillic characters in Latin order, so in case the eighth bit is stripped, text is still decipherable on an ASCII terminal through case-reversed transliteration.)
This has led to confusion, and also to an almost total inability for multilingual communication, especially across different alphabets. Enter Unicode.
Unicode throws away the traditional single-byte limit of character sets, and even with two bytes per-character this allows a maximum 65,536 characters. Although this number is extremely high when compared to seven-bit and eight-bit encodings, it is still not enough for a character set designed to be used for symbols and scripts used only by scholars, and symbols that are only used in mathematics and other specialised fields.
Unicode has been mapped in many different ways, but the two most common are UTF (Unicode Transformation Format) and UCS (Universal Character Set). A number after UTF indicates the number of bits in one unit, while the number after UCS indicates the number of bytes. UTF-8 has become the most widespread means for the interchange of Unicode text as a result of its eight-bit clean nature, and it is the subject of this document.
UTF-8 is a variable-length character encoding, which in this instance means that it uses 1 to 4 bytes per symbol. So, the first UTF-8 byte is used for encoding ASCII, giving the character set full backwards compatibility with ASCII. UTF-8 means that ASCII and Latin characters are interchangeable with little increase in the size of the data, because only the first bit is used. Users of Eastern alphabets such as Japanese, who have been assigned a higher byte range are unhappy, as this results in as much as a 50% redundancy in their data.
UTF-8 allows you to work in a standards-compliant and internationally accepted
multilingual environment, with a comparitively low data redundancy. UTF-8 is
the preferred way for transmitting non-ASCII characters over the Internet,
through Email, IRC or almost any other medium. Despite this, many people regard
UTF-8 in online communication as abusive. It is always best to be aware of the
attitude towards UTF-8 in a specific channel, mailing list or Usenet group
before using
Now that you understand the principles behind Unicode, you're ready to start using UTF-8 with your system.
The preliminary requirement for UTF-8 is to have a version of glibc installed
that has national language support. The recommend means to do this is the
Next, we'll need to decide whether a UTF-8 locale is already available for our language, or whether we need to create one.
(Replace "en_GB" with your desired locale setting) # locale -a | grep 'en_GB' en_GB en_GB.utf8
From the output of this command line, we need to take the result with a suffix
similar to
(Replace "en_GB" with your desired locale setting) # localedef -i en_GB -f UTF-8 en_GB.utf8
Although by now you might be determined to use UTF-8 system wide, the author
does not recommend setting UTF-8 for the root user. Instead, it is best to set
the locale in your user's
export LANG="en_GB.utf8" export LC_ALL="en_GB.utf8"
setenv LANG "en_GB.utf8" setenv LC_ALL "en_GB.utf8"
Now, logout and back in to apply the change. We want these environment
variables in our entire environment, so it is best to logout and back in, or at
the very least to source
When Unicode first started gaining momentum in the software world, multibyte character sets were not well suited to languages like C, in which many of the day-to-day programs people use are written. Even today, some programs are not able to handle UTF-8 properly. Fortunately, most are!
There are several NLS options in the Linux kernel configuration menu, but it is important to not become confused! For the most part, the only thing you need to do is to build UTF-8 NLS support into your kernel, and change the default NLS option to utf8.
File Systems -->
Native Language Support -->
(utf8) Default NLS Option
<*> NLS UTF8
(Also <*> other character sets that are in use in
your FAT filesystems or Joilet CD-ROMs.)
If you plan on mounting NTFS partitions, you may need to specify an
For changing the encoding of filenames,
# emerge --ask app-text/convmv # convmv -f current-encoding -t utf-8 filename
For changing the
(substitute iso-8859-1 with the charset you are converting from) (Check the output is sane) # iconv -f iso-8859-1 -t utf-8 filename(Convert a file, you must create another file) # iconv -f iso-8859-1 -t utf-8 filename > newfile
To enable UTF-8 on the console, you should edit
The
(Change "uk" to your local layout) KEYMAP="-u uk"
It is wise to add
(We avoid putting these libraries in our world file with --oneshot) # emerge --oneshot --verbose --ask sys-libs/ncurses sys-libs/slang
We also need to rebuild packages that link to these, now the USE changes have been applied.
# revdep-rebuild --soname libncurses.so.5 # revdep-rebuild --soname libslang.so.1
All of the major desktop environments have full Unicode support, and will require no further setup than what has already been covered in this guide. This is because the underlying graphical toolkits (Qt or GTK+2) are UTF-8 aware. Subsequently, all applications running on top of these toolkits should be UTF-8-aware out of the box.
The exceptions to this rule come in Xlib and GTK+1. GTK+1 requires a
iso-10646-1 FontSpec in the ~/.gtkrc, for example
style "user-font"
{
fontset="-misc-fixed-*-*-*-*-*-*-*-*-*-*-iso10646-1"
}
widget_class "*" style "user-font"
If an application has support for both a Qt and GTK+2 GUI, the GTK+2 GUI will generally give better results with Unicode.
TrueType fonts have support for Unicode, and most of the fonts that ship with
Xorg have impressive character support, although, obviously, not every single
glyph available in Unicode has been created for that font. To build fonts
(including the Bitstream Vera set) with support for East Asian letters with X,
make sure you have the
Also, several font packages in Portage are Unicode aware.
# emerge terminus-font intlfonts freefonts cronyx-fonts corefonts
Window managers not built on GTK or Qt generally have very good Unicode support, as they often use the Xft library for handling fonts. If your window manager does not use Xft for fonts, you can still use the FontSpec mentioned in the previous section as a Unicode font.
Terminal emulators that use Xft and support Unicode are harder to come by.
Aside from Konsole and gnome-terminal, the best options in Portage are
defutf8 on
Vim, Emacs and Xemacs provide full UTF-8 support, and also have builtin
detection of UTF-8 files. For further information in Vim, use
Nano currently does not provide support for UTF-8, although it has been planned for a long time. With luck, this will change in future. At the time of writing, UTF-8 support is in Nano's CVS, and should be included in the next release.
Currently,
The C shell,
Irssi has complete UTF-8 support, although it does require a user to set an option.
/set term_charset UTF-8
For channels where non-ASCII characters are often exchanged in non-UTF-8
charsets, the
The Mutt mail user agent has very good Unicode support. To use UTF-8 with Mutt,
put the following in your
set send_charset="utf8"(outgoing character set) set charset="utf8"(display character set)
Further information is available from the
There are numerous UTF-8 test websites around.
When using one of the text-only web browsers, make absolutely sure you are using a Unicode-aware terminal.
If you see certain characters displayed as boxes with letters or numbers inside, this means that your font does not have a character for the symbol or glyph that the UTF-8 wants. Instead, it displays a box with the hex code of the UTF-8 symbol.
To enable dead keys in X, you need a layout that supports it. Most European
layouts already have dead keys with the default variant. However, this is not
true of North American layouts. Although there is a degree of inconsistency
between layouts, the easiest solution seems to be to use a layout in the form
"en_US" rather than "us", for example. The layout is set in
Section "InputDevice"
Identifier "Keyboard0"
Driver "kbd"
Option "XkbLayout" "en_US" # Rather than just "us"
(Other Xkb options here)
EndSection
This change will come into effect when your X server is restarted. To apply the
change now, use the
It is probably easiest to describe dead keys with examples. Although the results are locale dependent, the concepts should remain the same regardless of locale. The examples contain UTF-8, so to view them you need to either tell your browser to view the page as UTF-8, or have a UTF-8 locale already configured.
When I press AltGr and [ at once, release them, and then press a, 'ä' is produced. When I press AltGr and [ at once, and then press e, 'ë' is produced. When I press AltGr and ; at once, 'á' is produced, and when I press AltGr and ; at once, release them, and then press e, 'é' is produced.
By pressing AltGr, Shift and [ at once, releasing them, and then pressing a, a
Scandinavian 'å' is produced. Similarly, when I press AltGr, Shift and [ at
once, release
AltGr can be used with alphabetical keys alone. For example, AltGr and m, a Greek lower-case letter mu is produced: 'µ'.