UTF-8
维基百科，自由的百科全书
UTF-8（8位元的Universal Character Set/Unicode Transformation Format）是Unicode的其中一種變動長度字元編碼。它可以用來代表Unicode標準中的任何通用字元，然而UTF-8中，位元組及位元的起始編碼的分派與ASCII是一致的（原來處理ASCII的軟件在不需要作出變動或作出很小的變動後，使可維持其他功能）。基於以上的原因，在電子郵件、網頁及其他使用利用位元來儲存或streamed的地方中，它穩定地成為優先採用的編碼。
UTF-8中，每一字元依據Unicode符號，採用一至四個位元組。128 US-ASCII位元組(Unicode範圍由U+0000至U+007F)只需要一個位元組用作編碼。帶有變音符號的拉丁文、希臘文, 西里爾字母, 亞美尼亞語, 希伯來文, 阿拉伯文, 敘利亞文 及馬爾地夫語(Unicode範圍由U+0080至U+07FF)需要二個位元組用作編碼。其他基本多文種平面（BMP）中的字元使用三個位元組作編號（BMP差不多包含所有常用的字元）。四位元組編碼將用在輔助平面字元.
利用四個位元組作為一個字元字似乎是利用過多的系統空間，但字符通常都包含在BMP之內，而且，UTF-16（UTF-8的主要替代編碼）同樣需要四個位元組用作以上字符的編碼。UTF-8和UTF-16的效率因應字符的分佈範圍而有所不同，不過，透過使用一些傳統的壓縮系統可以使不同的編碼系統中的差異變得微不足道。當傳統的演算法在一些較短的文字上表演得不太好及資料大小很重要時，可以考慮使用Standard Compression Scheme for Unicode（SCSU）。
互聯網工程工作小組 (IETF)要求所有網際網路協議使用UTF-8字元編碼，至少是其中一種可支援的編碼。[1] The 互聯網郵件聯盟 (IMC)建議所有用來觀看及收發電子郵件的軟件使用UTF-8作為編碼。所有主要的電子郵件軟體中，只有Eudora不支援UTF-8編碼。[1]
目录 |
[编辑] 歷史
1992年早期，為建立良好的位元組串編碼（byte-stream encoding）供多位元組字節（multi-byte character sets）使用，一個正式的研究開始了。ISO 10646的初稿中有一個不被需要的附錄，名為UTF。當中包含了一個供32位元的字元使用的位元組串編碼方式，這個編碼方式並沒有令人滿意的性能基礎，但提出將0-127的範圍保留給ASCII的概念，從而提供了對應舊系統的兼容性。
1992年7月，X/Open委員會XoJIG開始尋求一個較佳的編碼方式，UNIX 系統實驗室（UNIX System Laboratories, USL）的Dave Prosser提交了一份計劃書，當中包含一個有較快完成編碼特性的系統及介紹了一個改進處，其中，7位元的ASCII符號只代表原來的意思，所有連續的多位元組只會包含8位元符號，即那此被設定了高位元的。
1992年8月，這份計劃書由IBMX/Open的代表流傳給有興趣的團體。與始同時，貝爾實驗室Plan 9作業系統工作小組的肯·汤普逊對這編碼系統作出重大的修改，使到編碼可以自我協調（self-synchronizing），不必讀取字串的開首，也能找出字元的編碼範圍。1992年9月2日，汤普逊的設計初稿在美國新澤西州一架餐車的餐桌墊上有了定案。接下來的日子，Pike及汤普逊將此計劃完成，並將這編碼系統完全使用在Plan 9當中，及後在X/Open中將他們的成果散佈出去。
1993年1月25-29日的在聖地牙哥舉行的USENIX會議中，作出第一次正式的發佈。
1996年微軟明確地說明旗下的CAB（MS Cabinet）全面支援UTF-8編碼制式的字串，但事實上UTF-8的編碼器從來沒有加以實行。
[编辑] 描述
現時有好幾份在定義上有小許不同的文件是關於UTF-8的：
- RFC 3629 / STD 63（2003），這份文件制定了UTF-8是標準的網際網路協議元素
- 第四版，The Unicode Standard，§3.9－§3.10（2003）
- ISO/IEC 10646-1:2000附加文件D（2000）
以上的定義取代了以下那些被淘汰的定義：
- ISO/IEC 10646-1:1993修正案2／附加文件R（1996）
- 第二版，The Unicode Standard，附錄A（1996）
- RFC 2044（1996）
- RFC 2279（1998）
- 第三版，The Unicode Standard，§2.3（2000）及勘誤表#1：UTF-8 Shortest Form（2000）
- Unicode Standard 附加文件#27: Unicode 3.1（2001）
事實上，所有定義的基本原理都是相同的，它們之間最主要的不同是能夠支援的字元範圍及無效輸入的處理方法。
Unicode字元的位元在UTF-8的位元組較低的位元中被分為不同的組別。在U+0080的以下字元都使用內含其字元單一位元組編碼，這些編碼與7位元的ASCII字元是相對應的。在一些需要使用四個位元組的案例中，最高有效位元會設定成1，防止與7位元的ASCII字元混淆及保持標準位元組主導字串（standard byte-oriented string）運作順利。
| 代碼範圍 十六進制 |
標量值(scalar value 二進制 |
UTF-8 二進制 / 十六進制 |
註釋 |
|---|---|---|---|
| 000000 - 00007F 128個代碼 |
00000000 00000000 0zzzzzzz | 0zzzzzzz(00-7F) | ASCII等值範圍，位元組由零開始 |
| 七個z | 七個z | ||
| 000080 - 0007FF 1920個代碼 |
00000000 00000yyy yyzzzzzz | 110yyyyy(C2-DF) 10zzzzzz(80-BF) | 第一個位元組由110開始，接著的位元組由10開始 |
| 三個y；二個y；六個z | 五個y；六個z | ||
| 000800 - 00FFFF 63488個代碼 |
00000000 xxxxyyyy yyzzzzzz | 1110xxxx(E0-EF) 10yyyyyy 10zzzzzz | 第一個位元組由1110開始，接著的位元組由10開始 |
| 四個x；四個y；二個y；六個z | 四個x；六個y；六個z | ||
| 010000 - 10FFFF 1048576個代碼 |
000wwwxx xxxxyyyy yyzzzzzz | 11110www(F0-F4) 10xxxxxx 10yyyyyy 10zzzzzz | 由11110開始，接著的位元組由10開始 |
| 三個w；二個x；四個x；四個y；二個y；六個z | 三個w；六個x；六個y；六個z |
For example, the character aleph (?), which is Unicode U+05D0, is encoded into UTF-8 in this way:
- It falls into the range of U+0080 to U+07FF. The table shows it will be encoded using two bytes, 110yyyyy 10zzzzzz.
- Hexadecimal 0x05D0 is equivalent to binary 101-1101-0000.
- The eleven bits are put in their order into the positions marked by "y"-s and "z"-s: 11010111 10010000.
- The final result is the two bytes, more conveniently expressed as the two hexadecimal bytes 0xD7 0x90. That is the encoding of the character aleph (?) in UTF-8.
So the first 128 characters (US-ASCII) need one byte. The next 1920 characters need two bytes to encode. This includes Latin alphabet characters with diacritics, Greek, Cyrillic, Coptic, Armenian, Hebrew, and Arabic characters. The rest of the BMP characters use three bytes, and additional characters are encoded in four bytes.
By continuing the pattern given above it is possible to deal with much larger numbers. The original specification allowed for sequences of up to six bytes covering numbers up to 31 bits (the original limit of the universal character set). However, UTF-8 was restricted by RFC 3629 to use only the area covered by the formal Unicode definition, U+0000 to U+10FFFF, in November 2003. With these restrictions, the following byte values never appear in a legal UTF-8 sequence:
| Codes (binary) | Codes (hexadecimal) | Notes |
|---|---|---|
| 1100000x | C0, C1 | Overlong encoding: lead-byte of a 2 byte sequence, but code point <= 127 |
| 1111111x | FE, FF | Invalid: lead-byte of a 7 or 8 byte sequence |
| 111110xx 1111110x |
F8, F9, FA, FB, FC, FD | Restricted by RFC 3629: lead-byte of a 5 or 6 byte sequence |
| 11110101 1111011x |
F5, F6, F7 | Restricted by RFC 3629: lead byte of codepoint above 10FFFF |
While the last two categories were technically allowed by earlier UTF-8 specifications, no characters were ever assigned to the code points they represent so they should never have appeared in actual text.
The design of the algorithm has some similarities with Huffman coding.
[编辑] UTF-8的分支
[编辑] Java
In normal usage, the Java programming language supports standard UTF-8 when reading and writing strings through Template:Javadoc:SE and Template:Javadoc:SE.
However, Java also supports a non-standard variant of UTF-8 called Template:Javadoc:SE for object serialization, for the Java Native Interface, and for embedding constants in class files. There are two differences between modified and standard UTF-8. The first difference is that the null character (U+0000) is encoded with two bytes instead of one, specifically as 11000000 10000000. This ensures that there are no embedded nulls in the encoded string, presumably to address the concern that if the encoded string is processed in a language such as C where a null byte signifies the end of a string, an embedded null would cause the string to be truncated.
The second difference is in the way characters outside the Basic Multilingual Plane are encoded. In standard UTF-8 these characters are encoded using the four-byte format above. In modified UTF-8 these characters are first represented as surrogate pairs (as in UTF-16), and then the surrogate pairs are encoded individually in sequence as in CESU-8. The reason for this modification is more subtle. In Java a character is 16 bits long; therefore some Unicode characters require two Java characters in order to be represented. This aspect of the language predates the supplementary planes of Unicode; however, it is important for performance as well as backwards compatibility, and is unlikely to change. The modified encoding ensures that an encoded string can be decoded one UTF-16 code unit at a time, rather than one Unicode code point at a time. Unfortunately, this also means that characters requiring four bytes in UTF-8 require six bytes in modified UTF-8.
Because modified UTF-8 is not UTF-8, one needs to be very careful to avoid mislabelling data in modified UTF-8 as UTF-8 when interchanging information over the Internet.
[编辑] Mac OS X
The Mac OS X Operating System uses canonically decomposed Unicode, encoded using UTF-8 for file names in the filesystem. This is sometimes referred to as UTF-8-MAC. In canonically decomposed Unicode, the use of precomposed characters is forbidden and combining diacritics must be used to replace them.
This makes sorting far simpler but can be confusing for software built around the assumption that precomposed characters are the norm and combining diacritics only used to form unusual combinations. This is an example of the NFD variant of Unicode normalization - most other platforms, including Windows and Linux, use the NFC form of Unicode normalization, which is also used by W3C standards, so NFD data must typically be converted to NFC for use on other platforms or the Web.
This is discussed in Apple Q&A 1173.
[编辑] 設計UTF-8的理由
As a consequence of the design of UTF-8, the following properties of multi-byte sequences hold:
- The most significant bit of a single-byte character is always
0. - The most significant bits of the first byte of a multi-byte sequence determine the length of the sequence. These most significant bits are
110for two-byte sequences;1110for three-byte sequences, and so on. - The remaining bytes in a multi-byte sequence have
10as their two most significant bits.
UTF-8 was designed to satisfy these properties in order to guarantee that no byte sequence of one character is contained within a longer byte sequence of another character. This ensures that byte-wise sub-string matching can be applied to search for words or phrases within a text; some older variable-length 8-bit encodings (such as Shift-JIS) did not have this property and thus made string-matching algorithms rather complicated. Although this property adds redundancy to UTF-8–encoded text, the advantages outweigh this concern; besides, data compression is not one of Unicode's aims and must be considered independently. This also means that if one or more complete bytes are lost due to error or corruption, one can resynchronize at the beginning of the next character and thus limit the damage.
Also due to the design of the byte sequences, if a sequence of bytes supposed to represent text validates as UTF-8 then it is fairly safe to assume it is UTF-8. The chance of a random sequence of bytes being valid UTF-8 and not pure ASCII is 1 in 32 for a 2 byte sequence, 5 in 256 for a 3 byte sequence and even lower for longer sequences.
While natural languages encoded in traditional encodings are far from random byte sequences they are also unlikely to produce byte sequences that would pass a UTF-8 validity test and then be misinterpreted (obviously pure ASCII text would pass a UTF-8 validity test but provided the legacy encodings under consideration are also ASCII based this is not a problem). For example, for ISO-8859-1 text to be misrecognized as UTF-8, the only non-ASCII characters in it would have to be in sequences starting with either an accented letter or the multiplication symbol and ending with a symbol.
The bit patterns can be used to identify UTF-8 characters. If the byte's first hex code begins with 0–7, it is an ASCII character. If it begins with C or D, it is an 11 bit character (expressed in two bytes). If it begins with E, it is 16 bit (expressed in 3 bytes), and if it begins with F, it is 21 bits (expressed in 4 bytes). 8 through B cannot be first hex codes, but all following bytes must begin with a hex code between 8 through B. Thus, at a glance, it can be seen that "0xA9" is not a valid UTF-8 character, but that "0x54" or "0xE3 0xB4 0xB1" is a valid UTF-8 character.
[编辑] 過長的資料排列（overlong forms）、輸入無效及保安的考慮
The exact response required of a UTF-8 decoder on invalid input is not uniformly defined by the standards. In general, there are several ways a UTF-8 decoder might behave in the event of an invalid byte sequence:
- Insert a replacement character (e.g. '?', '�').
- Ignore the bytes.
- Interpret the bytes according to a different character encoding (often the ISO-8859-1 character map).
- Not notice and decode as if the bytes were some similar bit of UTF-8.
- Stop decoding and report an error (possibly giving the caller the option to continue).
It is possible for a decoder to behave in different ways for different types of invalid input.
RFC 3629 only requires that UTF-8 decoders must not decode "overlong sequences" (where a character is encoded in more bytes than needed but still adheres to the forms above). The Unicode Standard requires a Unicode-compliant decoder to "…treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."
Overlong forms are one of the most troublesome types of UTF-8 data. The current RFC says they must not be decoded but older specifications for UTF-8 only gave a warning and many simpler decoders will happily decode them. Overlong forms have been used to bypass security validations in high profile products including Microsoft's IIS web server. Therefore, great care must be taken to avoid security issues if validation is performed before conversion from UTF-8, and it is generally much simpler to handle overlong forms before any input validation is done.
To maintain security in the case of invalid input, there are two options. The first is to decode the UTF-8 before doing any input validation checks. The second is to use a decoder that, in the event of invalid input, returns either an error or text that the application considers to be harmless. Another possibility is to avoid conversion out of UTF-8 altogether but this relies on any other software that the data is passed to safely handling the invalid data.
Another consideration is error recovery. To guarantee correct recovery after corrupt or lost bytes, decoders must be able to recognise the difference between lead and trail bytes, rather than just assuming that bytes will be of the type allowed in their position.
[编辑] 優點及缺點
A note about string length:
In general, it is not possible to determine from the number of code points in a Unicode string how much space it needs to be displayed, or where on a screen the cursor should be placed in a text buffer after displaying a string; combining characters, variable-width fonts, non-printing characters and right-to-left characters all contribute to this.
So while the number of octets in an UTF-8 string is related in a more complex way to the number of code points than for UTF-32, it is very rare to encounter a situation where this makes a difference in practice.
- General
- Advantages
- UTF-8 is a superset of ASCII. Since a plain ASCII string is also a valid UTF-8 string, no conversion needs to be done for existing ASCII text. Software designed for traditional extended ASCII character sets can generally be used with UTF-8 with few or no changes.
- Sorting of UTF-8 strings using standard byte-oriented sorting routines will produce the same results as sorting them based on Unicode code points. (This has limited usefulness, though, since it is unlikely to represent the culturally acceptable sort order of any particular language or locale.)
- UTF-8 and UTF-16 are the standard encodings for XML documents. All other encodings must be specified explicitly either externally or through a text declaration. [2]
- Any byte oriented string search algorithm can be used with UTF-8 data (as long as one ensures that the inputs only consist of complete UTF-8 characters). Care must be taken with regular expressions and other constructs that count characters, however.
- UTF-8 strings can be fairly reliably recognized as such by a simple algorithm. That is, the probability that a string of characters in any other encoding appears as valid UTF-8 is low, diminishing with increasing string length. For instance, the octet values C0, C1, F5 to FF never appear. For better reliability, regular expressions can be used to take into account illegal overlong and surrogate values (see the W3 FAQ: Multilingual Forms for a Perl regular expression to validate a UTF-8 string).
- Disadvantages
- A badly-written (and not compliant with current versions of the standard) UTF-8 parser could accept a number of different pseudo-UTF-8 representations and convert them to the same Unicode output. This provides a way for information to leak past validation routines designed to process data in its eight-bit representation.
- Advantages
- Compared to single-byte legacy encodings
- Advantages
- UTF-8 can encode any Unicode character, avoiding the need to figure out and set a "code page" or otherwise indicate what character set is in use, and allowing output in multiple languages at the same time.
- Disadvantages
- UTF-8 is larger than the appropriate legacy encoding for everything except diacritic-free, Latin-alphabet text.
- Legacy encodings using a single byte per character make string cutting and joining easy even with simple-minded APIs.
- Advantages
- Compared to multi-byte legacy encodings
- Advantages
- UTF-8 can encode any Unicode character. In most cases, legacy encodings can be converted to Unicode and back with no loss and—as UTF-8 is an encoding of Unicode—this applies to it too.
- Character boundaries are easily found from anywhere in an octet stream (scanning either forwards or backwards). This implies that if a stream of bytes is scanned starting in the middle of a multibyte sequence, only the information represented by the partial sequence is lost and decoding can begin correctly on the next character. Similarly, if a number of bytes are corrupted or dropped then correct decoding can resume on the next character boundary. Many legacy multi-byte encodings are much harder to resynchronise.
- A byte sequence for one character never occurs as part of a longer sequence for another character as it did in older variable-length encodings like Shift-JIS (see the previous section on this). For instance, US-ASCII octet values do not appear otherwise in a UTF-8 encoded character stream. This provides compatibility with file systems or other software (e.g., the printf() function in C libraries) that parse based on US-ASCII values but are transparent to other values.
- The first byte of a multibyte sequence is enough to determine the length of the multibyte sequence. This makes it extremely simple to extract a substring from a given string without elaborate parsing. This was often not the case in legacy multibyte encodings.
- Efficient to encode using simple bit operations. UTF-8 does not require slower mathematical operations such as multiplication or division (unlike the obsolete UTF-1 encoding).
- Disadvantages
- UTF-8 is generally larger than the appropriate legacy encoding for everything except diacritic-free, Latin-alphabet text. East Asian scripts generally had two bytes per character in their legacy encodings yet take three bytes per character in UTF-8.
- Advantages
- Compared to UTF-7
- Advantages
- UTF-8 uses significantly fewer bytes per character for all non-ASCII characters.
- UTF-8 encodes "+" as itself whereas UTF-7 encodes it as "+-".
- Disadvantages
- UTF-8 requires the transmission system to be eight-bit clean. In the case of e-mail this means it has to be further encoded using quoted printable or base64 in some cases. This extra stage of encoding carries a significant size penalty. However, this disadvantage is not so important an issue anymore because most mail transfer agents in modern use are eight-bit clean and support 8BITMIME SMTP extension as specified in RFC 1869.
- Advantages
- Compared to UTF-16
- Advantages
- Byte values of 0 (The ASCII NUL character) do not appear in the encoding unless U+0000 (the Unicode NUL character) is represented. This means that legacy C library string functions (such as strcpy()) that use a null terminator will not incorrectly truncate strings.
- Since ASCII characters can be represented in a single byte, text consisting of mostly diacritic-free Latin letters will be around half the size in UTF-8 than it would be in UTF-16. Text in many other alphabets will be slightly smaller in UTF-8 than it would be in UTF-16 because of the presence of spaces.
- Most existing computer programs (including operating systems) were not written with Unicode in mind, and using UTF-16 with them would create major compatibility issues, as it is not a superset of ASCII. UTF-8 allows programs to treat ASCII as they always did, and changes behaviour only for non-ASCII characters that were different by location anyway.
- In UTF-8, characters outside the basic multilingual plane are not a special case.
- UTF-8 uses a byte as its atomic unit while UTF-16 uses a 16-bit word which is generally represented by a pair of bytes. This representation raises a couple of potential problems of its own.
- When representing a word in UTF-16 as two bytes, the order of those two bytes becomes an issue. A variety of mechanisms can be used to deal with this issue (for example, the Byte Order Mark), but they still present an added complication for software and protocol design.
- If an odd number of bytes are removed from the beginning of UTF-16-encoded text, the result will be either invalid UTF-16 or completely meaningless text. In UTF-8, if part of a multi-byte character is removed, only that character is affected and not the rest of the text.
- UTF-16 is often mistaken to be constant-length, leading to code that works for most text but suddenly fails for non-BMP characters. Retrofitting code tends to be hard, so it's better to implement support for the entire range of Unicode from the start.
- Disadvantages
- Characters above U+0800 in the BMP use three bytes in UTF-8, but only two in UTF-16. As a result, text in [for example] Chinese, Japanese or Hindi takes up more space when represented in UTF-8. However, this advantage is partly offset by the fact that characters below U+0080 (Latin letters, numbers and punctuation marks, space, carriage return and line feed) that frequently appear in those text take only one byte in UTF-8 while they take two bytes in UTF-16.
- Advantages
[编辑] 使用UTF-8的原因
ASCII轉换成UCS-2，在編碼前插入一個0x0。用這些編碼，會含括一些控制符，比如 " 或 '/'，這在UNIX和一些C函數中，將會産生嚴重錯誤。因此可以肯定，UCS-2不適合作為Unicode的外部編碼，也因此誕生了UTF-8。
[编辑] UTF-8的編碼方式
UTF-8是UNICODE的一種變長度的編碼表達方式 〈一般UNICODE為雙位元組(指UCS2)〉，它由Ken Thompson于1992年建立，現在已經標准化為RFC 3629。UTF-8就是以8位为单元对UCS进行编码，而UTF-8不使用大尾序和小尾序的形式，每個使用UTF-8儲存的字符，除了第一個字節外，其餘字節的頭兩個位元都是以 "10" 開始，使文字處理器能夠較快地找出每個字符的開始位置。
但為了與以前的ASCII碼相容 (ASCII為一個位元組)，因此 UTF-8 選擇了使用可變長度字節來儲存 Unicode：
| UCS-2编码 | UTF-8字节流 |
|---|---|
| U-00000000 – U-0000007F: | 0xxxxxxx |
| U-00000080 – U-000007FF: | 110xxxxx 10xxxxxx |
| U-00000800 – U-0000FFFF: | 1110xxxx 10xxxxxx 10xxxxxx |
| U-00010000 – U-001FFFFF: | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
| U-00200000 – U-03FFFFFF: | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
| U-04000000 – U-7FFFFFFF: | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
-
- 在ASCII碼的範圍，用一個位元組表示，超出ASCII碼的範圍就用位元組表示，這就形成了我們上面看到的UTF-8的表示方法，這様的好處是當UNICODE文件中只有ASCII碼時，儲存的文件都為一個位元組，所以就是普通的ASCII文件無異，讀取的時候也是如此，所以能與以前的ASCII文件相容。
-
- 大於ASCII碼的，就會由上面的第一位元組的前幾位表示該unicode字元的長度，比如110xxxxxx前三位的二進位表示告訴我們這是個 2BYTE的UNICODE字元；1110xxxx是個三位的UNICODE字元，依此類推；xxx 的位置由字符編碼數的二進製表示的位填入. 越靠右的 x 具有越少的特殊意義.只用最短的那個足夠表達一個字符編碼數的多字節串. 注意在多字節串中, 第一個字節的開頭"1"的數目就是整個串中字節的數目.。
ASCII字母繼續使用1字節儲存，重音文字、希臘字母或西里爾字母等使用2字節來儲存，而常用的漢字就要使用3字節。辅助平面字元則使用4字節。
在UTF-8文件的開首，很多時都放置一個U+FEFF字符 (UTF-8 以 EF,BB,BF 代表)，以顯示這個文字檔案是以UTF-8編碼。
[编辑] UTF-8的特性
| UTF-8 | |
|---|---|
| Smallest code point | 0000 |
| Largest code point | 10FFFF |
| Code unit size | 8 bits |
| Byte order | N/A |
| Minimal bytes/character | 1 |
| Maximal bytes/character | 4 |
- UCS 字符 U+0000 到 U+007F (ASCII) 被編碼為字節 0x00 到 0x7F (ASCII 兼容)，這也意味著只包含 7 位 ASCII 字符的文件在 ASCII 和 UTF-8 兩種編碼方式下是一樣的.
- 所有 >U+007F 的 UCS 字符被編碼為一個多個字節的串, 每個字節都有標記位集。因此，ASCII 字節 (0x00-0x7F) 不可能作為任何其他字符的一部分。
- 表示非 ASCII 字符的多字節串的第一個字節總是在 0xC0 到 0xFD 的範圍裡，並指出這個字符包含多少個字節。多字節串的其餘字節都在 0x80 到 0xBF 範圍裡，這使得重新同步非常容易，並使編碼無國界，且很少受丟失字節的影響。
- 可以編入所有可能的 231個 UCS 代碼
- UTF-8 編碼字符理論上可以最多到 6 個字節長, 然而 16 位 BMP 字符最多只用到 3 字節長。
- Bigendian UCS-4 字節串的排列順序是預定的。
- 字節 0xFE 和 0xFF 在 UTF-8 編碼中從未用到，同時，UTF-8以位元組為編碼單元，它的位元組順序在所有系統中都是一様的，没有位元組序的問題，也因此它實際上并不需要BOM。
- 與 UTF-16 或其他 Unicode 編碼相比，對於不支援 Unicode 和 XML 的系統，UTF-8 更不容易造成問題。
【註】
- UTF為UCS / Unicode Transformation Format“Unicode轉換格式”的縮寫。
- UCS的中文全稱為：信息技術--通用多八位編碼字符集 (Universal Multi-octet Coded Character Set)，由ISO/IEC 10646 標準描述。
[编辑] UTF-8編碼的缺點
[编辑] 不利于正则表达式检索
正则表达式可以进行很多英文高级的模糊检索。例如，[a-h]表示a到h间所有字母。
同样GBK编码的中文也可以这样利用正则表达式，比如在只知道一个字的读音而不知道怎么写的情况下，也可用正则表达式检索，因为GBK编码是按读音排序的。只是UTF-8不是按读音排序的，所以会对正则表达式检索造成不利影响。但是這種使用方式並未考慮中文中的破音字，因此影響不大。Unicode是按部首排序的，因此在只知道一個字的部首而不知道如何發音的情况下, UTF-8 可用正则表达式检索而GBK不行。
[编辑] 其他
與其他 Unicode 編碼相比，特別是UTF-16，在 UTF-8 中 ASCII 字元佔用的空間只有一半，可是在一些字元的 UTF-8 編碼佔用的空間就要多出，特別是中文、日文和韓文（CJK）這樣的象形文字，所以具體因素因文檔而異，但不論哪種情況，差別都不可能很明顯。
[编辑] 註釋
[编辑] 參考
- Alt codes
- ASCII
- Byte Order Mark
- Comparison of email clients#Features
- Comparison of Unicode encodings
- Character encodings in HTML
- ISO 8859
- iconv - a standardized API used to convert between different character encodings
- GB18030
- UTF-8 in URIs
- Unicode and e-mail
- Unicode and HTML
- Universal Character Set
- UTF-16
- UTF-9 and UTF-18
[编辑] 由Unicode Consortium出版的書
- The Unicode Standard, Version 5.0, Fifth Edition, The Unicode Consortium, Addison-Wesley Professional, 27 October 2006. ISBN 0-321-48091-0
- The Unicode Standard, Version 4.0, The Unicode Consortium, Addison-Wesley Professional, 27 August 2003. ISBN 0-321-18578-1
[编辑] 外部連結
- RFC 3629：UTF-8 標準
- RFC 2277：IETF policy on character sets and languages
- Rob Pike tells the story of UTF-8's creation
- Original UTF-8 paper
- UTF-8 test pages by University Hannover and the World Wide Web Consortium
- Unix/Linux: UTF-8和Unicode的常見問題集, Linux Unicode HOWTO, UTF-8 and Gentoo
- The Unicode/UTF-8-character table displays UTF-8 in a variety of formats (with Unicode and HTML encoding information)
- Online Tool for URL encoding/decoding according to RFC 3986 and RFC 3629 (JavaScript, GPL)
- UTF-8 測試頁
- 另一個 UTF-8 測試頁
- UTF-8
- UTF-8 and Debian
| Unicode 相關的條目 |
|---|
| ISO 10646 通用字符集 | UTF-7 | UTF-8 | UTF-16 / UCS-2 | UTF-32 / UCS-4 |
| Unicode编码表 | 基本多文種平面 | 辅助平面 | 中日韓統一表意文字 | CJKV | IICore |
