Warning, /utilities/konsole/tests/text-files/UTF-8-test.txt is written in an unsupported language. File is not indexed.
0001 UTF-8 decoder capability and stress test
0002 ----------------------------------------
0003
0004 Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
0005
0006 This test file can help you examine, how your UTF-8 decoder handles
0007 various types of correct, malformed, or otherwise interesting UTF-8
0008 sequences. This file is not meant to be a conformance test. It does
0009 not prescribes any particular outcome and therefore there is no way to
0010 "pass" or "fail" this test file, even though the texts suggests a
0011 preferable decoder behaviour at some places. The aim is instead to
0012 help you think about and test the behaviour of your UTF-8 on a
0013 systematic collection of unusual inputs. Experience so far suggests
0014 that most first-time authors of UTF-8 decoders find at least one
0015 serious problem in their decoder by using this file.
0016
0017 The test lines below cover boundary conditions, malformed UTF-8
0018 sequences as well as correctly encoded UTF-8 sequences of Unicode code
0019 points that should never occur in a correct UTF-8 file.
0020
0021 According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
0022 receiving UTF-8 shall interpret a "malformed sequence in the same way
0023 that it interprets a character that is outside the adopted subset" and
0024 "characters that are not within the adopted subset shall be indicated
0025 to the user" by a receiving device. A quite commonly used approach in
0026 UTF-8 decoders is to replace any malformed UTF-8 sequence by a
0027 replacement character (U+FFFD), which looks a bit like an inverted
0028 question mark, or a similar symbol. It might be a good idea to
0029 visually distinguish a malformed UTF-8 sequence from a correctly
0030 encoded Unicode character that is just not available in the current
0031 font but otherwise fully legal, even though ISO 10646-1 doesn't
0032 mandate this. In any case, just ignoring malformed sequences or
0033 unavailable characters does not conform to ISO 10646, will make
0034 debugging more difficult, and can lead to user confusion.
0035
0036 Please check, whether a malformed UTF-8 sequence is (1) represented at
0037 all, (2) represented by exactly one single replacement character (or
0038 equivalent signal), and (3) the following quotation mark after an
0039 illegal UTF-8 sequence is correctly displayed, i.e. proper
0040 resynchronization takes place immageately after any malformed
0041 sequence. This file says "THE END" in the last line, so if you don't
0042 see that, your decoder crashed somehow before, which should always be
0043 cause for concern.
0044
0045 All lines in this file are exactly 79 characters long (plus the line
0046 feed). In addition, all lines end with "|", except for the two test
0047 lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
0048 U+0000 and U+007F. If you display this file with a fixed-width font,
0049 these "|" characters should all line up in column 79 (right margin).
0050 This allows you to test quickly, whether your UTF-8 decoder finds the
0051 correct number of characters in every line, that is whether each
0052 malformed sequences is replaced by a single replacement character.
0053
0054 Note that as an alternative to the notion of malformed sequence used
0055 here, it is also a perfectly acceptable (and in some situations even
0056 preferable) solution to represent each individual byte of a malformed
0057 sequence by a replacement character. If you follow this strategy in
0058 your decoder, then please ignore the "|" column.
0059
0060
0061 Here come the tests: |
0062 |
0063 1 Some correct UTF-8 text |
0064 |
0065 You should see the Greek word 'kosme': "κόσμε" |
0066 |
0067 2 Boundary condition test cases |
0068 |
0069 2.1 First possible sequence of a certain length |
0070 |
0071 2.1.1 1 byte (U-00000000): " "
0072 2.1.2 2 bytes (U-00000080): "" |
0073 2.1.3 3 bytes (U-00000800): "ࠀ" |
0074 2.1.4 4 bytes (U-00010000): "𐀀" |
0075 2.1.5 5 bytes (U-00200000): "" |
0076 2.1.6 6 bytes (U-04000000): "" |
0077 |
0078 2.2 Last possible sequence of a certain length |
0079 |
0080 2.2.1 1 byte (U-0000007F): ""
0081 2.2.2 2 bytes (U-000007FF): "߿" |
0082 2.2.3 3 bytes (U-0000FFFF): "" |
0083 2.2.4 4 bytes (U-001FFFFF): "" |
0084 2.2.5 5 bytes (U-03FFFFFF): "" |
0085 2.2.6 6 bytes (U-7FFFFFFF): "" |
0086 |
0087 2.3 Other boundary conditions |
0088 |
0089 2.3.1 U-0000D7FF = ed 9f bf = "" |
0090 2.3.2 U-0000E000 = ee 80 80 = "" |
0091 2.3.3 U-0000FFFD = ef bf bd = "�" |
0092 2.3.4 U-0010FFFF = f4 8f bf bf = "" |
0093 2.3.5 U-00110000 = f4 90 80 80 = "" |
0094 |
0095 3 Malformed sequences |
0096 |
0097 3.1 Unexpected continuation bytes |
0098 |
0099 Each unexpected continuation byte should be separately signalled as a |
0100 malformed sequence of its own. |
0101 |
0102 3.1.1 First continuation byte 0x80: "" |
0103 3.1.2 Last continuation byte 0xbf: "" |
0104 |
0105 3.1.3 2 continuation bytes: "" |
0106 3.1.4 3 continuation bytes: "" |
0107 3.1.5 4 continuation bytes: "" |
0108 3.1.6 5 continuation bytes: "" |
0109 3.1.7 6 continuation bytes: "" |
0110 3.1.8 7 continuation bytes: "" |
0111 |
0112 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |
0113 |
0114 " |
0115 |
0116 |
0117 " |
0118 |
0119 3.2 Lonely start characters |
0120 |
0121 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |
0122 each followed by a space character: |
0123 |
0124 " |
0125 " |
0126 |
0127 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |
0128 each followed by a space character: |
0129 |
0130 " " |
0131 |
0132 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |
0133 each followed by a space character: |
0134 |
0135 " " |
0136 |
0137 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |
0138 each followed by a space character: |
0139 |
0140 " " |
0141 |
0142 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |
0143 each followed by a space character: |
0144 |
0145 " " |
0146 |
0147 3.3 Sequences with last continuation byte missing |
0148 |
0149 All bytes of an incomplete sequence should be signalled as a single |
0150 malformed sequence, i.e., you should see only a single replacement |
0151 character in each of the next 10 tests. (Characters as in section 2) |
0152 |
0153 3.3.1 2-byte sequence with last byte missing (U+0000): "" |
0154 3.3.2 3-byte sequence with last byte missing (U+0000): "" |
0155 3.3.3 4-byte sequence with last byte missing (U+0000): "" |
0156 3.3.4 5-byte sequence with last byte missing (U+0000): "" |
0157 3.3.5 6-byte sequence with last byte missing (U+0000): "" |
0158 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" |
0159 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" |
0160 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" |
0161 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" |
0162 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" |
0163 |
0164 3.4 Concatenation of incomplete sequences |
0165 |
0166 All the 10 sequences of 3.3 concatenated, you should see 10 malformed |
0167 sequences being signalled: |
0168 |
0169 "" |
0170 |
0171 3.5 Impossible bytes |
0172 |
0173 The following two bytes cannot appear in a correct UTF-8 string |
0174 |
0175 3.5.1 fe = "" |
0176 3.5.2 ff = "" |
0177 3.5.3 fe fe ff ff = "" |
0178 |
0179 4 Overlong sequences |
0180 |
0181 The following sequences are not malformed according to the letter of |
0182 the Unicode 2.0 standard. However, they are longer then necessary and |
0183 a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 |
0184 decoder" should reject them just like malformed sequences for two |
0185 reasons: (1) It helps to debug applications if overlong sequences are |
0186 not treated as valid representations of characters, because this helps |
0187 to spot problems more quickly. (2) Overlong sequences provide |
0188 alternative representations of characters, that could maliciously be |
0189 used to bypass filters that check only for ASCII characters. For |
0190 instance, a 2-byte encoded line feed (LF) would not be caught by a |
0191 line counter that counts only 0x0a bytes, but it would still be |
0192 processed as a line feed by an unsafe UTF-8 decoder later in the |
0193 pipeline. From a security point of view, ASCII compatibility of UTF-8 |
0194 sequences means also, that ASCII characters are *only* allowed to be |
0195 represented by ASCII bytes in the range 0x00-0x7f. To ensure this |
0196 aspect of ASCII compatibility, use only "safe UTF-8 decoders" that |
0197 reject overlong UTF-8 sequences for which a shorter encoding exists. |
0198 |
0199 4.1 Examples of an overlong ASCII character |
0200 |
0201 With a safe UTF-8 decoder, all of the following five overlong |
0202 representations of the ASCII character slash ("/") should be rejected |
0203 like a malformed UTF-8 sequence, for instance by substituting it with |
0204 a replacement character. If you see a slash below, you do not have a |
0205 safe UTF-8 decoder! |
0206 |
0207 4.1.1 U+002F = c0 af = "" |
0208 4.1.2 U+002F = e0 80 af = "" |
0209 4.1.3 U+002F = f0 80 80 af = "" |
0210 4.1.4 U+002F = f8 80 80 80 af = "" |
0211 4.1.5 U+002F = fc 80 80 80 80 af = "" |
0212 |
0213 4.2 Maximum overlong sequences |
0214 |
0215 Below you see the highest Unicode value that is still resulting in an |
0216 overlong sequence if represented with the given number of bytes. This |
0217 is a boundary test for safe UTF-8 decoders. All five characters should |
0218 be rejected like malformed UTF-8 sequences. |
0219 |
0220 4.2.1 U-0000007F = c1 bf = "" |
0221 4.2.2 U-000007FF = e0 9f bf = "" |
0222 4.2.3 U-0000FFFF = f0 8f bf bf = "" |
0223 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" |
0224 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" |
0225 |
0226 4.3 Overlong representation of the NUL character |
0227 |
0228 The following five sequences should also be rejected like malformed |
0229 UTF-8 sequences and should not be treated like the ASCII NUL |
0230 character. |
0231 |
0232 4.3.1 U+0000 = c0 80 = "" |
0233 4.3.2 U+0000 = e0 80 80 = "" |
0234 4.3.3 U+0000 = f0 80 80 80 = "" |
0235 4.3.4 U+0000 = f8 80 80 80 80 = "" |
0236 4.3.5 U+0000 = fc 80 80 80 80 80 = "" |
0237 |
0238 5 Illegal code positions |
0239 |
0240 The following UTF-8 sequences should be rejected like malformed |
0241 sequences, because they never represent valid ISO 10646 characters and |
0242 a UTF-8 decoder that accepts them might introduce security problems |
0243 comparable to overlong UTF-8 sequences. |
0244 |
0245 5.1 Single UTF-16 surrogates |
0246 |
0247 5.1.1 U+D800 = ed a0 80 = "" |
0248 5.1.2 U+DB7F = ed ad bf = "" |
0249 5.1.3 U+DB80 = ed ae 80 = "" |
0250 5.1.4 U+DBFF = ed af bf = "" |
0251 5.1.5 U+DC00 = ed b0 80 = "" |
0252 5.1.6 U+DF80 = ed be 80 = "" |
0253 5.1.7 U+DFFF = ed bf bf = "" |
0254 |
0255 5.2 Paired UTF-16 surrogates |
0256 |
0257 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" |
0258 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" |
0259 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" |
0260 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" |
0261 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" |
0262 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" |
0263 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" |
0264 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" |
0265 |
0266 5.3 Other illegal code positions |
0267 |
0268 5.3.1 U+FFFE = ef bf be = "" |
0269 5.3.2 U+FFFF = ef bf bf = "" |
0270 |
0271 THE END |