ChatGPT解决这个技术问题 Extra ChatGPT

Example invalid utf8 string?

I'm testing how some of my code handles bad data, and I need a few series of bytes that are invalid UTF-8.

Can you post some, and ideally, an explanation of why they are bad/where you got them?


N
Nemanja Trifunovic

Take a look at Markus Kuhn's UTF-8 decoder capability and stress test file

You'll find examples of many UTF-8 irregularities, including lonely start bytes, continuation bytes missing, overlong sequences, etc.


p
philfreo

In PHP:

$examples = array(
    'Valid ASCII' => "a",
    'Valid 2 Octet Sequence' => "\xc3\xb1",
    'Invalid 2 Octet Sequence' => "\xc3\x28",
    'Invalid Sequence Identifier' => "\xa0\xa1",
    'Valid 3 Octet Sequence' => "\xe2\x82\xa1",
    'Invalid 3 Octet Sequence (in 2nd Octet)' => "\xe2\x28\xa1",
    'Invalid 3 Octet Sequence (in 3rd Octet)' => "\xe2\x82\x28",
    'Valid 4 Octet Sequence' => "\xf0\x90\x8c\xbc",
    'Invalid 4 Octet Sequence (in 2nd Octet)' => "\xf0\x28\x8c\xbc",
    'Invalid 4 Octet Sequence (in 3rd Octet)' => "\xf0\x90\x28\xbc",
    'Invalid 4 Octet Sequence (in 4th Octet)' => "\xf0\x28\x8c\x28",
    'Valid 5 Octet Sequence (but not Unicode!)' => "\xf8\xa1\xa1\xa1\xa1",
    'Valid 6 Octet Sequence (but not Unicode!)' => "\xfc\xa1\xa1\xa1\xa1\xa1",
);

From http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php#54805


D
Dave Moten

The idea of patterns of ill-formed byte-sequences can be gotten from the table of well-formed byte sequences. See "Table 3-7. Well-Formed UTF-8 Byte Sequences" in the Unicode Standard 6.2.

    Code Points    First Byte Second Byte Third Byte Fourth Byte
  U+0000 -   U+007F   00 - 7F
  U+0080 -   U+07FF   C2 - DF    80 - BF
  U+0800 -   U+0FFF   E0         A0 - BF     80 - BF
  U+1000 -   U+CFFF   E1 - EC    80 - BF     80 - BF
  U+D000 -   U+D7FF   ED         80 - 9F     80 - BF
  U+E000 -   U+FFFF   EE - EF    80 - BF     80 - BF
 U+10000 -  U+3FFFF   F0         90 - BF     80 - BF    80 - BF
 U+40000 -  U+FFFFF   F1 - F3    80 - BF     80 - BF    80 - BF
U+100000 - U+10FFFF   F4         80 - 8F     80 - BF    80 - BF

Here are the examples generated from U+24B62. I used them for a bug report: Bug #65045 mb_convert_encoding breaks well-formed character

// U+24B62: "\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD"    ."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD"

The oversimplification of range of trailing bytes([0x80, 0xBF]) can be seen in the various libraries.

// U+0800 - U+0FFF
\xE0\x80\x80

// U+D000 - U+D7FF
\xED\xBF\xBF

// U+10000 -  U+3FFFF
\xF0\x80\x80\x80

// U+100000 - U+10FFFF
\xF4\xBF\xBF\xBF

u
user1015281

,̆ was particularly evil. I see it as combined on ubuntu.

comma-breve


R
RARE Kpop Manifesto

This might not be exactly what OP asked but it's somewhat related :

if you happen to already have byte ordinance values (0 - 255) and wanna know whether a byte# is a valid UTF-8 starting point byte or not, I came up with this strange unified formula that returns a 1 (true) or 0 (false) :

function newUTF8start(__) {  

   return 118^(+__< 194) < (246-__)  }

s
shoosh

Fuzz Testing - generate a random sequence of octets. Most likely you'll get some illegal sequences sooner than later.


There's nothing worse than having heisenbugs or eisentests. Tests pass 10 times, you release the product, test fails.
@EricDuminil ever heard of srand() ?
Fair enough. Could you please mention it in the answer so I could revert my downvote?
Ahah. Well, there's always something new to learn, that's why I come to SO in the first place. I think your srand() advice is a good idea, it might help other people here.
We can create the invalid strings directly, we don't need randomness to try and eventually find them, though string processing libraries would (probably!) benefit from fuzzing just in case.