A couple of things about UTF-8 have eluded me for a while …
I knew that the first bit of ASCII (the bit people agreed on) is the same in ASCII and UTF-8
I knew that the rest of Unicode needs 2 or 3 bytes
But I wasn’t clear how you could tell how many bytes needed to be read at a time
And mostly I didn’t need to because the computer does it all for me - but those bits of vagueness can catch you out and so I went down the rabbit hole and it turns out to be fairly short.
First off : it’s only the first 128 characters (7 bits) that are the same.
In binary these all start with a 0
So in UTF-8 if you read a bytes and it starts zero - you just need that one byte.
Any UTF-8 byte that starts with a 1 is part of a multi-byte sequence
Starting codes in UTF-8 have the following meaning
0 : single byte
110 : start of a two byte sequence
1110 : start of a three byte sequence
11110 : start of a four byte sequence
10 : a byte that is part of a multi-byte sequence
The consequences of this are
all of the less well defined ASCII chars start with a 1 and are not valid UTF-8 single byte characters
you can’t just read any random byte and know what character it is (it might be part of a multi byte character)
But you can read any byte and know if it is a single byte char, part of a multi byte char (and which way to go to read the rest of it)
A more subtle point about characters in extended ASCII like ‘£’ decimal 163 Hex A3 binary 10100011 (which may be encoded as other characters in other flavours of ASCII - but being British the is the one I’m used to)
This is encoded as the two bytes
Decimal [194 163]
Binary [11000010 10100011]
Note that the second byte is the same as in extended ASCII
But it is not valid UTF-8 as a single byte - the leading 10 tells you it is part of a multi byte character
Taking the two bytes
The 110 to start says “this is a two byte character” and don’t form part of the code value
the 10 of the second bytes says “this isn’t the only byte”
What is left is 000010100011 or 163
So 8 bit numbers don’t stay the same in UTF-8 - only 7 bit ones.
But the 8 bit bytes may be there and if you search for a byte you might find it - but then it wouldn’t mean what you think it does.
As illustrated by this comment
str := "Ǵo£lang" data := byte(str) fmt.Println(data) // £ = 163 fmt.Println(strings.IndexByte(str, 163)) // Output: // [199 180 111 194 163 108 97 110 103] // 4
Note that this naive search by byte tells us the byte is at position 4 - when what we want is the character (or run as Go calls them) at position 2
If we tried to search for a run with integer value greater than 8 bits we would get a compile error.
The Wikipedia page on UTF-8 is very good
And Joel Splosky provides a good intro: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
Many years ago I battled with Reading a Unicode Excel file in PHP but UTF-16 is a slightly different beast.