Ode to a Shipping Label

A poem about mojibake <https://imgur.com/4J7Il0m>, whose original author might be Carlos Bueno on Facebook <https://www.facebook.com/cmb/posts/619241744770551:0>, shows a shipping label that serves as an excellent example for this section, addressed to the surname L&AMP;AMP;ATILDE;&AMP;AMP;SUP3;PEZ. [image: A package addressed to a name including "L&AMP;AMP;ATILDE;&AMP;AMP;SUP3;PEZ"] [image]

We can use ftfy not only to fix the text that was on the label, but to show us what happened to it (like the poem does):

>>> from ftfy import fix_and_explain, apply_plan
>>> shipping_label = "LóPEZ"
>>> fixed, explanation = fix_and_explain(shipping_label)
>>> fixed
'LóPEZ'
>>> explanation
[('apply', 'unescape_html'),


 ('apply', 'unescape_html'),


 ('apply', 'unescape_html'),


 ('encode', 'latin-1'),


 ('decode', 'utf-8')]

The capitalization is inconsistent because the encoding of a lowercase "ó" is in there, but everything was printed in capital letters.

The explanation may even be able to be applied to different text with the same problem:

>>> label2 = "CARRé"
>>> apply_plan(label2, explanation)
'CARRé'

Functions that fix text

The function that you'll probably use most often is ftfy.fix_text(), which applies all the fixes it can to every line of text, and returns the fixed text.

ftfy.fix_and_explain() takes the same arguments as ftfy.fix_text(), but provides an explanation, like we saw in the first section.

Unlike ftfy.fix_text(), ftfy.fix_and_explain() doesn't separate the text into lines that it fixes separately -- because it's looking for a unified explanation of what happened to the text, not a different one for each line.

A more targeted function is ftfy.fix_encoding_and_explain(), which only fixes problems that can be solved by encoding and decoding the text, not other problems such as HTML entities:

This function has a counterpart that returns just the fixed string, without the explanation. It still fixes the string as a whole, not line by line.

The return type of the ..._and_explain functions is a kind of NamedTuple called ExplainedText:

These explanations can be re-applied to text using apply_plan():

Showing the characters in a string

A different kind of explanation you might need is simply a breakdown of what Unicode characters a string contains. For this, ftfy provides a utility function, ftfy.explain_unicode().

A command-line utility that provides similar information, and even more detail, is lunasorcery's utf8info <https://github.com/lunasorcery/utf8info>.

The TextFixerConfig object

The top-level functions of ftfy take a config argument that is an instance of ftfy.TextFixerConfig. If this argument is None, the configuration will use its default values.

Keyword arguments

The top-level functions also accept keyword arguments in place of a config argument. Given these keyword arguments, they will pass them to the ftfy.TextFixerConfig constructor, overriding the default values of those configuration options.

I really need to guess the encoding of some bytes

I understand. Sometimes we can't have nice things.

Though it's not part of the usual operation of ftfy, ftfy does contain a byte-encoding-guesser that tries to be less terrible than other byte-encoding-guessers in common cases. Instead of using probabilistic heuristics, it picks up on very strong signals like "having a UTF-16 byte-order mark" or "decoding successfully as UTF-8".

This function won't solve everything. It can't solve everything. In particular, it has no capacity to guess non-Unicode CJK encodings such as Shift-JIS or Big5.

Read the Python Unicode HOWTO

The Python Unicode HOWTO <https://docs.python.org/3/howto/unicode.html> is a useful introduction to how to use Unicode correctly in Python. If you find yourself confused about the difference between bytes and characters, or you need to unlearn bad habits from Python 2, it's a great place to start.

Assume UTF-8

Assume text is in UTF-8 unless you have a specific reason to believe it isn't.

In the 2020s, UTF-8 is everywhere <http://utf8everywhere.org/>, especially in text meant to be transferred over the Internet. Most mojibake comes from decoding correct UTF-8 as if it were some other encoding.

In Python 3, you should use the Unicode string type (str) for all operations. You should open text files in UTF-8:

openfile = open(filename, encoding='utf-8', errors='replace')

When you are specifically working with bytes and you need to turn them into text, you should decode them as UTF-8:

text = bytebuffer.decode('utf-8', 'replace')

The exceptions, the cases where you're not using UTF-8, are few but relevant. If you're interacting with C APIs, you'll need to represent your text as bytes in the format the API expects. Windows APIs in particular expect UTF-16.

We're mostly past the dark days when encodings were "character maps" of 256 possible characters, one byte per character. An unfortunate thing that keeps them alive is Microsoft Excel, whose "Export" feature will pick a 256-character encoding based on your computer's operating system and default language. So:

Don't export CSVs from Excel

I know that I'm telling you not to do something that may seem like a requirement of doing your job. But don't export CSV files from Excel if you have any other choice. Though Excel CSVs look right on basic ASCII characters, on any other text, it either won't work or won't do what you want. Excel CSVs aren't even interoperable between different computers.

My recommendation is to use Google Sheets to create CSVs, and keep Excel files in .xlsx format so the Unicode won't be mangled.

If you must export a CSV-like file from Excel, you can find an option to tell Excel to export in "Unicode Text", and it will create a tab-separated UTF-16 file. This is not a very widely-used format, but at least it's not mojibake.

You can follow these unwieldy directions from a SalesForce help article <https://help.salesforce.com/articleView?id=000324657&type=1&mode=1> to use Excel and Notepad to create a UTF-8 CSV. You can see why I don't recommend this process.

Don't use chardet

Encoding detection on raw bytes is not a good idea. It was important in the '90s, during the rough transition to Unicode -- and the most popular way of doing it, chardet, hasn't changed since the '90s.

A heuristic designed before there was multilingual social media, before there were emoji, is not going to work correctly in the 2020s.

When chardet sees the correct UTF-8 encoding of an emoji, it will have no idea what it's looking at, because it won't match anything in its training data. Often, it will guess that it's Turkish encoded in Windows-1254. On other reasonable text, it will guess the "iso-8859-2" encoding, an encoding that you'd very rarely see used intentionally. Because the problem is inherent to the design of chardet, it's not easily fixed.

chardet was built on the assumption that "encoding detection is language detection", which is no longer true. Web sites now contain text in multiple languages, and for the most part they use UTF-8 regardless of the language.

I've strengthened my recommendation from "don't trust chardet's output" to "don't use chardet", because there's no realistic way to use chardet without trusting its output. We've reached a situation where major Python packages such as requests assume that chardet is correct, and yet the changing nature of text means that chardet is more wrong with each passing year.

So how should you interpret raw bytes of text if you're not told what encoding they're in? As UTF-8. Text is UTF-8 until proven otherwise.

ASCII isn't extended

A sign that something is about to go wrong with encodings is if a developer is talking about "extended ASCII".

ASCII is a set of 128 character codes (95 of them displayable). It has not had any new characters added to it since the backslash was added in 1967.

Because ASCII is a 7-bit encoding but our computers use 8-bit bytes, it seems clear that ASCII could be extended to assign a meaning to all 256 possible bytes. There are many different encodings that have done so, and they're all incompatible with one another, which is why treating bytes as characters as a bad idea and why we have Unicode now.

Many developers refer to one of these encodings as "extended ASCII", whose colloquial meaning is "the encoding of 256 characters that I learned first". Its meaning is completely dependent on the country you were in and the operating system you were using when you started programming:

"Extended ASCII" doesn't specify which encoding you mean, and often indicates that you don't realize that different people are thinking of different sets of 256 characters.

Instead of "extended ASCII", say the name of the encoding such as "Latin-1", "Windows-1252", "Windows-1250", "codepage 437", or maybe "I don't know what it is but it looks right on my machine".

And then revise things so that you use UTF-8, which is still a superset of ASCII but can represent every Unicode character.

The "badness" heuristic

The "UTF-8 detector" heuristic

A more narrow heuristic is defined in chardata.py as UTF8_DETECTOR_RE. This heuristic looks for specific sequences of mojibake characters that come from the decoding of common UTF-8 sequences.

Text that matches this regular expression can be partially fixed by ftfy.fixes.decode_inconsistent_utf8(), even when the string as a whole doesn't decode consistently.

Because of this, the expression requires that the match isn't preceded by likely UTF-8 characters -- if this were allowed, then it might pick two or three characters out of a larger mess of mojibake to decode as another character while leaving the rest untouched. This makes the problem more confusing, doesn't really solve anything, and can even pile up recursively to decode as entirely arbitrary characters.

The "lossy UTF-8" heuristic

chardata.py also includes LOSSY_UTF8_RE, which is used similarly to the "UTF-8 detector" heuristic. This regular expression matches sequences that look like they were incorrectly decoded from UTF-8, but with characters replaced by question marks or the Unicode replacement character �.

Characters that match this heuristic will be replaced by � in the ftfy.fixes.replace_lossy_sequences() fixer.

"Sloppy" encodings

Variants of UTF-8