The UTF-8 BOM (EF BB BF): why three invisible bytes break some parsers and not others
Why EF BB BF at the start of a UTF-8 file is skipped by browsers but rejected by JSON.parse, following the rules in Unicode, RFC 3629, RFC 8259 (JSON) and the WHATWG Encoding and HTML standards.
Put the three bytes EF BB BF at the start of a UTF-8 file and some programs read it without complaint, while some parsers stop at the first character. Those bytes are a BOM, or byte order mark.
This usually is not a bug on either side. Different standards give the BOM different jobs. The post on JSON errors covered it briefly; this article looks at the rules standard by standard.
A BOM is one character: U+FEFF
A BOM is the character U+FEFF placed at the start of data. Its byte representation depends on the encoding.
| Encoding | BOM bytes |
|---|---|
| UTF-8 | EF BB BF |
| UTF-16 big-endian | FE FF |
| UTF-16 little-endian | FF FE |
| UTF-32 big-endian | 00 00 FE FF |
| UTF-32 little-endian | FF FE 00 00 |
In UTF-16 and UTF-32, the BOM indicates byte order. UTF-8 has only one byte order, so its BOM can only act as a signature meaning “this is UTF-8.” The Unicode Standard 18.0 says a BOM is not required for UTF-8 and neither recommends nor discourages its use.
Away from the start, it is content
U+FEFF is named ZERO WIDTH NO-BREAK SPACE. It was historically used to prevent a line break between words. Unicode now recommends U+2060 WORD JOINER for that purpose because it cannot be confused with a BOM.
A U+FEFF away from the beginning is content, not a signature. RFC 3629 says it MUST NOT be interpreted as a signature anywhere except the beginning.
RFC 3629 separates the rules for a leading U+FEFF:
- Protocol elements required to be UTF-8 SHOULD forbid the signature.
- Elements with an encoding-identification mechanism that implementations are expected to always use properly SHOULD also forbid it.
- Elements with no such mechanism SHOULD NOT forbid it when enforcing a ban would be impractical.
- Elements with a mechanism that implementations cannot be expected to always use properly SHOULD NOT forbid it either.
- Where the signature is forbidden, a leading U+FEFF MUST be interpreted as a zero-width no-break space.
The RFC says larger elements such as MIME entities can fall into either of the two SHOULD NOT cases, depending on their source and how reliably they are labeled.
JSON: do not add one; ignoring one is optional
RFC 8259 §8.1 says JSON exchanged outside a closed ecosystem MUST use UTF-8. Implementations MUST NOT add a BOM to JSON sent over a network. Parsers MAY ignore one for interoperability.
That permission is optional. JSON whitespace consists only of space, tab, LF and CR, so a parser that does not skip the BOM sees U+FEFF as an invalid first character.
JSON.parse('\uFEFF{"a":1}')
// SyntaxErrorThis behavior was checked with Node.js v24.13.0 on the draft date.
According to Python’s documentation, the json module does not add a BOM when serializing and raises ValueError when a string to deserialize starts with one. When decoding bytes, 'utf-8' keeps a leading U+FEFF as a character, while 'utf-8-sig' skips the first three bytes.
Browser fetch() follows another path. Under the Fetch standard, both Response.text() and Response.json() perform UTF-8 decoding that removes a leading BOM. A file can therefore load through fetch but fail if another reader preserves the BOM and passes the resulting string to JSON.parse.
On the web, browsers check the BOM first
The WHATWG Encoding decode algorithm checks for EF BB BF, FE FF or FF FE before processing the content. If it finds one, the corresponding encoding wins and the BOM is removed.
HTML encoding sniffing also checks the BOM first. A match gives the encoding certain confidence. The charset from HTTP Content-Type and any <meta> declaration are considered later.
There are two separate kinds of requirements here. HTML requires conforming documents to use UTF-8. Browser processing rules still define how to decode legacy encodings such as Shift_JIS, EUC-KR and windows-1252.
If a document has no BOM, no encoding in Content-Type, and is not an iframe srcdoc document, the author must declare the encoding with <meta charset> or the equivalent http-equiv declaration. The declaration must occur within the first 1024 bytes.
TextDecoder has a counterintuitive option name. With the default ignoreBOM: false, the decoder recognizes and removes the BOM. Setting it to true preserves U+FEFF in the output.
Summary
| Source | Treatment of a BOM |
|---|---|
| Unicode Standard 18.0 | Not required for UTF-8; neither recommended nor discouraged |
| RFC 3629 | Should be forbidden in always-UTF-8 elements and some reliably labeled elements |
| RFC 8259 | Must not be emitted in networked JSON; parsers may ignore it |
| WHATWG Encoding and HTML processing | Checked before labels and removed |
| HTML authoring rules | Conforming documents must use UTF-8 |
JSON.parse, Python json |
A leading U+FEFF in a string is an error |
For networked JSON and other formats that already require or declare UTF-8, leave the BOM off unless the format or receiving tool explicitly expects one. If incoming data already has a BOM, handle it while decoding bytes rather than cleaning up the resulting string later.
When a parser fails at the first character for no visible reason, inspect the bytes:
od -An -tx1 -N3 fileAn output beginning with ef bb bf identifies a UTF-8 BOM.