· Development

Why character counts disagree: bytes, code points and what you see

Why Word, a web form, an SMS and JavaScript's .length give different counts for the same text: UTF-8 bytes, UTF-16 code units, code points and grapheme clusters, and which one to use.

DevelopmentUnicode

A text message holds 160 characters, until you add a single emoji. Then your phone quietly switches the whole message to a different encoding and the limit drops to 70. Paste the same text into a web form and you get yet another number; run .length on it in a browser console and you get a third.

None of them is wrong. They disagree about what “one character” is, and there are four common answers.

Same text, four numbers

Input UTF-8 bytes UTF-16 code units (JS .length) Code points What you see
A 1 1 1 1
é 2 1 1 1
日 3 1 1 1
👍 4 2 1 1
🇺🇸 8 4 2 1
👨‍👩‍👧 18 8 5 1

The family emoji in the last row is one symbol on screen. In Unicode it is five: man, woman and girl, glued together by two zero width joiners (ZWJ, U+200D). A flag is two “regional indicator” letters (U and S here) that fonts draw as one.

The family emoji measured four ways. As a grapheme cluster, what a reader sees, it is 1. As code points it is 5: U+1F468, U+200D, U+1F469, U+200D, U+1F467. As UTF-16 code units it is 8, because each person is a surrogate pair and each joiner is one unit. As UTF-8 it is 18 bytes: 4 per person and 3 per joiner.

Code points are Unicode’s numbers

Unicode gives every character a number: A is U+0041, é is U+00E9, 👍 is U+1F44D. Each number is a code point. To store or send text, those numbers are turned into bytes, and the length depends on how (the encoding).

UTF-16, and why .length says 2 for one emoji

JavaScript strings are UTF-16, and .length counts 16-bit code units, not characters. Anything up to U+FFFF fits in one unit. Anything above it, which includes nearly all emoji, is split into a surrogate pair of two units. So "👍".length is 2. Java’s String.length() and C#’s string.Length count the same way.

This bites the moment a username contains an emoji. name.slice(0, 10) can cut a surrogate pair in half and leave a broken character (�) at the end. Spreading the string counts code points instead: [...name].length. Even that gives 5 for the family emoji.

UTF-8 bytes: what storage and networks see

The web, JSON and most files use UTF-8. ASCII takes 1 byte, accented Latin letters 2, Chinese, Japanese and Korean characters 3, and emoji 4. Bytes matter wherever something has a byte limit: a column, a header, a protocol field.

Databases are the usual trap. In MySQL with utf8mb4, VARCHAR(100) means 100 characters (code points), but rows and indexes have separate byte limits. The older utf8 (really utf8mb3) stores at most 3 bytes per character, so inserting an emoji fails with Incorrect string value. In Oracle, VARCHAR2(100) can mean 100 bytes, depending on the length-semantics setting. Check which one your column counts before you set a form limit.

The SMS at the top is the same story at a lower level. The GSM 7-bit alphabet fits 160 characters into 140 bytes. An emoji isn’t in that alphabet, so the message falls back to UCS-2 at 2 bytes a character: 70 of them.

What a reader calls one character

The unit a person perceives as one character is what Unicode calls an extended grapheme cluster, defined in UAX #29. The family emoji, a flag, a thumbs-up with a skin tone (👍🏽, two code points) and an é typed as e plus a combining accent (U+0301) are each one cluster.

Intl.Segmenter counts them in browsers and Node.js:

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
[...seg.segment("👨‍👩‍👧")].length; // 1
"👨‍👩‍👧".length;                  // 8

If you show users a character limit, this is the number that matches their intuition. Nobody counts one emoji as eight.

Two “café”s that aren’t equal

é can be stored as one code point (U+00E9, the form called NFC) or as e followed by a combining acute accent (NFD). They look identical and are both one grapheme, but "café" === "café" is false, and their lengths differ by one. Files named on macOS often arrive in NFD, which is why a search for a filename sometimes finds nothing. Normalize before comparing or storing: str.normalize("NFC").

Which number to trust

Pick by purpose:

  • A limit you show to people: count grapheme clusters.
  • A limit set by storage or an API: count in that system’s unit (code points for MySQL utf8mb4, UTF-8 bytes where there is a byte cap), and check it again on the server.
  • Keep the two in the same unit. When the form counts one way and the database another, you get text that passes in the browser and is cut off or rejected on save.

To check a draft quickly, paste it into the character counter. It shows characters with and without spaces, UTF-8 bytes and lines, and counts anything that appears as one character, like 👨‍👩‍👧, as one. The text stays in your browser.

References

  • Unicode Standard Annex #29, Unicode Text Segmentation
  • Unicode Standard Annex #15, Unicode Normalization Forms
  • RFC 3629, UTF-8, a transformation format of ISO 10646
  • ECMAScript specification (ECMA-262), The String Type
  • MySQL 8.0 Reference Manual, “The utf8mb4 Character Set” and “String Type Storage Requirements”
  • 3GPP TS 23.038, Alphabets and language-specific information (GSM 7-bit default alphabet)

Back to the blog