What is a token in ChatGPT?

A chunk of text the model reads as one unit: a common word with the space before it, a piece of a rarer word, a punctuation mark. OpenAI's rule of thumb for English is that a token is about 4 characters, or three-quarters of a word, so 100 tokens is about 75 words.

How many tokens is a word?

About 1.3 on average in English, by OpenAI's three-quarters-of-a-word rule. Common words are usually one token with their leading space; rare words, names and typos split into several. With the small vocabulary on the stage, 31 characters make 13 tokens.

Is a token the same in every model?

No. Each model family has its own tokenizer. GPT-2's had 50,257 tokens, GPT-4's has about 100,000 and GPT-4o's about 200,000, and the same word can split differently: “strawberry” on its own is str, aw, berry for GPT-4 and st, raw, berry for GPT-4o.

Why do tokenizers use byte-pair encoding?

It gives a fixed-size vocabulary that still covers any text. Common words become single tokens, and anything rare falls back to smaller pieces, down to single bytes, so no word is ever unknown. Sennrich and colleagues brought it from data compression to machine translation in 2016 to handle rare words.

Live model · ChatGPT tokens

How ChatGPT's Tokens Actually Work

ChatGPT never sees letters. Before your message reaches the model, a tokenizer cuts it into tokens, chunks of text learned by byte-pair encoding, and swaps each one for a number. In GPT-4o's tokenizer, “ strawberry” with the space before it is a single token, number 101830. Counting its r's means knowing what that number spells.

Tokenizing a question

The text is cut into 8 chunks at spaces and punctuation, as GPT-2's tokenizer does, and each of its 31 characters starts as its own token, numbered by its byte: “H” is 72.

StopByte-pair encoding

Text

Characters31Tokens31Merges used0Vocabulary1,256

The tokens

Character by character

721111193210997110121321143911532971141013210511032115116114971199810111411412163

What the model receives

Merges used, latest first

Learned	Pair	Token
(none)

CH1 Just merged

How to read the stage

This tokenizer learned 1,000 merges from the first three chapters of Alice in Wonderland. Each character of “How many r's are in strawberry?” starts as a token numbered by its byte, and merges apply in the order learned, the book's commonest pairs first. After 18 merges, 31 characters are 13 tokens. The book never has “strawberry”, so it ends up as six tokens: “ st” with its space, then ra, w, b, er and ry.

Why the r's are hard to count

A token's number says nothing about the letters inside it. To count r's, a model has to have learned what each of its tokens spells, then add across them. In GPT-4o's tokenizer, “strawberry” on its own is st, raw and berry: tokens 302, 1618 and 19772. With the space before it, as in a sentence, it's one token. Reversing a word or naming its third letter runs into the same wall.

Where the merges come from

Training counts every pair of neighbouring tokens and merges the most common into a new one, again and again. In the book's first sentence, “i” + “n” appears 5 times, so it's merge 1, and “in” + “g” appears 4 times, so “ing” is merge 2. Nothing told it about suffixes. Real tokenizers run the same loop on bytes, over far more text, until the vocabulary reaches its size.

Learning merges from one sentence

103 characters in 22 chunks, each character a token numbered by its byte. Training counts every pair of neighbouring tokens inside a chunk, then merges the most common, 12 times.

StopTraining BPE

Training textMerges

#	Pair	New token	Seen
(none)

CH1 New token

Watch the Python behind this model run, line by line

What the model leaves out

This vocabulary has 1,256 tokens, learned from about 31,000 characters of one book; GPT-4's has about 100,000 and GPT-4o's about 200,000, so common words are one token each. Real tokenizers work on bytes, so every language and emoji fits, and GPT-4's chunking rules differ in details from GPT-2's, used here. This one takes plain ASCII. The algorithm is the same: count pairs, merge the most common, apply merges in order.

Questions

What is a token in ChatGPT?: A chunk of text the model reads as one unit: a common word with the space before it, a piece of a rarer word, a punctuation mark. OpenAI's rule of thumb for English is that a token is about 4 characters, or three-quarters of a word, so 100 tokens is about 75 words.
How many tokens is a word?: About 1.3 on average in English, by OpenAI's three-quarters-of-a-word rule. Common words are usually one token with their leading space; rare words, names and typos split into several. With the small vocabulary on the stage, 31 characters make 13 tokens.
Is a token the same in every model?: No. Each model family has its own tokenizer. GPT-2's had 50,257 tokens, GPT-4's has about 100,000 and GPT-4o's about 200,000, and the same word can split differently: “strawberry” on its own is str, aw, berry for GPT-4 and st, raw, berry for GPT-4o.
Why do tokenizers use byte-pair encoding?: It gives a fixed-size vocabulary that still covers any text. Common words become single tokens, and anything rare falls back to smaller pieces, down to single bytes, so no word is ever unknown. Sennrich and colleagues brought it from data compression to machine translation in 2016 to handle rare words.

Drill the mechanism

Each training step is a counter: a hash map from every adjacent pair to how often it occurs, then the most frequent one. That's Top K Frequent Elements with k = 1, run once per merge.