Tokens and words are different counts

The word counter on this site counts words. A language model counts tokens. The two numbers meet in English prose and then walk apart, and the apart cases are the ones that hit a limit.

What a token is

A tokenizer splits text into pieces from a fixed vocabulary built for that model. A common word may be one piece. A less common word may be several pieces stuck together. Spaces, punctuation, and line breaks are pieces too. The model never sees "a word." It sees the ids of those pieces, and it predicts the next id. That is the unit inside a context window, and it is the unit on most API bills.

Because the vocabulary belongs to the model, an unchanged paragraph can be 180 tokens for one model and a different number for the next. A calculator that says "words times 1.3" is guessing at one family's English average. It is the wrong tool the day you switch models.

Where the counts diverge

Ordinary English sentences are often in the neighborhood of one token per short word. I will not turn that into a conversion rate. It is a hint that a 500-word note and a 500-token note are similar in size for some English models, and then you should stop using the hint.

Code tokenizes less kindly. Identifiers, brackets, and indentation are pieces. A minified line and a pretty-printed line are different token counts of the same program. Emoji and text in scripts that were rare in the tokenizer's training data often take more pieces per word than English does. A bilingual paragraph can surprise a budget that was planned on English samples.

A PDF extract is a frequent offender. Line breaks in the middle of sentences, hyphenation at the margin, and a header repeated on every page all become tokens. Cleaning the extract to normal paragraphs reduces the count and, more usefully, makes the text easier for the model to read.

Which count you wanted

If the box says "150 characters" or "60 words," use a character counter or a word counter. The tokenizer is irrelevant to a CMS field.

If the error says the request exceeded the context, or the invoice line is priced per million tokens, use the tokenizer for that model id. Count the whole request you send, including the system instruction and the passages you attached, not only the sentence in the text box.

Sampling draws one token at a time, which is why temperature feels like it changes wording one fragment at a time. A limit of 200 tokens on the reply can cut a sentence off in the middle of a word piece. Raise the reply limit, or ask for a shorter answer. Do not assume the model finished the thought.

How to count the request you will actually send

Open the tokenizer the provider publishes for that model id. Paste the system instruction, then a typical history, then one real document chunk, then the question. The number on that paste is the number that has to fit in the window. The sentence in the text box is the small part.

Do the same paste with the line breaks left in, and again after you join the lines into paragraphs. A PDF extract that broke every line will usually cost more tokens than the cleaned paragraph, and it is harder to read. Cleaning is worth it for both reasons. I am not printing a token count for a sample here, because the count belongs to the tokenizer you are about to call, and a number from a different model would be a false budget.

If the bill is per million tokens, the same paste is the estimate. Multiply by the calls you expect, and include the reply, which is also tokens. A retry that sends the whole history again spends the budget again. A cache hit may not. The invoice is the check, and the tokenizer is the preview.

A word count still belongs on the page when a person set the limit in words. A meta title, a classified ad, and a text field are word or character limits. Run those through the word counter. Running them through a tokenizer answers a question the field did not ask.

Two limits on one page is a common mix-up. The CMS field says 60 characters. The model error says the context was exceeded. Fix the field with a character count. Fix the model call by shortening the attached history or the passage, then recount with that model's tokenizer. Treating one number as if it answered both questions is how a title gets padded and a prompt stays over the limit.