What temperature changes when a model samples the next token

Temperature is a dial on how the next token is chosen. It does not search the web, and it does not raise the model's accuracy. It changes how peaked the distribution is before a token is drawn.

Scores, then a pick

At each step the model produces a score for every token in its vocabulary. Those scores are turned into probabilities. Sampling draws one token according to those probabilities, appends it, and repeats. Tokens are the units being drawn, which is why a "word" can take several draws.

Temperature scales the scores before they become probabilities. A lower temperature sharpens the distribution. The leading token takes more of the probability, and the rare tokens fade. A higher temperature flattens the distribution. Tokens that were unlikely get a larger share, so the draw varies more.

A temperature of 0 is a special case in many APIs. The implementation skips the draw and takes the highest-scoring token. People call that greedy decoding. It is a code path, and it is not a mathematical promise that two machines will emit the same bytes. Kernels, batching, and a response cache can still split two calls.

Top-p and top-k

Top-k keeps the k highest tokens and drops the rest, then samples inside that list. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities sum to at least p. A top-p of 0.9 throws away the long tail that would otherwise supply the odd word.

Temperature and top-p are usually both on. Temperature reshapes the probabilities. Top-p cuts what remains. If a reply is swinging between a solid extraction and a stray sentence, look at both dials before you rewrite the prompt for the fourth time.

What to set, as a practice

For pulling a field out of a paragraph, or for asking the model to copy a JSON shape, a low temperature keeps the draw on the likely token. You still validate the output. A low temperature will repeat a wrong but common pattern with great consistency.

For a list of alternative headlines, a higher temperature is a way to see other wordings. Read them. The dial did not check whether they are true.

I am not going to name a best number. Providers document different ranges, and a 0.7 on one API is not a calibrated unit you can carry to another. Start from that provider's default, move down when you need the same shape again, and move up when every reply is a paraphrase of the first.

Caches and repeated questions

Ask the same question twice and you may see the same paragraph because a cache returned the first answer. That can look like a low temperature when the second call never reached the model. Ask a question the cache has not stored, or change a parameter the cache key includes, if you are trying to see the sampler.

Temperature also will not repair a missing passage. If the source text was never attached, a low temperature just makes the guess more stable. The fix is the text you attach, which is the subject of retrieval before the model writes.

One prompt, two settings

Pick a paragraph you wrote, and ask the model to list the dates and the amounts in it. Run it at the provider's lowest temperature, then again with the temperature raised and top-p left wherever that API's default sits. Do this on a prompt the cache has not stored, or the second call may be the first reply handed back.

At the low setting the list usually repeats the same nouns and the same order. That is useful when you are checking a parser, because a stable wrong answer is easier to spot than a wandering one. At the higher setting you see other wordings, and you also see items that were not in the paragraph. Those extra items are the cost of the dial. They are not a sign that the model "found" something in the source.

If you need the list to match a schema, keep the temperature low and validate the text after it returns. A schema error at a high temperature is often a rare token that slipped into a key name. Lowering the temperature and checking the output is a smaller change than rewriting the instruction.

Record the model id next to the setting. A temperature value is not a unit you can carry from one provider's API to another's, and a model update can move the default. The note you want later is "this id, this temperature, this top-p, this paragraph," not a single number in a style guide. If a later reply drifts, that note tells you whether the dial moved or the model id did.