LLM temperature and top-p: a worked sampling example

Temperature in an LLM is a generation setting that changes how concentrated the next-token probability distribution is before sampling. Lower positive temperature generally favors the already more likely tokens; higher temperature makes the distribution flatter. Top-p, also called nucleus sampling, restricts sampling to a set of likely tokens whose cumulative probability reaches a chosen threshold.

By Ai Flowi · Published 10 October 2026

Sampling selects the next token

A generative language model scores possible next tokens using the prompt and the tokens already generated. A sampling procedure turns those scores into a selection. The process repeats as the response grows, so a different early token can lead to different later text. These parameters affect selection; they do not supply missing facts or change the model’s learned weights.

A token can be part of a word rather than a complete answer choice. The example below treats four abstract candidates as single tokens to make the arithmetic inspectable. It models one generation step only, not a complete response, classification confidence or the probability that a statement is true.

A four-choice probability table

The probabilities are authored teaching inputs, not measurements from a model. Assume candidates A, B, C and D have probabilities 0.50, 0.30, 0.15 and 0.05 at the stage where nucleus filtering is applied. They are already sorted from largest to smallest and sum to 1.00.

A alone falls short of 0.8. Adding B reaches the threshold, so this example keeps A and B and removes C and D from that step. The retained mass is 0.80. Dividing each retained probability by 0.80 gives 0.50 / 0.80 = 0.625 and 0.30 / 0.80 = 0.375. The filtered probabilities sum to 1.

Either retained token can be sampled. Top-p of 0.8 does not mean that B is chosen, that every response is different, or that the output has a 0.8 probability of being correct. The threshold defines a probability mass used for candidate selection, not a factual reliability score.

The crossing token belongs in the nucleus. With the same toy distribution and a threshold of 0.75, A still falls short and A plus B still forms the smallest sufficient prefix, even though their mass is 0.80. A threshold of 0.90 needs C as well. These are arithmetic examples, not suggested provider settings.

What temperature changes in the same example

In the usual softmax formulation, positive temperature T divides the token scores, called logits, before conversion into probabilities. If the baseline probabilities came from those same scores at T = 1, their rescaled probabilities are proportional to p^(1/T). Normalize by dividing each transformed value by their total.

For a toy T = 0.5, square the baseline values. They become 0.25, 0.09, 0.0225 and 0.0025, with a sum of 0.365. Dividing by 0.365 produces the distribution below. This is a calculation using the assumed inputs, not an observed response to a slider.

Rounding means the displayed column may not sum to exactly 1. Temperature has changed relative likelihood, while the order remains A, B, C, D. A higher positive T would flatten this distribution instead. Neither direction establishes which candidate meets a task’s acceptance rule.

How do temperature and top-p interact?

The worked calculation treats temperature adjustment as happening before nucleus selection. At T = 0.5, A plus B has approximately 0.9315 probability before filtering. A top-p threshold of 0.8 still retains those two candidates, but renormalization now gives A about 0.7353 and B about 0.2647. Keeping the same candidates does not mean keeping the same odds.

Do not assume every API applies sampling controls in this exact sequence or exposes the same options. A top-k control, where available, limits the number of candidates rather than cumulative probability. Constraints, penalties and model-specific behavior can also affect the distribution. Confirm the effective settings for the actual configured model before interpreting output differences.

T = 0 is outside the positive-temperature formula above. APIs may handle zero using a greedy selection mode or another special rule. Even where zero requests the most likely token, it is not a universal promise of identical text across calls, model revisions or serving environments. It also cannot make an unsupported answer true.

Documented controls in AI Flowi Workflow

Consult the Language Model reference for the request interface; the sampling distinction is that temperature is exported as slider, with no named top_p field in that entry.

The Vertex AI catalogue entry separately exposes temperature and top_p as float, and top_k as int, so the worksheet must distinguish which component supplied each requested control.

Language Model and Vertex AI status: documented in the catalogue; not production-tested by Ai Flowi. The export supplies no ranges, defaults or verified sampling order for these fields. Field presence does not establish acceptance by every selected model. This page provides no tested combined workflow or universal slider recommendation.

Use your own provider account and the API key or other credentials required by the selected service. Keep secrets outside the evaluation log. Inspect the actual response or error rather than assuming a saved setting took effect.

A repeated-output evaluation log

Use the same task input, instruction and source evidence for each comparison. Change one supported sampling parameter at a time, and record the unchanged settings too. The following is a blank manual worksheet, not a native logging feature or completed test.

Trial ID:
Model and version, if exposed:
Exact prompt and source-evidence version:
Requested temperature, top-p and top-k where supported:
Effective settings confirmed, or unknown:
Attempt number and captured output:
Provider error or missing output:
Task acceptance check and reviewer decision:
Different wording, different decision, or unchanged:
Reason for accepting or rejecting this setting:

Repeat each candidate setting on the same reviewed inputs, preserving every attempt. For a label task, separate changed wording from changed labels. For a source-based answer, inspect whether claims still match the evidence. Do not discard an error and count only the attractive responses, or infer token probabilities merely from a small collection of complete answers.

Keep instruction design in prompt engineering and action checks in the AI agent evaluation guide; this log measures what changes when the sampling controls change.

Repeated identical answers do not establish that a temperature field was ignored: one likely opening token can dominate several samples, and distinct token paths can produce similar visible text. Conversely, a changed answer does not prove the requested setting took effect if the model version or source context also changed. Mark unconfirmed effective settings as unknown rather than drawing a parameter conclusion from appearance alone.