Temperature is a decoding hyperparameter that controls how much randomness goes into text generation. At the end of every step, a language model produces a logit – an unnormalized score – for each token in the vocabulary. Before those logits are turned into probabilities by the softmax function, they are divided by the temperature T. Values of T below 1 sharpen the distribution, because the leading candidate’s relative advantage grows; values above 1 flatten it and give even unlikely tokens a chance. As T approaches zero, sampling turns into deterministic greedy decoding, where the most probable continuation is always taken. Temperature changes nothing about what the model has learned – only the way choices are drawn from the learned distribution. In practice it is combined with other ways of restricting the candidates, above all top-k and nucleus sampling (top-p). A high temperature increases the diversity of the output, but also the risk of hallucination and of drifting off the point, because autoregressive generation locks every misstep into the context for all the steps that follow.
Think of temperature as a nerve dial on your narrator. Turn it all the way down and you get a bureaucrat: ask what was for lunch and the answer comes back identical every time, curt and safe. In the middle you get a colleague who tells the story naturally and now and then reaches for an unexpected word. Turn it up and you get a daydreamer on his third coffee – colourful and interesting, until he starts inventing things. For pulling data out of an invoice you want the bureaucrat. For brainstorming slogans you want the daydreamer. The important part is that the dial raises neither intelligence nor knowledge – only the willingness to reach for the less obvious option.