Nucleus sampling, commonly called top-p, is a decoding method that narrows the set of candidates for the next token dynamically, according to their cumulative probability. Tokens are ordered from the most probable and taken into the nucleus until their sum crosses a threshold p – 0.9, for example; the rest of the vocabulary is cut away and the probabilities within the nucleus are renormalized. The key advantage over the older top-k method, which takes a fixed number of candidates, is precisely this variable size of the selection. When the model is almost certain about the next word, the nucleus contains one or two tokens and the output stays matter-of-fact; when the situation is open, the nucleus widens to dozens of options and the text does not lose variety. The method emerged as an answer to two well-known problems: temperature sampling across the whole vocabulary occasionally pulls a completely nonsensical token out of the long tail of the distribution, while deterministic decoding produces flat, repetitive text. In practice top-p, top-k and temperature are combined, and the usual advice is to actively tune only one of them – most often temperature – and leave the others at their defaults.
Imagine choosing lunch and the waiter offers you only those items that together make up ninety percent of everything this place sells. In a pizzeria that means three dishes, because almost everybody orders one of three pizzas – so nothing exotic for you. In a bistro with an evenly spread menu, though, the waiter lists fifteen items, because the orders there fan out. Either way you get a sensible choice, and you are never offered the one dish that a single eccentric ordered all year.