Perplexity is the most widespread quality metric for language models, defined as the exponential of the average cross-entropy per token. It is interpreted as the weighted average number of options the model is wavering between at each step: a perplexity of ten means the model is as uncertain in choosing the next token as if it were picking uniformly from ten candidates. Lower is therefore better, and the lower bound is one, meaning perfect prediction. Because it is a direct function of the loss used in pre-training, perplexity is tracked during training as the main indicator of progress and is the basic quantity in the scaling laws. It has, however, two fundamental limitations. Values are not comparable between models with different tokenisation, nor between different corpora, so an absolute number means nothing without context. And above all, low perplexity does not mean usefulness: a model can be excellent at predicting text and still be disobedient, untruthful or dangerous. That is why it is today complemented by task benchmarks and human evaluation.
Perplexity is a measure of surprise. Imagine guessing the next word in someone else’s sentence. If your perplexity is two, you are always deciding between two words – you understand the text very well. If it is a thousand, you are groping as if in a language you have never heard. So the number says how much the text can still catch you out. But mind what it does not say: a person who perfectly predicts what the newspaper will print need not be a good adviser. Predicting and being useful are two different abilities.