Scaling laws are empirically established relationships describing how the performance of a language model improves as the number of parameters, the volume of training data and the compute expended all grow. The key observation is that the error falls according to a power law – each multiplication of the inputs reduces the loss by a predictable proportion – and that this relationship holds across many orders of magnitude. For the first time, this made it possible to plan the training of giant models in advance: from a few small runs you can extrapolate what ten times the resources will buy. A fundamental correction arrived in 2022 with the Chinchilla research, which showed that the large models of the day were heavily under-trained relative to their data; for a given compute budget it is optimal to grow parameters and data in roughly the same proportion, which settled at something like twenty tokens per parameter. The practical consequence is a shift towards smaller models trained for much longer, which are cheaper to run at inference. A side finding, and a theoretically uncomfortable one, is that the smooth decline in loss shows up in specific capabilities as a series of jumps – that is emergent behaviour.
It is like a recipe where you know that doubling the pot and the ingredients makes the dish a quarter better – and that it holds whether you are cooking for ten people or ten thousand. Extremely useful, because you can work out in advance whether that extra quarter is worth twice the price. Chinchilla added an important culinary rule to it: there is no point buying an ever bigger pot if you keep putting the same small amount of ingredients in it. Most of the first giant models were exactly that – enormous pots with thin soup.