Few-shot learning is the approach in which a model is given several example pairs of task and correct answer in the prompt and infers the desired behaviour for a new input from them. With large language models one speaks of in-context learning, because – the name notwithstanding – not a single weight is updated. The model merely recognises the pattern in the presented context and continues it, which is a form of meta-learning acquired during pre-training. The capability was systematically described with the GPT-3 model and is often cited as an example of emergent behaviour, since it is practically absent in small models. In practice it is enormously valuable wherever a zero-shot instruction fails: the examples anchor the output format exactly, unify terminology, and delimit edge cases better than any verbal description. The quality and order of the examples matter more than intuition suggests, and an unbalanced representation of classes can systematically skew the result. The price is consumption of the context window, and with it higher latency and cost for every call.
It is like not explaining the rules to a new colleague but putting three finished examples in front of him: “This complaint is handled like this, this one like this, and this one like this. Now do the fourth.” Most people grasp it immediately and better than if you read them the policy document. A model works the same way, and as with a person, the examples matter: give it three cases where the answer was always “reject” and it will start rejecting everything. And examples are not free – they take up room in its working memory and you pay for them again with every query.