A batch is a group of training samples pushed through the model together, from which a single update of the parameters is computed. The batch size is one of the key hyperparameters and represents a trade-off between the quality of the gradient estimate and computational efficiency. A small batch gives a noisy but often beneficial gradient – the random scatter helps escape poor minima and acts as a regularizer – while making worse use of the accelerator’s parallel capacity. A large batch gives a smoother, more faithful estimate, permits a higher learning rate and keeps the hardware far busier, but enlarging it excessively harms generalization and beyond a certain point stops delivering any speed-up at all. When the desired batch will not fit in memory, gradient accumulation is used: the gradient is computed from several smaller portions in turn and the parameters are updated only once they have been summed. In the training of large language models the batch is measured not in samples but in tokens, and routinely runs into the millions, because that is the only way to keep thousands of accelerators working in parallel sensibly occupied.
Imagine you want to find out how the residents of a city vote. Asking each person individually and revising your estimate after every answer is exhausting, and the estimate lurches wildly depending on who you happen to run into. Counting all five hundred thousand people first and only then adjusting your estimate is absurdly slow – you would get an answer once a year. A sensible pollster asks a thousand people, forms a picture from them, adjusts the estimate and moves on to the next thousand. That thousand is the batch: big enough not to be a fluke, small enough to handle right away.