Cross-validation is a method for estimating a model’s true performance that makes frugal use of a limited amount of data. The most widespread variant is k-fold cross-validation: the dataset is split into k roughly equal parts and the process is repeated k times, with one part serving as the test set and the remaining k−1 as the training set in each round. The results are averaged, and their variance additionally tells you how stable the model is. The motivation is the unreliability of a single data split: with a small sample, one randomly chosen test set may turn out unluckily easy or unluckily hard, making the quality estimate misleading. The composition of the folds deserves particular attention – with imbalanced classification, stratification is used to keep the class ratio identical in every fold, whereas with time series the order must not be shuffled and the future must never appear in the training fold. The method is indispensable for fair hyperparameter tuning, where it is often nested into two levels. The price is k times the computational cost, which makes it impractically expensive for large deep models; there a single fixed validation set has to do.
Imagine you have to assess five job candidates and you only have five test assignments. If you gave each candidate a different one, the results would not be comparable – one of them may have drawn the easier task. Cross-validation handles it differently: you run five rounds. In each round four assignments are used for preparation and one for the real examination, and the next round examines on a different one. At the end you take the average. You learn not only how good a candidate is, but how consistent – because if they aced one round and bombed another, that is information every bit as valuable as the average itself.