Gradient boosting is an ensemble method that builds a strong model sequentially from many weak ones – almost always from short decision trees. It differs fundamentally from bagging: bagging trains models in parallel and independently, whereas boosting chains them one after another, with each new model learning to correct the errors of the sum so far. Formally it is gradient descent in function space – a new tree is fitted to the negative gradient of the loss function with respect to the current prediction, which for squared loss means directly to the residuals. Each tree’s contribution is multiplied by a small learning rate, which is the main regularisation mechanism: many tiny corrections generalise better than a few large ones. Because the method reduces bias above all, it can – unlike a random forest – overfit, and therefore requires careful tuning of the number of trees with early stopping. The XGBoost, LightGBM and CatBoost implementations add regularisation, handling of missing values and of categorical features, and rank among the strongest tools on tabular data, where they routinely beat neural networks.
The difference from a voting committee of experts is that here the work happens in a queue, not all at once. The first expert writes a rough estimate. The second does not tackle the whole problem – he is handed only a list of what the first got wrong and writes a small correction. The third fixes what the first two left. After five hundred such minor corrections the result is very accurate. The cardinal rule is that each fixer may add only a small fraction of his correction: if he corrected at full strength, he would start chasing random noise in the data too, and the whole chain would overfit.