Gradient descent is the fundamental optimisation algorithm of machine learning. It searches for the setting of model parameters that minimises the loss function, proceeding iteratively: at the current point it computes the gradient – the vector of partial derivatives of the loss with respect to all parameters, pointing in the direction of steepest increase – and moves the parameters a small step in the opposite direction. The step length is set by the learning rate. Computing the gradient over the whole dataset is unthinkable with large corpora, so in practice the stochastic variant is used, estimating the gradient from a random mini-batch of samples. The resulting noise makes the trajectory jitter, but it also helps escape flat and poor regions. Higher-level optimisers add momentum, which damps oscillation across narrow ravines, and adaptive rescaling of the step for individual parameters, as Adam does. In deep networks the loss surface is not convex, so there is no guarantee of finding the global minimum; empirically, though, in a very high-dimensional space most of the minima reached lead to similarly good results. Gradients for all layers are supplied by backpropagation.
Imagine standing in thick fog on a hilly plain, wanting to reach the lowest point. You cannot see a step ahead, but with your feet you can feel which way the ground slopes most steeply. You take a step that way and feel around again. The step must not be too small, or you will still be there at nightfall, nor too large, or you will overshoot the valley and start bouncing from slope to slope. And since feeling out the whole plain is impossible, you rely on probing just around your feet – so you wander a bit, but that has its upside too: you will not get stuck in the first shallow puddle you meet.