The learning rate is the hyperparameter that determines how large a step gradient descent takes in the direction of the negative gradient. It is often called the single most important setting in the whole of training, because a bad choice will ruin any architecture. Too small a value leads to unbearably slow convergence and to getting stuck in the first shallow minimum; too large a value makes the optimization overshoot the minimum, start oscillating, and the loss may diverge to infinity altogether. In practice the learning rate is not held constant but follows a schedule. The standard is a warmup phase, during which the value rises linearly from zero – gradients in the first steps are large and unstable and a big step would throw the model off – followed by a gradual decay, most often along a cosine curve, so that the model can settle gently into the minimum it has found. Adaptive optimizers such as Adam additionally derive an effective step for each parameter separately from the history of its gradients. The optimal value is related to the batch size: as the batch grows, the learning rate is usually raised proportionately.
Go back to the person looking for the lowest point of a plain in thick fog. The learning rate is the length of their stride. Shuffling forward a centimetre at a time, they may well be on the right path, but they will reach the valley in three days. Leaping ten metres at a time, they fly over the valley and bounce off the opposite slope – and will keep bouncing like that forever. The sensible approach is to start cautiously until the terrain becomes familiar, then stride out properly, and take shorter and shorter steps the closer the goal gets. That is exactly what the training schedule of every large model looks like today.