Underfitting is a state in which a model cannot capture even the patterns present in the training data, and therefore shows a high error on the training and the validation set at the same time. That simultaneity is what distinguishes it from overfitting, where the training error is low and the validation error high. In terms of the bias-variance tradeoff, this is the high-bias region: the model holds too simple a picture of the data. There are usually three causes. Insufficient capacity, where a linear model is used to describe an obviously non-linear relationship, or the network is too small for the complexity of the task. Unsuitable features, which simply do not contain the information needed for the prediction. And optimization failures – too low a learning rate, too short a training run, excessive regularization, or a vanishing gradient that never lets the signal reach the lower layers. Diagnosis is fairly easy, because both error curves sit high and close together, and the remedy is straightforward: enlarge the model, train for longer, weaken the regularization or add more informative features.
An underfitted model is the student who prepared for a physics exam by learning a single sentence: “Everything falls down.” He is not overfitted – he is not clinging anxiously to specific past questions – he simply cannot answer even the questions he was given. He fails the written test and the oral exam equally. You spot it easily: he cannot manage even the examples you worked through with him. The remedy is therefore the opposite of the one for overfitting. You do not need to rein him in – quite the reverse: give him a stronger toolkit, more time with the books, and stop forbidding him to use formulas.