A loss function is the mathematical expression of how bad a model’s prediction is with respect to the desired output. Its value is the single number minimised during training, so its choice de facto defines what the model will learn – anything not captured in the loss is a matter of indifference to the model. For regression tasks the mean squared error is used, which penalises large deviations heavily, or the more robust absolute error. For classification, cross-entropy is used, measuring the mismatch between the predicted and the true distribution; in language models it is computed over the whole vocabulary for every predicted token, and its exponential gives perplexity. Contrastive losses such as triplet loss or InfoNCE, by contrast, work with pairs and triplets of samples and teach the model to pull similar ones together and push dissimilar ones apart, which is the basis of training embeddings. Regularisation terms penalising large weights are often added to the loss. A key property is differentiability, without which the gradient cannot be computed and backpropagation cannot be run.
The loss function is a report card. The model answers something, the loss function grades it – and for the whole of training it strives for one thing only: a better grade. So what you grade matters enormously. If you score a driver purely on arrival time, he will arrive fast and with dented wings. If, in predicting property prices, you penalise the square of the deviation, the model will be excessively afraid of a handful of luxury villas and will slightly overprice ordinary flats because of them. Grading, then, is not a detail tacked on at the end – it is effectively the statement of the whole task.