A hyperparameter is a setting of a model or of the training process that is not obtained by learning from data but chosen in advance. That is what distinguishes it in principle from parameters – the weights and biases – which gradient descent optimizes. Hyperparameters include architectural choices such as the number of layers, the dimension of the hidden representations or the number of heads in multi-head attention; optimization settings such as the learning rate, the batch size and the number of epochs; and regularization quantities such as the dropout rate or the strength of the weight penalty. Decoding parameters used at inference, temperature or top-p for instance, fall into the same category. Tuning them is always done against a validation set, never against the test set, so that information does not leak and the results are not overstated. The most naive method is grid search, random search tends to be more efficient, and the most sophisticated is Bayesian optimization, which builds a surrogate model of how the outcome depends on the settings. For large models, where a single training run is extremely expensive, hyperparameters are derived from smaller runs and extrapolated.
If you are making soup, the parameters are the things that settle themselves during cooking – how soft the vegetables get, how the flavours come together. The hyperparameters, by contrast, are the decisions you make before you even turn on the stove: how big a pot, how high a flame, how much salt, how long to simmer. The soup will not suggest them to you. You have to choose, cook, taste and change them next time. And the key rule: taste on a practice portion, not on the one you intend to serve – otherwise you will never learn how the soup tastes to someone who does not know what you put in it.