Logistic regression is a statistical method and one of the basic machine learning algorithms, used for classification tasks. Despite its name it therefore does not serve for regression (predicting a continuous value) but for predicting the probability that a given input point belongs to a particular category.
How logistic regression works
At the heart of logistic regression lies the logistic function, also known as the sigmoid function, f(x) = 1/(1+e⁻ˣ). This function has the characteristic “S” shape and converts any input into an output in the range from 0 to 1. That output can be interpreted as a probability.
The process runs in two main steps:
- Computing a score: the model first computes a weighted sum of the input features, much as in linear regression. The result is a single number that may be arbitrarily large or small.
- Applying the logistic function: this number is fed into the logistic function, which “squashes” it into the interval (0, 1). The result is a probability.
For the final decision a threshold is applied, usually 0.5. If the computed probability is higher than 0.5, the data point is assigned to class 1 (“spam”, say). If it is lower, it belongs to class 0 (“not spam”).
Types of logistic regression
According to the number and nature of the categories, three main types are distinguished:
- Binary logistic regression: the commonest type, addressing problems with two possible outcomes (yes/no, true/false). Example: predicting whether a customer will repay a loan or default.
- Multinomial logistic regression: used for problems with more than two categories that are independent of one another and have no natural ordering. Example: classifying a picture of an animal into the categories dog, cat or bird.
- Ordinal logistic regression: used in situations where there are more than two categories but they have a meaningful order or ranking. Example: rating customer satisfaction on a scale of low, medium, high.
Advantages and disadvantages
Despite the existence of more advanced methods, logistic regression remains very popular, for several reasons.
Advantages
- Simplicity and speed: it is computationally very efficient, fast to train and undemanding of resources.
- Interpretability: the model’s results are easy to understand. You can see exactly what influence (weight) each input feature has on the final result, which is crucial in fields such as medicine or finance.
- Good as a baseline model: thanks to its simplicity it serves as an excellent starting point for testing and comparison with more complex models.
Disadvantages
- Assumption of linearity: the model assumes a linear relationship between the input features and the outcome. If the relationship is complex and non-linear, the model may perform poorly.
- Sensitivity to outliers: extreme values in the data can significantly affect the results.
- Limited performance: on complex problems such as image recognition it usually does not reach the accuracy of more advanced models such as neural networks.