A decision tree is a model that derives a prediction through a sequence of conditional questions about feature values. At internal nodes the data are split by a single threshold or category, and the leaves hold the resulting class in classification or a number in regression. Training proceeds greedily and recursively: at each node the possible splits are searched and the one that most reduces the impurity of the resulting subsets is chosen – measured by the Gini coefficient or entropy in classification and by variance in regression. The growing tree must be constrained, because left alone it will branch down to one sample per leaf and overfit perfectly; maximum depth, a minimum number of samples per node, or subsequent pruning serve that purpose. Its strengths are easy interpretability, the ability to work with numerical and categorical features without normalisation, and automatic capture of non-linearities and interactions. Its weaknesses are high instability – a small change in the data can rearrange the tree – and an angular decision boundary made of rectangular blocks. Precisely for that reason trees are deployed almost exclusively as part of ensembles such as random forests or gradient boosting.
It is exactly the questionnaire pinned to a surgery door: “Do you have a temperature above 38 degrees? Yes → Are you coughing? Yes → Stay at home.” Each question splits people into two groups, and after a few questions you know where the patient belongs. The charm is that you can explain the decision at any time – you simply read out the path you took. The drawback shows up when you add three new patients to the sample and the whole questionnaire suddenly rearranges itself. Which is why practice relies not on one questionnaire but on the opinion of many differently assembled questionnaires at once.