Explainable AI (XAI) is the field concerned with making model decisions comprehensible to humans. It responds to the black-box nature of deep models, where the reason for a particular prediction cannot be read directly from millions or billions of weights. A distinction is drawn between interpretability, the property of models transparent by construction – logistic regression or a decision tree – and explainability, the after-the-fact reconstruction of the behaviour of an already finished opaque model. Among the most used post hoc methods are LIME, which fits a local linear surrogate in the neighbourhood of a specific input, and SHAP, which apportions the contribution of individual features according to Shapley values from cooperative game theory. For images, saliency maps are used, and for language models, mechanistic interpretability, which looks for specific circuits and uses sparse autoencoders to disentangle individual features in the activations. The motivation is not merely scientific: regulations including the European AI Act and the GDPR establish a right to explanation in sensitive areas. A fundamental caveat is that an explanation tends to be plausible without necessarily being faithful – even reasoning a model states out loud may be an after-the-fact rationalisation.
Imagine an experienced doctor who looks at a patient and says “this is pneumonia” – and is right ninety-nine per cent of the time. But ask him why and all he says is “I can see it”. For the patient and for the insurer that is unacceptable. Explainable AI looks for ways to get out of such an expert: “it was the crackling in the left lobe and a fever above 39 degrees that decided it; without the crackling I would have said something else.” But mind one trap: an explanation invented after the fact can sound convincing and still not describe what actually went on in the head.