IoU (Intersection over Union, also the Jaccard index) is a metric of overlap between two regions, defined as the size of their intersection divided by the size of their union. It ranges from zero for complete disagreement to one for perfect overlap. In computer vision it is the basic evaluation tool for object detection, where it compares a predicted bounding box with the reference annotation, and for segmentation, where it is computed over pixel masks. Its significance lies in replacing the binary verdict “right or wrong” with a continuous measure of localisation quality, where only the chosen threshold – traditionally 0.5 – decides what counts as a found object. Precisely for that reason today’s main detection metric, mean average precision, is computed as an average over a whole range of IoU thresholds from 0.5 to 0.95, so as to reward the accuracy of the outline too, not merely the finding. IoU is also used in the non-maximum suppression algorithm, where it serves to identify duplicate predictions of the same object. In a modified, differentiable form as the Dice or GIoU loss it enters the loss function directly.
Imagine two surveyors independently marking the boundary of the same plot on a map. How well did they agree? Saying “yes” or “no” is not enough. The sensible thing is to take the area both of them agreed on and divide it by the total area marked by at least one of them. If they drew exactly the same thing, you get one. If they do not overlap at all, you get zero. And if one drew a small square inside the other’s large rectangle, you get a low number – exactly as it should be, because in reality they disagreed substantially, even though their boundaries touch.