Optical flow is a vector field describing the apparent motion of individual image points between two consecutive video frames. For every pixel it states where it moved in the following frame, turning a sequence of static images into a description of the scene’s dynamics. Classical methods start from the brightness constancy assumption – that a point keeps its intensity as it moves – and from the assumption of local motion smoothness; the first yields the basic optical flow equation, which however has one constraint and two unknowns per pixel, so regularisation or a local window must be added, as the Lucas-Kanade and Horn-Schunck algorithms do. A notorious consequence is the aperture problem: motion along an edge cannot be determined through a small window. Further difficulties are large displacements, addressed by a pyramidal hierarchy of resolutions, lighting changes, repetitive textures and occlusions, where no correspondence exists. Today’s approaches are learned, with architectures such as FlowNet and RAFT. Optical flow underpins motion compensation in video codecs, image stabilisation, object tracking, action recognition, and visual odometry for robots and drones.
Close one eye and move your head slowly sideways. Nearby objects sweep quickly across your view, while a distant hill barely shifts – and it is precisely from these differing speeds that your brain estimates what is near and what is far. Optical flow is that information written down as an arrow for every point in the image. It also illustrates its famous weakness nicely: look at a long vertical pole through a paper tube and let the pole slide straight upwards, and you will see nothing at all. Without knowing the surroundings, motion along an edge simply cannot be detected.