The Vision Transformer (ViT) is an architecture that applies the transformer to image data without any convolution. The picture is divided into a grid of non-overlapping patches, for example sixteen by sixteen pixels; each patch is linearly projected into a vector, and the resulting sequence – augmented with positional encoding and a special classification token – is processed as if it were a sentence. The key difference from convolutional networks is a global receptive field already in the first layer: the attention mechanism can relate two opposite corners of the image instantly, whereas convolution needs many layers to do so. The price is the absence of built-in assumptions about locality and translation invariance, which are what let convolution learn from small datasets. ViT therefore loses to convolutional networks on medium-sized data, but surpasses them when pre-trained on hundreds of millions of images – a textbook manifestation of the scaling laws. The quadratic complexity of attention in the number of patches led to hierarchical variants such as the Swin Transformer. ViT is today the standard image encoder of multimodal models.
A convolutional network is like a person examining a picture through a magnifying glass, gradually assembling an overall impression from small details – first edges, then eyes, finally a face. A Vision Transformer is like a person who cuts the picture into a hundred cards, spreads them out on the table and compares every card with every other from the very start: “this card with an ear probably relates to that distant card with the other ear.” It is a very powerful method, but it comes with one condition: such a person must have seen an enormous number of pictures beforehand, because unlike the one with the magnifying glass, nobody told him that things which belong together tend to be near each other.