CLIP (Contrastive Language-Image Pre-training) is a model introduced in 2021 that trained a shared representation space for images and text. It consists of two encoders – one for images, one for text – which project their input into the same latent space. Training is contrastive and remarkably simple: within a batch of several hundred million image-caption pairs, the model maximises the cosine similarity of matching pairs and minimises the similarity of all incorrect combinations. Crucially, the training signal comes from freely available captions on the internet rather than hand-annotated categories, which enabled an unprecedented scale of data. The result is zero-shot classification: a new task can be defined simply by writing textual descriptions of the classes and picking the one whose embedding is closest to the image – with no training examples and no fine-tuning. CLIP became the backbone of modern multimodality: it serves as the text conditioning module of diffusion models for image generation, as the visual encoder of multimodal language models, and as a tool for semantic search in image archives.
Imagine a bilingual dictionary with sentences on one side and photographs on the other – and both translated into a single shared, artificial language of coordinates. The sentence “a tabby cat on a windowsill” and an actual photograph of such a cat are written almost identically in that language. Once you have this, you can find an image from a description and a description from an image, without ever compiling a list of categories. Best of all, you can come up with a category nobody mentioned before – say “a product with chipped paint” – and the system will recognise it, because it understands the description rather than a list.