Multimodality is a model’s ability to process or generate several types of data at once – text, image, audio, video or structured signals. The technical basis is a shared representation space: each modality has its own encoder that turns the input into a sequence of tokens or embeddings projected into the same latent space, where the transformer then handles them uniformly. An image is typically cut into a grid of patches, which a linear projection turns into visual tokens; audio is converted into a spectrogram and that into windows. The links between modalities are learned from paired data – images with captions, videos with subtitles – and contrastive training plays the key role, pulling the representation of a picture and of its description towards each other, as the CLIP model established. Cross-attention is used to feed one modality into another. The practical consequence is that boundaries between tasks dissolve: the same model answers questions about a photograph, reads tables off a screenshot or operates a user interface. In security terms it brings new attack surfaces, because instructions can be hidden inside an image.
People are multimodal as a matter of course. Asked “what is wrong with this picture?”, you effortlessly connect what you see with what you hear and answer in words. You do not keep a separate department for images and another for words in your head; you have one understanding of the world that both feed into. Models lacked that privilege for a long time – one system handled text, another handled pictures, and between them there was silence. Multimodality means a photograph is cut into pieces that behave inside the model like words, and from there seeing and reading are one and the same substance.