If you have taken an interest in artificial intelligence in recent years, you have probably come across large language models such as ChatGPT, Claude or Gemini. These LLM (Large Language Models) have become synonymous with AI in the eyes of the public, but the reality is far more varied and fascinating. The world of artificial intelligence offers a wide family of specialised models and architectures, each of which solves specific tasks in its own way.
Today we will not be talking about classic machine learning, which has been with us for decades and which in fact today works with perhaps a greater volume of data and in far more areas of our lives than LLMs. Instead, in today’s article we will focus on some of the types of neural models and architectures that have only come into being in recent years and that represent the cutting edge of current research.
From multimodal models processing text and images at the same time, through neural networks analysing complex relationships, all the way to audio models recognising speech and generating music. From miniature models running comfortably on your phone to giant systems with trillions of parameters. From purely neural approaches to hybrid systems combining symbolic reasoning with deep learning.
Why are LLMs alone not enough?
Imagine using only a hammer for every task. It might work, but a screwdriver would certainly be more suitable for screws. It is similar with AI models. While LLMs handle work with text, generating answers or code and general conversation brilliantly, there are tasks that require a completely different approach.
Traditional decoder-only LLMs work on the principle of the transformer architecture with unidirectional attention. It works like this: the model reads the input text from left to right and, using the attention mechanism , it can look only at the preceding tokens (not at future ones), which is called causal masking. Thanks to this it can efficiently generate coherent text token by token. There are, however, also encoder-decoder LLMs, which use bidirectional attention in their encoder part. That means it can see the whole input text at once. This comes in useful for example in translation, where it is often necessary to understand the context of a whole sentence or paragraph before you start translating. For example, a German verb often stands at the end of the sentence, but for a correct translation into Czech you need to know that verb right at the beginning. Or take idioms and set phrases – their meaning can change according to what follows later in the text.
In an LLM, text is first split into tokens, which are converted into vectors by embedding layers. Imagine it as cutting a sentence into individual pieces – sometimes these are whole words, sometimes only parts of words or even individual letters, depending on what is most useful for the model. Each such piece then gets its numeric “business card” – a vector containing information about what this piece means and how it is usually used.
The attention mechanism then allows the model to focus on the relevant parts of the input when generating each new token. It works rather like when you are reading a book and need to remember something that was written earlier – the model “looks through” all the preceding words and decides which of them are important for what it wants to write next. For instance, when it writes the word “she”, it has to remember who that refers to from the beginning of the sentence.
The model actually has several such attention “eyes” at the same time (multi-head attention), so it can follow different things at once – one “eye” can follow grammar, another meaning, a third long-range connections in the text. All this information is then processed through complex mathematical calculations that help the model decide which word to write next.
This approach is extremely effective for sequential text generation, but it has its limitations. The model has a limited “memory” (the context window), and therefore cannot remember an infinitely long text. It also has to generate words (tokens) one after another, which is slow, and in some types of models it cannot look ahead, which can be a problem for example when you need to generate text of an exact length, where it is useful to know the whole sentence in advance.
Multimodal models: when text is not enough
One of the biggest breakthroughs of recent years are VLMs (Vision-Language Models), models that can process images and text at the same time. These models can process images and text simultaneously, which opens up completely new possibilities. Instead of having to describe the image in words first, the model can analyse the photograph directly and answer questions about its content.
In practice this means that you can upload a photo of a meal and ask for a recipe, show the model a diagram and have it explained to you, or send a screenshot with an error message and ask for help. VLMs connect visual perception with language understanding in a way that comes close to human reasoning.
Technically, VLMs combine two different encoders (an encoder is the part of a neural network that converts input data into an internal representation). The visual encoder, often based on the Vision Transformer (ViT) architecture, converts the image into a sequence of partial tokens (small parts of the image converted into a numeric representation). The text encoder processes the language input in a similar way to a classic LLM. The key thing is connecting these modalities (different types of data – text, image) by means of cross-attention (a technique that allows the model to track relationships between different types of input) or a multimodal connector block (a component that joins information from different sources).
Specialisation in specific tasks
While LLM are like the Swiss army knives of the AI world, some tasks require highly specialised tools. SAM (Segment Anything Model) is a model from Meta designed exclusively for recognising and separating objects in images. It can identify the boundaries of various objects, people or areas in a photograph with incredible precision.
SAM uses a trio of specialised components. An image encoder based on the Vision Transformer processes the input image and creates a dense representation of image features. A prompt encoder lets the user specify what they want to segment using points, boxes or rough masks. The mask decoder then generates the final segmentation mask using cross-attention between the image features and the prompt representations.
Models intended for generating images, music or video work in a similar way. Diffusion models such as Stable Diffusion or DALL-E 3 combine the principle of gradual denoising with key components of the transformer architecture. They work by gradually removing noise from a random signal, while the text encoder (almost always a transformer from the CLIP model) processes the prompt and the U-Net denoising architecture uses cross-attention mechanisms from transformers in its internal blocks in order to “understand” what the image should look like according to the text description.
Efficiency and speed
No less important are SLMs (Small Language Models), which are smaller versions of large language models. These compact versions sacrifice part of their capability in exchange for speed and lower demands on computing power. They are ideal for mobile applications, embedded systems or situations where you need an immediate answer.
SLMs achieve their efficiency in several ways. Quantisation reduces the precision of numbers from 32-bit to 8-bit or even 4-bit representations, which saves a significant amount of memory. Pruning removes less important neurons and connections in the network. Knowledge distillation allows a smaller model to learn the behaviour of a large model by imitating its outputs. Modern SLMs such as Phi-3 or Gemma also make use of architectural innovations such as grouped-query attention or sliding-window attention.
SLMs can run directly on your phone or tablet without needing an internet connection. For many everyday tasks such as translation, summarisation or simple queries they deliver entirely sufficient results.
When extra power is needed
At the opposite end of the spectrum we find MoE (Mixture of Experts) models. These sophisticated systems contain dozens or hundreds of specialised “expert” networks, and for each specific query they activate only the most suitable ones. The result is enormous performance with relatively reasonable resource consumption.
The MoE architecture includes a routing network that decides which experts will be activated for a given token. Typically only a small percentage of all the experts is activated (for example 2 out of 64). That means the model can have trillions of parameters, but for a single forward pass it uses only a fraction of them. The Switch Transformer was one of the first successful MoE models, followed by modern implementations such as Mixtral 8x7B from Mistral AI or, in all probability, GPT-4 as well.
Imagine it as a specialised hospital where there is a corresponding expert for every problem. MoE models can deliver answers of a quality that would be unthinkable for monolithic models of a similar size.
Encoder-Decoder models: the best of both worlds
Between decoder-only LLMs and encoder-only MLMs there is a third pillar of the transformer architecture: encoder-decoder models such as T5 (Text-to-Text Transfer Transformer) or BART (Bidirectional and Auto-Regressive Transformers). These models combine the strengths of both approaches.
The encoder part uses bidirectional attention for a deep understanding of the input text, while the decoder part generates the output autoregressively. This architecture is extremely effective for sequence-to-sequence tasks (converting one sequence into another) such as machine translation, text summarisation or transcribing data from one format into another.
Technically, encoder-decoder models process the input in two phases. The encoder first creates a rich representation of the whole input text using bidirectional attention. The decoder then uses this representation by means of cross-attention to generate the output sequence Token by token.
Graph neural networks: when relationships matter
Graph Neural Networks (GNNs) represent a completely different approach to artificial intelligence, one that focuses on data structured as graphs. While LLMs work with sequences and VLMs with images, GNNs process entities and the relationships between them.
GNNs are indispensable in areas where the relations between objects are crucial. In social networks they analyse relationships between users for recommendation systems. In chemistry they help understand the structure of molecules for drug development. In transport they optimise routes in road networks.
Technically, GNNs use message-passing algorithms in which every node (entity) in the graph collects information from its neighbours. This process is repeated across several layers, which allows nodes to “see” ever more distant parts of the graph. Different variants such as graph convolutional networks (GCN) or graph attention networks (GAT) differ in the way they collect and weight information from neighbouring nodes.
World models: learning by observing reality
A completely different approach is represented by world models, which learn by observing the physical world instead of reading texts. The best-known example is V-JEPA (Video Joint-Embedding Predictive Architecture) from Meta, which represents a radical departure from current trends.
V-JEPA learns from more than a million hours of internet video, rather like small children who grasp gravity by dropping a spoon a few times and watching what happens. The model observes how objects move, how they react to forces, how they interact with each other, and gradually builds an internal notion of how the physical world works.
Technically, V-JEPA uses a Joint-Embedding Predictive Architecture, which makes predictions in an abstract feature space instead of in the space of raw pixels. Instead of trying to predict the exact value of every pixel in a future video frame, the model learns to predict its high-level semantic representation (embedding). This process allows the model to ignore unpredictable details such as leaves moving in the wind and to focus on semantically important information such as the trajectory of a moving object.
The V-JEPA architecture consists of three components. The encoder processes the visual input and converts it into an abstract representation. The predictor then predicts how this representation will change over time. The most interesting is the third component – a mechanism that allows the model to predict not only what will happen by itself, but also what will happen as a reaction to a specific action.
The result is a model that can work as a “simulator in the head”. When a robot “considers” which action to take, V-JEPA can predict the consequences of the various options and choose the best one without having to carry out dangerous experiments in the real world.
Audio models: sound as another modality
Sound processing represents another important area of AI that this article has so far passed over. Audio models fall into several categories according to their purpose.
Speech-to-Text models such as Whisper from OpenAI can convert spoken speech into text with surprising accuracy even in a noisy environment and in various languages. They often use a combination of CNNs (convolutional neural networks) for processing spectrograms and transformers for sequence modelling.
Text-to-Speech models generate a human-sounding voice from text input. Modern approaches such as neural vocoders use deep learning to create naturally sounding voice tracks with control over emotion, tone and speaking style.
Generative audio models such as MusicGen from Meta can create music on the basis of text descriptions. Like image diffusion models, they work by gradually refining an audio signal in a latent space.
Logical reasoning and symbolic processing
LAMs (Logic-Augmented Models) represent an interesting hybrid of neural networks and symbolic artificial intelligence. These models combine the ability of neural networks to learn from data with the precision of formal logic. They use external reasoning engines such as Prolog or SMT solvers to solve complex logical problems.
The architecture of a LAM typically includes a neural module for understanding natural language, a symbolic reasoning engine for logical inference and an integration layer that translates between neural representations and symbolic formulas. This combination allows models to solve tasks requiring precise logical reasoning that are difficult for purely neural networks.
Latent consistency and structured generation
LCMs (Latent Consistency Models) represent an advanced approach to generative tasks. Instead of the traditional diffusion process they use consistency training in a latent space. That means the model learns to generate consistent outputs from various levels of noise in a single-step process.
Technically, LCMs make use of a consistency function that maps any point on the trajectory from noise to a clean signal onto the same final point. This allows dramatically faster generation than traditional diffusion models, because it eliminates the need for an iterative denoising process.
Masked Language Models and bidirectional understanding
MLMs (Masked Language Models) such as BERT use a fundamentally different approach from autoregressive LLMs. Instead of generating text from left to right, MLMs learn to predict masked tokens on the basis of bidirectional context.
Bidirectional attention in an MLM allows the model to see the whole sequence at once, which is ideal for text-understanding tasks such as classification, named entity recognition or question answering. Pre-training takes place by means of the masked language modelling objective, in which 15% of tokens are randomly masked and the model learns to predict them.
Technical problems and optimisation
The development and deployment of advanced AI models brings a range of technical problems that require constant innovation in algorithms, hardware and optimisation strategies. They differ depending on the type of model, but some are common to most large-scale systems.
Large models contain trillions of parameters, which places enormous demands on graphics card memory and computing power. The memory problem is critical especially during training, when it is necessary to store not only the model parameters but also intermediate results and optimiser states. LLMs suffer from what is called exposure bias. During training they always see the correct tokens as the input for the next generation step, but during actual inference the model has to rely on its own, potentially erroneous, previous predictions, which can lead to an accumulation of errors. The solution lies in techniques such as Teacher Forcing with scheduled sampling, where there is a gradual transition from feeding in reference tokens to using the model’s own generated ones. In multimodal models such as VLMs and audio models it is crucial to ensure correct alignment of the modalities, so that the model correctly connects information from different data types. Contrastive learning is one of the main techniques that helps models learn how different modalities correlate – for example by pairing images with their captions in such a way that similar pairs have similar vector representations.
Memory efficiency is a critical area for all large models. Gradient checkpointing allows a trade-off between memory and computing time. Instead of storing all intermediate results for backpropagation, some activations are recomputed, which reduces memory consumption at the cost of a slightly longer training time. Mixed-precision training uses 16-bit floating-point numbers instead of the standard 32-bit ones for most calculations and for storing parameters. This dramatically reduces memory demands and speeds up calculations on modern GPUs, which have special cores for lower precisions.
The ZeRO optimiser, developed by Microsoft, distributes optimiser states, gradients and model parameters across several GPUs, which makes it possible to train models too large to fit on a single GPU. Innovative algorithms such as FlashAttention and PagedAttention work with memory for the attention mechanism more efficiently, which makes it possible to process longer sequences and train larger models. FlashAttention optimises I/O operations, while PagedAttention improves efficiency during inference through dynamic management of attention keys and values.
For deploying models into production, low latency and high throughput are crucial, and these are achieved through techniques such as model pruning, quantisation to lower bit depths and knowledge distillation, in which a smaller model learns to imitate the behaviour of a large one. Training trillion-parameter models requires massive distributed systems, which involves efficient data parallelism, model parallelism and pipeline parallelism.
The future?
The future of artificial intelligence looks like a dynamic and many-sided field in which specialisation and hybrid approaches will play an ever more important role.
Current developments show a clear trend towards specialisation. Instead of one universal model we will probably encounter ecosystems of various AI tools, each of which will be optimised for specific tasks. That means complex applications will not rely on a single LLM, but will intelligently combine different models – for example a VLM for visual analysis, a GNN for graph databases and a specialised SLM for fast local operations.
Retrieval-Augmented Generation (RAG) paradigm strongly dominates practical AI applications and its importance will continue to grow. RAG systems use vector representation models to search for relevant documents in extensive knowledge bases and generative models to formulate answers on the basis of the retrieved information. This hybrid architecture makes it possible to use the strengths of different approaches and solves key problems such as hallucination or outdated data, because the model has access to current and verified information. In the future we can expect even more sophisticated RAG systems that will be integrated with more complex data sources and logical inference mechanisms.
Techniques for joining several modalities are becoming ever more sophisticated and will be crucial for creating AI that understands the world in a more complex way. Early fusion combines modalities at the level of input features at the start of processing. Late fusion joins the outputs of separate modality-specific networks, where each modality is processed by a separate model. Intermediate fusion allows interaction between modalities at various levels of the processing pipeline, which leads to a deeper mutual understanding of information from different sources. We expect the development of more advanced cross-attention mechanisms and new architectures that will allow smoother interaction between different data types.
Integration with symbolic AI and logical reasoning is another important direction. LAMs (Logic-Augmented Models) represent only the beginning of a trend that will try to bridge the gap between neural networks and symbolic AI. Future models will probably integrate formal logic, ontologies and knowledge graphs more deeply, so that they can better solve tasks requiring planning, problem solving and explainability.
Researchers are also constantly experimenting with new architectural innovations. For example, Mamba (State Space Models) represents a promising alternative to transformers for processing long sequences, since it uses state space models that are more efficient in terms of memory and computational demands. RetNet (Retention Networks) is another alternative to transformers, aimed at improving the efficiency of training and inference for long sequences, with linear complexity. Mixed-depth models combine layers with different processing depths, which makes it possible to distribute computing resources more efficiently and achieve better performance.
For most ordinary users this diversity will be practically invisible, since applications and services will use the most suitable models in the background, while the user interface will remain simple and unified. Expect more intuitive interactions in which AI will be able to understand complex queries involving various modalities and provide more accurate and more contextually relevant answers. For developers and researchers, understanding the different types of AI models and their optimisation strategies will be crucial for the effective development and deployment of new AI solutions. The world of artificial intelligence is far richer than it might seem at first sight, and its diversity is the key to solving ever more complex tasks.
What does this mean for ordinary users?
For most people this diversity will be practically invisible. Applications and services will use the most suitable models for specific tasks in the background, while the user interface will remain simple and unified.
Nevertheless, understanding the different types of AI models will help you make better use of the available tools and perhaps even anticipate what new possibilities will appear in the future. The world of artificial intelligence is far richer than it might seem at first sight, and its diversity is the key to solving ever more complex tasks.
