LoRA (Low-Rank Adaptation) is a parameter-efficient method for fine-tuning large models. It starts from the observation that the change in weights needed to adapt a pre-trained model to a particular task has low intrinsic rank, and can therefore be approximated well by the product of two narrow matrices. The original weight matrix is frozen and a correction is added to its output in the form of the product of matrices A and B, where the shared inner dimension r is usually only in the single or double digits – against thousands in the original matrix. Only these adapters are trained, which typically amounts to less than one percent of the model’s parameters, and memory consumption for the optimizer and the gradients drops sharply as a result. The adapter’s influence is scaled by the ratio of alpha to r. Adapters are inserted above all into the query and value projections of the attention mechanism. Once trained, they can either be merged back into the weights with no slowdown at inference whatsoever, or kept separate so that you can switch between many specializations on top of one shared base. Combined with a quantized base model, the approach is known as QLoRA.
You do not have to rebuild an entire shop to make it look festive – a film stuck in the window will do. The building itself is untouched, the film weighs a few grams, and after the holidays you peel it off and put up the Easter one. LoRA is exactly that kind of film over an enormous model: the model itself runs to hundreds of gigabytes and nobody is going to retrain it for a single company, yet an adapter a few dozen megabytes in size can shift its tone and its expertise wherever you need them. And because you can keep twenty such films in a drawer, one model can serve the legal department, support and marketing – each with its own.