Sources checked
What is LoRA, compared with fine-tuning?
Edward Hu and colleagues define it in LoRA: Low-Rank Adaptation of Large Language Models. Full fine-tuning retrains all parameters.
They use GPT-3 175B as the example of why that becomes expensive: each fine-tuned copy is another 175-billion-parameter model to store and serve. LoRA freezes those weights and injects trainable rank-decomposition matrices into Transformer layers.
Against GPT-3 175B fine-tuned with Adam, they report 10,000 times fewer trainable parameters and a 3-times lower GPU memory requirement.
On quality, they say LoRA is on par or better than fine-tuning for RoBERTa, DeBERTa, GPT-2, and GPT-3, with fewer trainable parameters, higher training throughput, and, unlike adapters, no added inference latency. That is their comparison.
Fine-tuning is the full-update approach LoRA is trying to avoid storing, and QLoRA is a later method that stacks quantization on this idea.
Why the base checkpoint has to travel with the adapter
The frozen weights are part of the result. A low-rank update trained on one checkpoint is a change relative to those weights.
Loading it onto a different base, or a newer revision of the same name, is a different model until you test it. Record the base version, the data revision, and the rank settings together.
What LoRA does not fix
A smaller update does not review your examples. If the pairs or the instructions teach the wrong behavior, LoRA will train that behavior into the matrices.
It also does not add documents the base model lacks. Missing facts are a retrieval problem, not a rank problem.
Common questions
What is LoRA versus fine-tuning?
Fine-tuning, as this paper uses the term, retrains the model’s parameters. LoRA leaves those parameters frozen and trains small low-rank matrices instead.
Can one LoRA adapter be reused on any model of the same family?
Not automatically. It was trained against a specific frozen checkpoint.
Pair them and test the combination.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





