Fine-tuning need not be the first step for a language-model application. A team can state the task in a prompt, supply demonstrations and retrieve relevant documents without changing any weights. That work establishes whether the base model can do the job and exposes the failures that remain. LoRA and QLoRA make the next step cheaper, but they do not tell us whether to take it.
The distinction is between behaviour worth learning and information worth supplying. A stable output schema, a recurrent transformation or a consistent tone may justify adaptation. Current prices, private documents and changing policy text belong in the prompt or a retrieval system. Fine-tuning examples are a poor substitute for an updatable source of facts.
Establish the prompted baseline
Prompting changes the input, not the model. Instructions, demonstrations and decoding settings can be revised in minutes, while retrieval can replace evidence without another training run. This makes a prompted system the right control for an adaptation experiment. Use the same base model, representative requests and fixed decoding settings when comparing the two.
Measure more than average answer quality. Count malformed outputs, missed instructions and unsupported claims separately. Record prompt tokens and serving latency as well. A long prompt that already works may be cheaper to operate than a training pipeline; a shorter adapted prompt may repay its training cost only at sufficient volume.
OpenAI’s August guidance for supervised fine-tuning identifies steerability, reliable formatting and custom tone as useful targets. It also recommends combining fine-tuning with prompt engineering, retrieval and function calling rather than treating them as rivals. That division suits open-weight deployments too. Keep evidence outside the weights unless the behaviour itself must change.
The test set should remain outside every stage of prompt development and adapter training. Otherwise, a team can tune both approaches to the same examples and mistake familiarity for improvement. Include ordinary requests, difficult cases and examples that should trigger a refusal or an error. A gain on a general chat benchmark cannot settle this application-specific comparison.
LoRA learns a constrained weight update
Full fine-tuning produces a new value for every model parameter. LoRA instead freezes a pretrained projection matrix W₀ and learns an update with restricted rank. For an input x, the adapted projection is:
h = W₀x + BAx
If W₀ has d × k parameters, A has shape r × k and B has shape d × r, with rank r much smaller than either dimension. The full update would contain dk learned values; its factorisation contains r(d + k). Applying this pattern across selected attention or feed-forward projections yields a small adapter rather than another complete checkpoint.
The method begins with BA = 0, so the adapted model starts at the base model’s behaviour. Training computes gradients for A and B while leaving W₀ unchanged. This avoids gradients and optimiser state for the frozen parameters. It does not remove the base weights, activations or the backward computation needed to reach the adapters. Parameter-efficient does not mean memory-free.
Rank and layer selection remain experimental choices. The original paper found very small ranks sufficient on its tasks, but explicitly did not predict that result for every dataset. QLoRA’s authors later found that adapters on all linear Transformer blocks mattered in their instruction-tuning setup. Copying a rank from another project is no substitute for a controlled comparison.
LoRA’s headline savings are equally specific. Its GPT-3 175B experiments reported up to 10,000 times fewer trainable parameters and threefold lower training GPU memory than full fine-tuning with Adam. These figures describe that architecture and configuration. They are not constants for a seven-billion-parameter open-weight model with a different optimiser, sequence length and adapter layout.
QLoRA compresses the frozen base
On its own, LoRA leaves the base at its existing storage and computation precision. QLoRA attacks that larger memory term. It quantises the frozen pretrained weights to four bits, then backpropagates through them into higher-precision LoRA adapters. The base receives no weight update.
Three mechanisms make the recipe work. NormalFloat4 chooses quantisation levels for approximately normally distributed weights, allocating equal numbers of values to each bin. Block-wise scales preserve local ranges, and double quantisation compresses those scales rather than leaving every scale at 32 bits. Paged optimisers use unified memory to move optimiser state to CPU memory when temporary GPU pressure demands it. They address spikes; they do not make long sequences inexpensive.
Four bits describe storage, not the entire arithmetic path. QLoRA dequantises weight tensors to 16-bit brain floating point for matrix multiplication in the paper’s implementation. The LoRA factors also use the higher-precision computation type. Consequently, a smaller memory footprint does not imply a fourfold speed-up. Dequantisation, memory traffic, kernels and sequence length determine throughput.
The May paper demonstrated fine-tuning of a 65-billion-parameter model on one 48 GB GPU. Its experiments used the original LLaMA and T5 families, not Llama 2, which Meta released in July. At 33B and 65B, the paper compared QLoRA with 16-bit LoRA rather than establishing parity with full 16-bit fine-tuning. Those boundaries matter more than the memorable hardware figure.
Activation memory still grows with batch and sequence dimensions. The paper notes that activation gradients, not adapter parameters, dominate some LoRA training configurations. Gradient checkpointing can reduce that pressure at the cost of additional computation. QLoRA changes what fits; it does not repeal the training memory budget.
Adapters change the serving design
An adapter is small only in relation to its base. A deployment must retain the exact compatible base revision, tokenizer and adapter configuration. Record the target layers, rank, scaling and training-data lineage with the artefact. An adapter trained against one checkpoint is not a universal plug-in for another architecture or revision.
Serving offers two basic arrangements. Merge BA into W₀ and the resulting matrix has the original shape. LoRA therefore adds no inference latency by construction compared with a fully fine-tuned model. The cost is a complete merged copy for each resident variant, and the original paper notes the difficulty of batching different merged tasks in one forward pass.
Alternatively, keep the base shared and load adapters per tenant or task. This saves storage and makes rollback a matter of selecting the previous adapter. It also retains the extra projection calculations and adds routing, cache and batching decisions. A service must measure these costs under its actual mixture of requests. Quantised training does not prescribe quantised serving, and neither choice guarantees a faster response.
Open weights still come with terms
Llama 2 made this operating model concrete in July: Meta released pretrained and chat-tuned weights and starting code in sizes from 7B to 70B, free of charge for research and commercial use. “Open-weight” describes that access precisely. It should not imply unrestricted open-source licensing.
The community licence incorporates an Acceptable Use Policy, redistribution conditions and additional commercial terms for organisations above its stated monthly-active-user threshold. Local control over weights and training data does not remove those obligations. It also transfers responsibility for safety testing, access control, monitoring and incident response to the deployer.
Fine-tune when the evidence earns it
A sound decision follows the workload. First, build the strongest practical prompted and retrieved baseline. Next, identify a repeated behavioural deficit with a measurable cost. Then curate examples that demonstrate the desired behaviour, keeping a separate test partition. Finally, compare the adapter against that baseline on quality, total prompt cost, training expense and measured serving performance.
Reject the adapter if it merely matches a simpler prompt or improves one aggregate score while damaging another required behaviour. Keep retrieval for fresh facts and retain a regression set for capabilities the base already had. QLoRA’s own evaluation cautions that chatbot benchmarks can disagree and that automated judgements contain biases.
LoRA supplies a compact way to learn a repeated correction. QLoRA lowers the hardware needed to learn it over a large frozen base. Their value lies in making that correction testable and deployable, not in making adaptation compulsory.
Sources and further reading
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models — the update equations, memory savings and merged-versus-dynamic serving trade-off, version 2, 16 October 2021.
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs — NF4, double quantisation, paged optimisers, experiments and evaluation limits, version 1, 23 May 2023.
- Meta, Meta and Microsoft Introduce the Next Generation of Llama — the Llama 2 weights, starting code, release sizes and responsible-use resources, 18 July 2023.
- Meta, Llama 2 Community License Agreement — usage, redistribution, acceptable-use and additional commercial terms, release dated 18 July 2023; version committed 21 July 2023.
- OpenAI, GPT-3.5 Turbo fine-tuning and API updates — suitable behavioural targets and combination with prompting and retrieval, 22 August 2023.
- Mangrulkar and Paul, PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware — adapter-only checkpoints and shared-base loading, 10 February 2023.