A small language model is not a cheaper substitute for every large one. It is a component with a narrower capacity budget and a different operating envelope. The production question is therefore not whether a checkpoint counts as small, but whether the complete service can perform a defined task on the available machine with acceptable failures.
That requires separating techniques often packed into the same announcement. Curated or synthetic pretraining shapes what a model learns. Distillation trains a smaller student from a teacher. Quantisation changes the numerical representation of a trained model. Adapters specialise a frozen base. They can be combined, but each has a distinct quality risk and resource cost.
Bound the task before comparing checkpoints
Start with the work rather than a parameter threshold. A classifier, a support-reply drafter and an open-ended research assistant place different demands on knowledge and reasoning. Define the input distribution, required output, unacceptable errors and conditions under which the system should decline. Then build a domain test set that preserves routine cases alongside difficult and sensitive slices.
Run a stronger model as a reference where policy permits, but compare useful outcomes rather than its prose style. A small model may be sufficient when answers draw on supplied evidence or the output space is constrained. It is a poor default when the task needs broad factual recall, unfamiliar multi-step reasoning or reliable behaviour across many domains. Retrieval can supply current evidence; it cannot add whatever reasoning capacity the base model lacks.
Keep a prompted version of the small model in the comparison. If prompting already meets the requirement, pretraining a new model or fine-tuning an existing one adds machinery without earning its operating cost. The best candidate is the least demanding system that passes the task and hardware gates together.
Data curation is not distillation
Phi-3-mini provides a useful example of the first mechanism. The April report describes a 3.8-billion-parameter model trained on 3.3 trillion tokens drawn from heavily filtered web data and synthetic data. Its authors attribute the result to that data recipe. They do not present Phi-3 as a distilled model, so the synthetic material must not be relabelled as teacher-to-student compression.
Curation spends the model’s limited capacity on selected patterns instead of asking it to absorb the whole web. Synthetic generation can then expand coverage of chosen instructions or situations. TinyStories makes the boundary unusually clear: larger models generated stories within a deliberately restricted vocabulary and world, allowing much smaller models to produce coherent stories in that world. This supports specialisation, not a claim that the same models acquired broad adult knowledge.
A production data recipe needs controls for duplication, fabricated facts, unwanted bias and gaps in coverage. Audit the generator’s output and preserve provenance. Synthetic volume cannot compensate for a narrow prompt distribution, and fluency within a generated domain says little about performance on real requests outside it.
Distillation uses a different training signal. A student learns to reproduce a teacher’s output distribution, often alongside the ordinary task loss. DistilBERT combined this distillation loss with masked-language modelling and an objective aligning hidden-state directions. Reducing the number of layers produced a smaller general-purpose encoder that could then be fine-tuned for downstream tasks. The mechanism transfers behaviour through the objective; merely using text generated by a large model does not establish the same mechanism.
A student can also inherit the teacher’s errors and blind spots. Evaluate it directly on the target domain rather than treating the teacher’s quality as an inherited guarantee.
Quantisation and adapters solve different problems
Post-training weight quantisation enters after training. It stores weights with fewer bits and reconstructs higher-precision values for computation where the runtime requires them. Four-bit weight storage can make a checkpoint fit a device, but the word “four-bit” does not describe activations, the key–value cache or every arithmetic operation.
The quantisation method matters because rounding errors are not equally harmful everywhere. GPTQ uses calibration data and approximate second-order information to choose weight rounding while compensating remaining weights. AWQ instead identifies salient channels from activation magnitudes and scales them to reduce quantisation error. These are deliberate quality-preservation mechanisms, not proof that every model and task will survive the same bit width.
Measure the delta between the full-precision candidate and the exact quantised artefact on every important task slice. A small aggregate change can conceal damage to rare entities, instruction following or calibrated refusals. Include out-of-domain cases as well: calibration on one distribution need not protect behaviour on another.
Adapters address specialisation rather than storage alone. LoRA keeps the base weights fixed and learns small low-rank updates to selected projections from domain examples. QLoRA goes further for training: it backpropagates through a frozen four-bit base into higher-precision adapters, reducing the memory needed to fine-tune. That result concerns adaptation. It does not prescribe the serving format or guarantee faster inference.
An adapter also needs an exact compatible base revision and tokenizer. Test general capabilities that must survive specialisation, and decide whether to merge the update into a resident checkpoint or route requests to separate adapters. Merging simplifies execution but multiplies complete model copies; sharing a base saves storage while adding routing and batching work.
Turn the hardware budget into measurements
Parameter count multiplied by bits per weight gives only the first memory term. Add quantisation scales and metadata, runtime buffers, activations and the key–value cache. The cache grows with context and concurrent sessions, so a model that fits with a short prompt can still fail under production traffic. Measure peak resident memory at the intended context lengths and concurrency, not just the downloaded file size.
Latency has separate stages. Prompt processing affects time to first token; autoregressive decoding determines how quickly the rest arrives. Long inputs can make the first stage conspicuous even when subsequent generation is fast. Report both, with tail percentiles and timeouts. Then measure completed tokens per second under representative concurrent load. Single-request generation speed is not service throughput, and larger batches can improve utilisation while worsening an individual caller’s wait.
Phi-3-mini’s report makes this concrete without supplying a universal phone benchmark: its four-bit version occupied approximately 1.8 GB and generated more than 12 tokens per second offline on an iPhone 14. Those figures demonstrate feasibility on one device and configuration. They do not predict first-token delay, sustained throughput under thermal limits or performance on another handset.
Quantised speed depends on the execution path. GPTQ’s reported gains came from reduced memory movement with specialised kernels, while its paper explicitly excluded batched speed-ups and activation quantisation. AWQ likewise paired its format with kernels designed for regular memory access. If a runtime instead spends its savings on slow dequantisation or unsupported operators, a smaller checkpoint can remain slow. Benchmark the model, runtime and device as one unit.
Privacy and fallback belong in the routing policy
Local inference can keep prompts and responses off an external service and continue without a network. That is a meaningful privacy boundary, but not automatic data protection. Application logs, crash reports, analytics, cached prompts and OS backups can still disclose the same content. Set retention and access rules for each, and account for the device’s physical security.
Fallback needs the same scrutiny. Sending a failed local request to a hosted model can silently cross the boundary that justified local inference. Permit that route only for approved data classes, disclose it and apply the remote provider’s controls. Otherwise choose a fallback that preserves the boundary: a deterministic search or form, a queued retry, or human assistance.
Do not use a low generation score as a universal escalation signal. Detect failures the application can actually define, such as schema violations, missing evidence, timeouts or explicit out-of-scope classifications. Exercise those routes in release tests. A fallback that has never been run under load is another untested model path.
Select on cost per successful task
The final comparison should include the base model, any adapted version and each deployable quantisation level on the same frozen domain suite. Record quality changes by slice, peak memory, first-token and generation latency, throughput and fallback outcomes. Include failed requests and retries in operating cost. A cheaper model that needs a second attempt on difficult cases may cost more per completed task.
Choose the smallest configuration that clears every required gate with headroom for real traffic. Keep the larger or remote option for workloads whose breadth justifies it, rather than forcing one small checkpoint across every product surface. The advantage of a small model comes from a precise fit between learned behaviour and machine budget, not from the label itself.
Sources and further reading
- Abdin et al., Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (v1), 22 April 2024.
- Eldan and Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (v2), 24 May 2023.
- Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (v2), 16 October 2019.
- Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (v1), 31 October 2022.
- Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (v1), 1 June 2023.
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (v1), 23 May 2023.