A pretrained language model learns to continue text, not to satisfy the person supplying it. An instruction is only another sequence of tokens unless training changes what the model does with that sequence. FLAN and InstructGPT address this gap with different supervision. FLAN teaches a model to transfer supervised task formats to unseen task types. InstructGPT learns which responses people prefer, then optimises a model against that learned preference.
The distinction matters. Strong zero-shot accuracy does not establish that users will prefer an answer, and a preferred answer need not be true. Evaluating instruction following therefore requires several tests, not a single leaderboard.
FLAN generalises a supervised interface
FLAN starts with LaMDA-PT, a 137-billion-parameter decoder-only model, and fine-tunes it on 62 public datasets organised into twelve task clusters. Researchers wrote ten natural-language instruction templates per dataset, including some that reverse the original task. During training, an example is rendered through a template and the model learns to predict its target tokens. The loss remains supervised maximum likelihood; the innovation is the varied instructional interface across many tasks.
The evaluation split is more demanding than withholding individual datasets. To test natural language inference, for example, FLAN removes every dataset in that task cluster from instruction tuning. It also removes closely related clusters where necessary. Each held-out cluster requires a separate checkpoint. This design tests whether the model can apply an instruction to a kind of task it has not explicitly been taught.
The results support that narrower claim. Using the best development-set template, zero-shot FLAN beats zero-shot GPT-3 on 20 of 25 evaluated datasets. That comparison includes template selection, so it should not be read as uniform superiority for arbitrary wording. Gains are strong on tasks naturally expressed as instructions, but weaker where an instruction merely wraps sentence completion. Across seven commonsense-reasoning and coreference tasks, FLAN improves on only three over its untuned base. Supervised specialist models also remain better on most datasets.
T0 provides an independent check on multitask prompted training. Its 11-billion-parameter encoder-decoder model matches or exceeds zero-shot GPT-3 on nine of eleven held-out datasets, and training with more prompts generally improves median performance and reduces sensitivity to wording. Yet FLAN’s own ablations show worse held-out performance after instruction tuning at eight billion parameters and below, whereas T0 reports gains at three billion. Different architectures, pretraining objectives and prompt mixtures prevent a simple universal scale threshold.
Two evaluation limits remain. Task clusters involve subjective boundaries, and web pretraining can expose a model to evaluation material even when fine-tuning splits are clean. Held-out task performance measures transfer under a defined partition, not unrestricted understanding.
InstructGPT optimises a model of preference
InstructGPT begins with supervised fine-tuning too, but its demonstrations come from labellers responding to labeller-written prompts and prompts submitted to earlier models in OpenAI’s API Playground. About 13,000 training prompts establish a supervised policy. The next stage changes the supervision: labellers rank between four and nine candidate responses to a prompt.
A six-billion-parameter reward model learns to assign a scalar score to a prompt-response pair. Pairwise comparisons train it to score the preferred response above the alternatives. Roughly 33,000 prompts supply this reward-model dataset. Proximal policy optimisation then updates the supervised policy on another 31,000 unlabelled API prompts to increase the learned reward. A penalty on divergence from the supervised policy restrains the model from moving beyond the reward model’s useful range.
This is reinforcement learning from human feedback, not a larger collection of correct answers. The reward model compresses comparative judgements into a scalar reward signal. Earlier human-feedback work on summarisation showed why restraint matters: light optimisation improved human ratings, but further optimisation eventually diverged from the reward model’s predictions and reduced actual preference.
On prompts from held-out API customers, evaluators prefer the 1.3-billion-parameter InstructGPT to 175-billion-parameter GPT-3. At the larger size, they prefer InstructGPT outputs to GPT-3 outputs 85±3% of the time, and to few-shot GPT-3 outputs 71±4% of the time. These are substantial gains on that prompt distribution, not evidence that the smaller model is more capable in general.
The comparison with multitask training is equally specific. OpenAI fine-tuned its own GPT-3 baselines on the FLAN and T0 mixtures; these are not the original FLAN and T0 models. Labellers prefer InstructGPT to both on the API distribution. Open-ended generation and brainstorming account for about 57% of that distribution, while classification and question answering account for about 18%. Public NLP mixtures cover a different workload. Neither result makes one training recipe universally superior.
Preferences need independent behavioural tests
Human preference answers a relative question: which response did these raters choose under these instructions? It does not directly measure truthfulness, toxicity, bias or retained capability. InstructGPT therefore needs its public evaluations, and their mixed results are as informative as the preference wins.
TruthfulQA supplies 817 questions across 38 categories, designed to elicit falsehoods common in human text. It measures truthfulness and informativeness separately because always declining to answer would be truthful but useless. InstructGPT produces truthful and informative answers about twice as often as GPT-3 on this benchmark. On closed-domain API tasks, where responses should contain no information absent from the input, its hallucination rate is 21% against GPT-3’s 41%. Neither figure licenses a claim of general factual reliability. The models still invent facts and sometimes accept false premises.
Toxicity results depend on context. InstructGPT produces fewer toxic outputs than GPT-3 when asked to be respectful, but that advantage disappears without the respectful prompt. It shows no significant improvement on the paper’s Winogender and CrowS-Pairs bias evaluations. RealToxicityPrompts itself relies partly on an automated toxicity classifier known to produce biased false positives, which makes human evaluation and attention to affected groups necessary.
Better instruction following can also improve compliance with harmful requests. When explicitly prompted for toxic output, InstructGPT can be more toxic than GPT-3. Alignment here means movement towards selected labellers’ judgements, not a safety guarantee or universal human values.
Finally, optimisation can erase useful behaviour. PPO reduces performance on SQuAD, DROP, HellaSwag and French-to-English translation relative to GPT-3. PPO-ptx mixes pretraining updates with preference optimisation and greatly reduces these regressions without a material preference loss. It does not eliminate every deficit, and reintroducing pretraining data can also reintroduce undesirable behaviour.
Production teams must choose the target distribution
The practical choice starts with the workload. Multitask supervised tuning suits teams with labelled examples, recognisable task types and trustworthy automatic metrics. Preference modelling suits open-ended work where people can rank candidates more reliably than they can specify one ideal answer. A broad system may need both: task diversity for transfer and preference data for response quality.
Data governance becomes part of model quality. InstructGPT’s prompts are over 96% English and its roughly 40 contractors are not representative of everyone affected by deployment. Their instructions, screening and organisational context shape the learned reward. A separate group of labellers reproduces the broad preference result, but this does not establish agreement among wider user populations. Record who supplied prompts and judgements, split evaluation data by user, and preserve disagreement rather than forcing every comparison into a consensus.
A release gate should keep the four measurements separate. Hold out complete task types where generalisation is claimed. Sample human evaluations from the intended workload, with fixed prompts and decoding settings. Test truthfulness and harmful behaviour under both ordinary and adversarial instructions. Finally, rerun a frozen capability suite against the untuned baseline and set explicit rollback thresholds for unacceptable regressions. Multiple generations per prompt help expose rare but severe failures that an average score can hide.
FLAN shows that supervised examples can teach a reusable task interface. InstructGPT shows that comparative feedback can steer that interface towards selected human preferences. Neither removes the need to define whose preferences count, which failures matter and which capabilities must survive training.
Sources and further reading
- Wei et al., Finetuned Language Models Are Zero-Shot Learners — the ICLR 2022 FLAN paper, including task-cluster splits and ablations, 8 February 2022.
- Sanh et al., Multitask Prompted Training Enables Zero-Shot Task Generalization — the T0 paper on prompt diversity and held-out tasks, 13 December 2021.
- Ouyang et al., Training Language Models to Follow Instructions with Human Feedback — the InstructGPT method, evaluations and limitations, 4 March 2022.
- Stiennon et al., Learning to Summarize from Human Feedback — earlier evidence on reward modelling and over-optimisation, 15 February 2022.
- Lin, Hilton and Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods — the benchmark and its evaluation criteria, May 2022.
- Gehman et al., RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models — the toxicity testbed and classifier caveats, 24 September 2020.