OpenAI announced GPT-4 on 14 March with a striking comparison: its simulated Uniform Bar Exam score sat around the 90th percentile of test takers, against roughly the 10th for GPT-3.5. The technical report published the next day supplies broader evidence of improvement. GPT-4 scores 86.4% on MMLU in a five-shot setting, compared with 70.0% for GPT-3.5, and outperforms earlier general language models across the reported academic suite.
These results demonstrate more than polished conversation. They show a substantial gain in the ability to answer difficult questions across many domains under specified test conditions. Yet they do not establish uniform competence, factual reliability or fitness for an application. The report itself contains the qualifications needed to resist that leap: uneven exam results, contaminated evaluation data, prompt-dependent measurements and post-training trade-offs.
The useful question is therefore not whether the benchmarks are genuine. It is what each result measures, under which configuration, and which uncertainties survive the headline score.
Capability gains are real but uneven
MMLU provides a broad test of knowledge and problem solving across 57 subjects. Its original authors designed it to expose blind spots as well as average performance, having found earlier models strong in some subjects but near chance in others. GPT-4’s result marks a large advance on that defined task. It does not turn 57 subject scores into evidence of equally strong performance on every task, or on work that differs from multiple-choice questions.
The wider launch table makes the variation visible. GPT-4 improves markedly on AP Calculus BC but shows no gain over GPT-3.5 on the reported AP English examinations. It solves 31 of 41 easy Leetcode problems but only three of 45 hard ones. Its Codeforces rating remains below the fifth percentile. On DROP, GPT-4’s 80.9 F1 score exceeds GPT-3.5 and the compared general language model, but trails the 88.4 achieved by a benchmark-specific system. A general model can make a major advance without dominating every specialised alternative.
Human percentiles need equally careful treatment. A simulated bar exam combines multiple-choice and free-response components, approximates published scoring methods and maps the result to an earlier test-taker distribution. It measures performance on that simulation, not the judgement, client interaction or professional accountability expected of a lawyer. Other percentiles inherit the boundaries of their own rubrics and comparison groups. The conservative choice to report the lower end of a percentile range reduces some overstatement; it cannot make unlike forms of work equivalent.
Contamination changes the meaning of a test
A public benchmark can become training material. Exact overlap lets a model reproduce an answer rather than solve an unseen instance, while near duplicates and online explanations make the boundary less clear. OpenAI acknowledges that a minority of exam questions appeared in pre-training and reruns exams after removing detected cases. The reported score is the lower of the full and filtered variants.
The detection method samples three 50-character substrings from each processed evaluation example and searches for matches in the training data. This is a useful audit, not proof of independence. The report notes both false negatives, such as matches missed after small textual changes, and false positives. Some checks omit answers or multiple-choice options, which can also affect what gets removed. Filtered results occasionally improve: the LSAT score rises after detected questions are discarded. Contamination is therefore a methodological complication, not a simple instruction to subtract a fixed margin.
Two stronger examples show why disclosure matters. OpenAI excludes BIG-bench after finding that parts were inadvertently mixed into training. For GSM-8K, it deliberately includes part of the benchmark’s training set in pre-training to improve mathematical reasoning, while reporting contamination checks on the test set. The resulting 92.0% should be read, as the report recommends, somewhere between few-shot transfer and full benchmark-specific tuning. That is a different claim from success on an entirely unseen mathematical distribution.
Post-training data needs the same scrutiny. The report says it did not check RLHF data for overlap with TruthfulQA. This does not invalidate the large improvement reported after RLHF, but it leaves a specific uncertainty around a result used to support improved factuality. Transparent limitations are more informative than an unqualified clean-test assumption.
Prompting is part of the measured system
There is no configuration-free benchmark score hiding behind these tables. The academic evaluations specify their prompting: five-shot MMLU, zero-shot HumanEval, and five-shot chain-of-thought for GSM-8K. The multiple-choice exam prompts include worked answers and explanations. Multiple-choice answers are generated through an explanation sampled at temperature 0.3, sometimes followed by a separate answer-letter sample at zero temperature. Free-response answers use a different instruction format and temperature 0.6.
These choices alter the object being evaluated. Few-shot examples supply task format and domain cues; chain-of-thought elicits intermediate reasoning that a plain request may not produce. Decoding settings change the sampled output. A result consequently describes a model, prompt, examples and decoding procedure working together.
The report’s development process reinforces that point. Researchers iterated on exam methodology using non-held-out exams before running the final held-out versions. This is sounder than tuning on the test itself, but it still means the published pipeline was selected for the evaluation family. Some model snapshots also differ: multiple-choice and free-response exam runs use different GPT-4 versions, with an earlier snapshot for the biology olympiad. Aggregate exam evidence is strong, but it is not one untouched model answering every question under identical conditions.
For application assessment, copying a benchmark’s aggregate score without its setup would be a category error. The relevant comparison must preserve the intended prompting strategy and account for the tools, context and output constraints that surround the model.
Post-training changes more than capability
Pre-training and RLHF leave different signatures in the results. Across the multiple-choice exam sections, the base model averages 73.7% and the post-RLHF model 74.0%. OpenAI concludes that exam capability comes primarily from pre-training and is not substantially changed by this post-training. Free-response comparisons are harder because instruction following itself affects the usefulness of the answers.
TruthfulQA shows a different pattern. The base GPT-4 model is only slightly better than GPT-3.5, whereas RLHF produces a large gain. The benchmark’s design explains why that distinction matters. Its 817 questions across 38 categories seek imitative falsehoods: plausible errors learned from human misconceptions rather than simple gaps in knowledge. Predicting likely text and answering truthfully are related but different objectives. Post-training can shift behaviour towards the latter without creating equivalent gains on exam questions.
The improvement still has boundaries. TruthfulQA separates truthfulness from informativeness because refusing every question would score as truthful but provide no useful answer. Its authors also warn that strong performance does not establish truthfulness in specialised domains, long-form generation or extended adversarial conversations. A benchmark gain is evidence about a targeted failure mode, not a general certificate of factuality.
Calibration adds a trade-off. On an MMLU subset, the pretrained base model’s confidence closely tracks accuracy, but post-training substantially worsens that relationship. Better instruction following and targeted truthfulness can therefore coexist with poorer confidence estimates. A system cannot infer dependable uncertainty from fluent phrasing, and even token probabilities need evaluation after the final training stages.
From model evidence to application fitness
GPT-4’s release evidence supports a bounded conclusion: pre-training produces broad capability gains, prompting elicits them in particular ways, and post-training improves some behaviours unevenly. It also leaves known failures. The model hallucinates, makes simple reasoning errors, can introduce security vulnerabilities into code and largely lacks knowledge after September 2021. It does not learn from its interactions.
Application fitness depends on the cost and detectability of those failures in the intended workflow. A reviewer transforming supplied source text can inspect omissions and unsupported additions; an automated system giving consequential advice faces a harder verification problem. OpenAI accordingly recommends human review, grounding in additional context or avoiding high-stakes use altogether, matched to the application.
That recommendation should survive the leaderboard. Use the benchmarks to identify promising capabilities and likely failure modes, then test the complete configuration against representative work. Treat factual accuracy, calibration, robustness and safety as separate questions rather than features implied by exam performance. GPT-4’s results justify serious evaluation of what the model can enable. They do not justify delegating the evaluation to the headline scores.
Sources and further reading
- OpenAI, GPT-4 research announcement (14 March 2023): the launch comparisons, model limitations, training distinction and application precautions.
- OpenAI, GPT-4 Technical Report, arXiv v1 (15 March 2023): exam and academic results, evaluation settings, contamination appendices, RLHF comparisons and calibration evidence.
- Hendrycks et al., Measuring Massive Multitask Language Understanding, arXiv v3 (12 January 2021): MMLU’s design, subject coverage and analysis of lopsided performance and calibration.
- Lin, Hilton and Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods, arXiv v2 (8 May 2022): imitative falsehoods, truthfulness versus informativeness, prompting effects and the benchmark’s limits.