ALGOTECHNIC
All articles
7 min read

Reasoning Models and Inference Compute: What o1 and DeepSeek R1 Made Visible

How RL teaches longer reasoning trajectories, why verification remains domain-bound, and how to balance accuracy against latency and cost per solved task.

  • reasoning models
  • inference compute
  • reinforcement learning
A reasoning trajectory winds through checkpoints towards an answer, with training and inference budgets separated.

A language model once presented a relatively simple serving bargain: supply a prompt, generate a response and pay mainly for its visible tokens. Reasoning models make that bargain incomplete. OpenAI’s o1 and DeepSeek’s R1 produce extended chains of thought before their final answers. The useful unit of computation is no longer just a model invocation, but a trajectory whose length and search strategy can change the result.

This does not make training less important. It divides the problem. Training establishes the knowledge and behavioural repertoire available to a model; inference spends computation to select and extend a path through that repertoire. More time can improve a solution without adding any new fact to the weights. It can also produce a longer, confidently wrong solution. The engineering task is to buy additional accuracy only where the work rewards it.

Training teaches the policy; inference executes it

Pretraining and supervised fine-tuning provide the initial distribution over responses. Reinforcement learning (RL) then changes that distribution by sampling responses, scoring them and increasing the probability of behaviour associated with higher reward. The resulting checkpoint contains a learned policy. At inference, that policy still has to generate every reasoning token for each new problem.

OpenAI describes o1 as trained with large-scale RL to produce a long chain of thought, refine its strategies, try alternatives and recognise mistakes before answering. These are learned behaviours, not a separate symbolic reasoning engine switched on around an ordinary model. The system card also links o1’s capability gains to its use of test-time compute.

DeepSeek’s account exposes more of the mechanism. R1-Zero starts from DeepSeek-V3-Base and applies RL without a preliminary supervised fine-tuning stage. That is not learning from scratch: the base model’s pretraining remains essential. During RL, its average responses grow longer, and behaviours such as revisiting steps and exploring alternatives emerge. R1 itself is not a pure-RL artefact. It adds thousands of cold-start examples, reasoning-focused RL, rejection sampling and further supervised fine-tuning, followed by another RL stage for broader preferences.

The optimiser also matters. R1 uses Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath. For a question, GRPO samples a group of responses and scores them. It estimates each response’s advantage relative to the group rather than training the separate value model used by Proximal Policy Optimization. Removing that critic reduces training memory and computation, but generating and scoring groups remains substantial work. Neither this detail nor the published API price establishes R1’s total training cost; the paper does not supply one.

A longer trajectory is a resource, not a guarantee

Autoregressive generation makes each reasoning step available to the next. A model can calculate an intermediate value, test an approach, notice a contradiction or revise an earlier attempt in this workspace. Additional tokens therefore permit operations that a short response cannot contain. They also create more opportunities to recover when the first approach fails.

Yet length has no necessary connection to validity. A model can repeat an error, elaborate a false premise or replace a correct answer with an incorrect revision. In Snell and colleagues’ experiments on MATH, a revision model sometimes turned correct answers into wrong ones. Their broader result is more useful than a simple “more thinking wins” rule: the best allocation of test-time compute depended on problem difficulty and the base model. Sequential revisions suited some easier problems; harder ones benefited from a mixture of revision and independent sampling; the hardest showed little gain.

Those experiments used specially fine-tuned PaLM 2 models and mathematical tasks, not every production workload. Still, they undermine a uniform token budget. Extra inference helps most when the model already has the required knowledge and a plausible route to the answer. It cannot reliably compensate for missing capability, and a routing signal based on estimated difficulty consumes computation of its own.

The shape of expenditure matters too. One long revision chain is serial: later work depends on earlier output. Best-of-N sampling generates independent candidates and can run in parallel, but multiplies total generation. Beam search uses a verifier to retain promising partial paths, adding verifier calls and risking over-optimisation of its scores. Accuracy, response latency and service throughput therefore respond differently to the same nominal compute increase.

Verification works where the domain supplies a check

RL needs a reward that distinguishes useful trajectories. R1-Zero used rule-based accuracy rewards for domains with executable criteria. A deterministic mathematical answer could be placed in a specified format and checked by rules. A LeetCode response could be compiled and run against predefined tests. A format reward separately enforced the reasoning and answer tags.

This is concrete verification, but its boundary is the check. Passing selected tests does not prove that code satisfies an unstated requirement or handles every input. A correct numerical result can follow flawed reasoning. No equivalent rule verifies a nuanced summary, a legal interpretation or an open-ended design. R1’s later data pipeline did expand beyond rule-checked reasoning, using a generative reward model for some examples and preference models for general helpfulness and harmlessness. Those judgements are not deterministic proof.

Process supervision offers richer feedback than a final-answer label. In earlier MATH research, human annotators labelled intermediate steps, and the resulting process reward model ranked candidate solutions more effectively than an outcome-only model. But collecting those labels was expensive, and the verifier still made mistakes. DeepSeek reports that it avoided neural outcome and process reward models for R1-Zero because reward hacking and retraining complicated large-scale RL; its own attempts with process rewards and tree search encountered practical limits.

Nor is self-verification an independent observer. When the same model critiques its own trajectory, the critique can inherit the same misconception. An explicit “checking” phase may improve the response while remaining wrong. Treat it as another learned operation whose value must be measured, not as evidence that correctness has been established.

Even the visible reasoning text needs restraint. OpenAI warns that chains of thought may not be fully legible or faithful to the underlying computation. A polished trace can explain an answer without being a reliable audit log. Production checks should inspect outputs and execute independent tests where possible rather than infer safety from an apparently sensible narrative.

Training uses sampled trajectories and domain rewards to update a policy; inference generates a bounded trajectory, applies available checks and measures accuracy, final-answer latency and cost per solved task.
RL changes the policy before deployment; each request then pays for its own trajectory. Deterministic checks cover only some domains, and a plausible reasoning trace is not itself proof.

Price the solved task, not the token

Long reasoning turns output length into an operational variable. DeepSeek’s launch priced its deepseek-reasoner API at $0.14 per million cached input tokens, $0.55 per million uncached input tokens and $2.19 per million output tokens. Its evaluation protocol allowed up to 32,768 output tokens per benchmark response. That ceiling is not a typical length or a recommended production setting; it shows how much output a reasoning evaluation may need to accommodate.

A low token price does not determine task economics. Include all generated reasoning, visible answers, repeated attempts, candidate samples and verifier calls. Failed requests count too. The useful measure is total spend divided by tasks completed to the required standard, with quality held to the same test for every candidate system.

Latency needs an equally precise definition. Time to first visible token can hide most of the work if reasoning precedes the answer. Measure time to an accepted final result, including retries and execution checks. Then test under concurrency: parallel candidates may shorten a difficult request’s wait while consuming capacity that other requests need.

A production policy can start with a short budget for routine requests and reserve longer trajectories for difficult, high-value cases. Escalate on signals the application can define, such as a failed schema check, test failure or timeout. Cap tokens and wall-clock time, and decide what happens when the budget expires. An unfinished chain is not an acceptable answer merely because it contains sophisticated reasoning.

Compare configurations on a frozen, representative task set. Record accuracy by task slice, final-answer latency, total output and cost per successful task. Include cases where extra reasoning damages a sound initial response. The choice is not training compute versus inference compute in the abstract. It is which trained policy, trajectory budget and independent checks meet the requirement at an acceptable operating cost.

Sources and further reading