OpenAI released ChatGPT as a free research preview on 30 November 2022. The model, fine-tuned from the GPT-3.5 series, sits behind an interface built for conversation rather than single prompts. A user can ask a follow-up, correct a premise or supply another constraint without constructing a new prompt from scratch. OpenAI also invites reports of bad answers and mistakes made by its content filter.
This accessibility changes more than presentation. ChatGPT’s training data and feedback preserve the structure of a dialogue. Human trainers demonstrate both sides of a conversation, then rank alternative assistant replies in context. Reinforcement learning from human feedback (RLHF) turns those rankings into a training signal. The result is a model tuned to continue an exchange usefully, not merely to complete a passage of text.
That distinction explains both the preview’s appeal and its limits. A conversational policy can become better at eliciting and acting on intent while remaining unreliable about facts. The reward model learns what trainers prefer; it does not acquire a dependable reference against which every claim can be checked. Treating a polished answer as verified knowledge mistakes the training objective for a guarantee.
Dialogue changes the supervision problem
A pretrained language model learns statistical regularities by predicting tokens. That objective supplies no explicit boundary between a user request and an assistant response, nor does it specify what should happen after an answer. InstructGPT had already shown that demonstrations and preference rankings could steer a model towards following instructions. ChatGPT adapts that recipe to multi-turn interaction.
For supervised fine-tuning, trainers write conversations while playing user and assistant. They can use model-written suggestions to compose responses, but the resulting examples encode human choices about the exchange. OpenAI mixes these conversations with earlier InstructGPT data converted into dialogue form. The supervised model therefore learns more than an answer style: it learns how an assistant turn relates to preceding turns.
The comparison stage retains that context. Trainers converse with the chatbot; OpenAI selects a model-written message, samples alternative completions and asks trainers to rank them. A reply is consequently judged as the next move in this conversation, not as an isolated response to a generic prompt. This matters when a constraint appeared several turns earlier or when a short follow-up only makes sense through its antecedent.
The visible interface exploits the same structure. Follow-ups let the user narrow a task, repair a misunderstanding or reject an unsupported assertion. OpenAI describes a model that can answer follow-up questions, admit mistakes, challenge incorrect premises and refuse inappropriate requests. These are learned behaviours to test, not properties every response will display. In particular, the launch notes that the model usually guesses at ambiguous intent instead of asking a clarifying question. Conversation creates the opportunity for clarification; it does not ensure that the model takes it.
From rankings to a response policy
RLHF separates the judgement people make from the optimisation the machine performs. First, supervised examples provide an initial policy: a distribution over assistant responses given a dialogue. Next, a reward model learns a scalar scoring function for candidate responses. Its training target comes from the relative rankings, so a higher score should predict a response trainers would prefer. Finally, Proximal Policy Optimization (PPO) updates the response policy to obtain higher rewards. OpenAI repeats data collection and optimisation over several iterations.
This arrangement makes comparison useful where a single target answer would be too restrictive. Many responses could address a request, but they differ in relevance, clarity and safety. Ranking alternatives lets trainers express a preference without writing the one correct continuation. Earlier work on summarisation established the same basic loop: collect comparisons, learn a reward model and optimise a policy against it.
The learned score is nevertheless a proxy. It compresses a multidimensional judgement into one number, shaped by the instructions, candidate responses and people supplying the rankings. InstructGPT’s authors are explicit that their procedure aligns a model with a particular group of labellers and researchers, not with universal human values. Their evaluation also separates preference from truthfulness, toxicity and bias because a preferred answer need not excel on every axis.
Optimisation adds another gap. A policy can exploit regularities in the reward model that do not correspond to what people actually want. The summarisation research found that predicted reward could keep rising while human preference deteriorated under excessive optimisation. InstructGPT’s PPO recipe therefore includes a penalty for moving too far from the supervised policy. This restraint reduces drift; it cannot turn an imperfect reward into a complete specification.
Dialogue makes the data more representative of interactive use, but it does not remove those constraints. Contextual rankings can teach a model to maintain a thread or acknowledge a correction. They cannot, by themselves, establish that its account of the world is accurate.
Fluency is not a fact check
OpenAI’s launch post gives unusual prominence to this failure mode. ChatGPT can produce plausible but incorrect or nonsensical answers. During reinforcement learning there is no source of truth in the training loop. A trainer can recognise some errors, especially in familiar or easily checked cases, but the reward model must generalise from a limited set of such judgements. Fluent, comprehensive prose can make an error harder for a rater to spot.
Related research on helpful and harmless dialogue assistants documents the problem directly. Preference models sometimes favour a sophisticated answer containing misleading details over an honest admission of ignorance. The authors did not expect crowdworkers to fact-check every response. Human preference remains valuable, but it is not interchangeable with verification.
Supervised demonstrations have a related difficulty. The ideal demonstration depends on what the model can reliably produce, while a trainer writes from the trainer’s own knowledge. Copying an expert’s confident answer does not necessarily teach the model the boundary of its own competence. Increasing caution is not a free repair either: OpenAI notes that it can make the model decline questions it could answer correctly. Useful deployment requires a balance between unsupported assertion and unhelpful refusal.
Other launch limitations reinforce the point. Small changes to phrasing, or another attempt at the same prompt, can change the answer. Rephrasing is therefore not an independent check. The model is also excessively verbose, which OpenAI attributes partly to trainers preferring longer responses that appear comprehensive and to over-optimisation. A learned preference can reward the surface signals of thoroughness while adding no evidence.
Safety has the same layered character. RLHF can encourage refusals of inappropriate requests, yet harmful instructions may still succeed and biased behaviour remains. An external content filter can block or warn about some unsafe material, but OpenAI expects both missed cases and mistaken interventions. Neither the policy nor the filter warrants treating the whole system as uniformly safe.
A research preview is not a finished product
The preview makes these behaviours available for broad inspection. Its accessible interface reduces the need for prompt-engineering expertise, while feedback controls let users flag problematic outputs and filter errors. This creates a channel for finding failures that laboratory evaluation missed. It does not mean every conversation becomes a reliable label, nor that public reaction supplies ground truth.
The release model matters operationally. ChatGPT is available as a hosted research system, not as open weights or a stable component with production assurances. OpenAI plans regular updates, so behaviour observed today may not carry over unchanged. The preview is a way to study an evolving model through use, not evidence that alignment has been completed.
Production use needs independent checks
A team evaluating ChatGPT should start with the cost of an incorrect answer. Drafting and restructuring text may leave a reviewer able to inspect every claim. Open-ended factual questions are harder: the interface supplies no general mechanism for establishing whether an answer is true. Supplying trusted source material and asking for a transformation keeps verification closer to the task, although the output still needs review for omissions and unsupported additions.
Evaluation should reproduce real conversations rather than stop at single prompts. Include follow-ups that depend on earlier constraints, ambiguous requests that merit clarification and corrections to false premises. Vary the wording and repeat trials to expose instability. Measure harmful outputs and unnecessary refusals together; improving one by indiscriminately worsening the other is not a useful safety result. Domain experts should review cases where raters without specialist knowledge could prefer a confident error.
Those tests support controls outside the model. Keep a person accountable for consequential use, check factual assertions against trusted evidence and record the model behaviour evaluated so that an update triggers fresh tests. Monitor the content filter’s false positives as well as its failures to intervene. These measures do not repair the reward model; they prevent its limitations from becoming invisible inside a product.
ChatGPT’s advance is the joining of contextual human feedback to an interface that ordinary users can steer through conversation. The same design also clarifies the boundary of RLHF. Dialogue can reveal intent and rankings can shape a useful response policy, but factual reliability requires evidence and evaluation beyond the preference signal. The fluent turn is an output to assess, not a reason to stop checking.
Sources and further reading
- OpenAI, Introducing ChatGPT (30 November 2022): the research-preview announcement, including its dialogue-data collection, RLHF method and stated limitations.
- Ouyang et al., Training language models to follow instructions with human feedback, arXiv v1 (4 March 2022): the InstructGPT recipe, evaluation boundaries and account of whose preferences alignment represents.
- Stiennon et al., Learning to summarize from human feedback, arXiv v2 (27 October 2020): the comparison–reward–PPO loop and evidence that excessive reward optimisation can diverge from human preference.
- Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arXiv v1 (12 April 2022): dialogue-specific preference collection, tensions between helpfulness and harmlessness, and failures of preference models on misleading answers.
- Glaese et al., Improving alignment of dialogue agents via targeted human judgements, arXiv v1 (28 September 2022): contemporary work on per-turn dialogue preferences, targeted rule evaluation and the difficulty of supervising nuanced behaviour.