Assigning an issue to a coding assistant changes more than the speed of writing code. It changes the unit of work. A completion tool suggests the next fragment; OpenAI’s Codex and GitHub’s Copilot coding agent can inspect a repository, edit several files, run commands and return a pull request. The engineering problem now spans the whole path from intent to integration.
That path needs explicit controls. A vague issue gives the agent room to choose the wrong requirement. Weak retrieval hides the code that determines correct behaviour. An unreliable environment turns verification into noise. Even a clean test run answers only the questions encoded in the tests. The pull request is therefore not a ceremonial final step: it is where a person compares intent, code and evidence before accepting responsibility for the change.
Start with a decision about behaviour
An issue assigned to an agent should state the observable behaviour required, not merely name a feature. Include a failing example or user journey, the expected result, relevant edge cases and boundaries that must remain unchanged. Say which compatibility constraints matter and which work is out of scope. This is a specification for accepting a change, not a script dictating every edit.
The distinction matters because several implementations can satisfy the same wording. A request to improve validation could mean rejecting malformed input, repairing it or producing a clearer error. Each choice can look plausible in a diff. Without an explicit decision, the agent makes a product choice while appearing to perform a mechanical task. A reviewer then has to reconstruct the intended behaviour from code that already embodies one interpretation.
SWE-bench formalised this setting by pairing an issue and repository snapshot with tests derived from its resolving pull request. The later SWE-bench Verified work exposed how difficult that pairing is to curate. Human annotators found underspecified problem statements and tests that imposed requirements absent from the issue. This is not just a dataset problem. A team faces the same mismatch when a ticket omits an acceptance condition but continuous integration enforces an unstated convention.
Repository instructions answer a different question. Codex can read AGENTS.md files for navigation, project conventions and test commands; Copilot also uses repository instructions. These files explain how to work here. They cannot decide what the issue means. Keep those contracts separate so a reviewer can tell whether a failure came from misunderstood intent or from ignorance of local practice.
Retrieve relationships, not just matching files
Repository-scale work does not mean placing every file in one prompt. Copilot clones the repository and analyses it with retrieval augmented generation powered by GitHub code search. Codex gives each task an isolated environment preloaded with the repository. Both designs let the agent inspect selected material as the task develops.
The useful unit of context is often a relationship: a public function with its callers, a schema with its migrations, or a component with its tests and styling conventions. Lexical matches may locate the symbol named in an issue while missing a wrapper that changes its behaviour. Conversely, returning too much similar code can bury the decisive path. Retrieval needs iteration after the first edit because new names, errors and dependencies become relevant.
SWE-bench’s original experiments illustrated this gap. Retrieval could include the files changed by the reference patch without supplying everything needed to understand their interactions. Its authors also observed worse behaviour as irrelevant context accumulated. Those results belong to particular early systems, not a fixed limit on current products, but the mechanism remains: access to a repository is not equivalent to understanding the relevant execution paths.
The execution environment supplies another form of context. Dependencies, generated files, runtime versions and test configuration must agree closely enough with development for results to mean something. A patch produced against a broken setup can fail for reasons unrelated to the issue. Teams should make setup repeatable and keep credentials outside the task unless they are strictly required. Isolation limits side effects; it does not make an unrepresentative environment informative.
Make the change executable
A coding agent improves on one-shot patch generation when it can observe the effects of its actions. It searches, edits, runs a focused check, reads the failure and revises. SWE-agent’s research shows why the interface around that loop matters. Bounded search results, visible edits and immediate syntax feedback reduce opportunities for small mistakes to propagate through a long session. The model is only one component; tools and their feedback shape what it can correct.
Codex can run test harnesses, linters and type checkers, then return a commit with file citations and terminal output. Copilot pushes commits to a draft pull request and exposes session logs. These artefacts separate a claim of completion from the work performed. A reviewer can inspect which files changed, which commands ran and whether the reported output supports the summary.
Checks should begin narrowly and widen as the change stabilises. A reproduction test can establish the reported failure before implementation. Focused tests give quick feedback during editing. Type checking, linting and a broader test suite can then expose effects beyond the first file. This ordering saves time, but speed is not the main purpose. Each check addresses a different failure: the issue persists, the new code is invalid, or existing behaviour has regressed.
The agent should report unresolved failures rather than negotiate them away. Deleting an assertion, weakening a fixture or excluding a test may produce a green run while changing the meaning of success. Review test edits with the same care as production code, and compare them with the issue’s acceptance conditions.
Treat tests as limited evidence
SWE-bench distinguishes tests that should change from failing to passing from those that should remain passing. The first group checks the requested repair; the second checks preservation of existing behaviour. That distinction is useful in production because a patch can fix the reported example while breaking another caller. Both kinds of evidence matter, and neither establishes general correctness.
Tests execute selected paths under selected conditions. They can miss concurrency, migration safety, accessibility, security and operational behaviour unless someone specifies and exercises those concerns. They can also inherit the agent’s mistaken interpretation. If the same agent writes the acceptance test and implementation from an ambiguous issue, agreement between them shows internal consistency, not fidelity to the requester’s intent.
Strong evidence therefore includes provenance. Record the exact commands, environment and results. Identify which acceptance condition each new test represents and which existing suites protect adjacent behaviour. Treat flaky or unrelated failures as signals to investigate, not details to omit. Codex’s citations and Copilot’s logs make this inspection possible; a concise claim that everything passed does not.
A benchmark result has a similar boundary. SWE-bench measures whether patches pass curated tests for a fixed collection of issues. It does not measure whether a team writes sound requirements, whether a repository’s tests cover production risks or whether a proposed design will remain maintainable. Use such evaluations to compare defined systems on defined tasks, not to certify unattended engineering.
Keep accountability at the pull request
The pull request gathers the specification, diff and execution record into one reviewable decision. Reviewers should first compare the change with the issue, then inspect implementation choices that tests cannot judge: fit with the architecture, handling of public contracts, security implications and the cost of future maintenance. Review comments can request another agent iteration, but a person must decide when the result is acceptable.
GitHub has preserved organisational controls around Copilot’s work. The agent pushes to its own branches, existing rulesets still apply and its pull requests require human approval before workflows run. The person who requested the pull request cannot approve it under the documented review rules. OpenAI likewise says Codex-generated code needs manual review and validation. These are necessary boundaries, not evidence that every repository has configured them adequately.
Delegation works best where those boundaries are already meaningful. GitHub recommends low-to-medium-complexity tasks in well-tested codebases. A reproducible bug with clear acceptance tests is a better assignment than an ambiguous migration across poorly understood services. Parallel tasks also multiply review work; producing more patches does not help if reviewers stop reading them.
The productive division is therefore specific. Let the agent search, edit and run checks in a controlled environment. Make the issue carry intent, the repository carry working knowledge and the test output carry reproducible evidence. Leave the merge decision with people who understand the consequences and remain accountable for them.
Sources and further reading
- OpenAI, Introducing Codex — isolated task environments, repository instructions, execution evidence and manual review, 16 May 2025; updated 3 June 2025.
- GitHub, GitHub Copilot: Meet the new coding agent — issue assignment, code-search retrieval, draft pull requests and review controls, 19 May 2025; updated 23 May 2025.
- Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — repository snapshots, patches and execution-based evaluation, v2, 5 April 2024.
- OpenAI, Introducing SWE-bench Verified — human review of underspecified issues, test validity and evaluation environments, 13 August 2024; updated 24 February 2025.
- Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — navigation, editing, execution feedback and error guardrails, v1, 6 May 2024.