Beyond the Leaderboard: How to Build a Coding-Agent Eval You Can Trust

Executive Summary (30-Second Brief)

Public coding-agent leaderboards screen models, but they do not predict whether an agent will deliver mergeable pull requests in your repositories. When METR asked open-source maintainers to review SWE-bench Verified patches that passed automated test suites, acceptance was 24.2 percentage points lower than the automated pass rate.

To build a decision-grade evaluation for your team, enforce four rules:

Evaluate the deployable package, not isolated model weights. Your score belongs to the combined model, runner, tool permissions, repository instructions (SKILL.md / AGENTS.md), and compute budget. Changing the runner alone can swing token consumption by 40x (The Scaffold Effect).

Gate tasks with two-sided grader checks. Standard test suites grant false credit to incomplete work (TESTJACK refuted 34.4% of passing trials in its corpus) while hardened verifiers often reject valid alternative designs (SpecGuardBench found 15% to 27% false rejection rates). Require both a reference patch and an independent valid implementation to pass, and require an unchanged repository and known incomplete or exploit patches to fail.

Quantify repeat variance and pin infrastructure. Identical agent configurations span 2.2 to 6.0 percentage points across reruns (On Randomness in Agentic Evals), and container resource limits alone can shift scores by 6 points (Anthropic). Pair candidates on the same task IDs, separate in-workflow retries from independent evaluation runs, and report paired uncertainty intervals.

Score human review acceptance when mergeability is the goal. Blind maintainers to agent identity, grade against a written rubric, and measure review and repair effort per accepted pull request.

Why Leaderboard Scores Fail Adoption Decisions

A coding-agent evaluation earns trust when it answers a specific decision about specified work under realistic conditions. If you are deciding whether an agent should prepare pull requests for your team, testing a model in isolation is not enough. You must test the full deployable package (model, runner, tools, repository instructions, and budget) on tasks drawn from that team’s actual work. Each task must be solvable without exposing its answer, graded by checks that accept valid alternatives while rejecting incomplete work, and repeated enough times to support the reported difference. DeepSWE demonstrates evaluating original repository tasks under a fixed runner, while METR’s maintainer study shows that automated test passes and human review acceptance can diverge sharply.

An agent eval consists of three parts: a collection of tasks, a runner that supplies tools and manages working context, and a grader that applies tests or review rules to the submitted work. The reported score belongs to that entire arrangement. Terminal-Bench 2.0 pairs models with compatible runners in its leaderboard chart (measuring tested packages), whereas DeepSWE pins one runner (mini-swe-agent) and prompt across model configurations to isolate model contrasts within that harness.

Write your decision contract before examining scores. For example, if the decision is whether an agent should prepare background bug-fix pull requests in a set of repositories for maintainer review, record:

The target repositories and issue families
Permitted network and tool access
Wall-clock time, token, and dollar budgets
Allowed human supervision or intervention
The primary outcome (such as an independently accepted patch) and the minimum improvement margin that justifies rollout

Five Failure Modes That Break Naive Evals

A passing benchmark result can mislead an engineering organization in five ways:

False credit: The grader accepts incomplete or broken work that fails to satisfy the written request.
False rejection: The grader denies a sound implementation because assertions overfit to an unstated detail or a specific reference design.
Contamination: The agent obtains answer information (such as a later upstream commit or package release) outside the information available at the task’s source cutoff.
Instability: A repeated attempt or grading run changes the verdict due to model stochasticity or infrastructure noise.
Transfer error: Measured benchmark success fails to carry over to your repository conventions and code review workflow.

A single task can suffer from several of these defects simultaneously, so log the specific cause of any failure or disputed pass rather than folding everything into a single headline percentage (TESTJACK, On Randomness in Agentic Evals).

The Transfer Gap: Passing Tests vs. Merging Code

The gap between passing a test suite and shipping mergeable code is substantial. METR asked open-source maintainers to review 296 agent pull requests across 95 issues from SWE-bench Verified (a human-filtered suite of repository coding tasks, per OpenAI’s annotation instructions) drawn from scikit-learn, Sphinx, and pytest. After adjusting for how reviewers treated original human patches (accepting 68% of 47 merged human patches in the hypothetical exercise), maintainer acceptance was 24.2 percentage points below the automated pass rate. Because METR gave agent patches a single review without a revision cycle or standard CI feedback, that 24.2-point gap describes those specific repositories and review conditions rather than a universal constant, but it proves why automated test passes cannot stand in for merge readiness.

Observational market trackers illustrate the volume of public agent activity, though they cannot substitute for a controlled local trial:

Amplifying’s State of the Coding Agent Market (observed November 2, 2026, October 26 snapshot) attributed 214.1 million output units across 30 agents, mixing commits and pull requests. In a separate sample of 268 merged agent pull requests (July 30 to August 28, 2026) in organization-owned repositories or repositories with at least 10 stars, 44% had a GitHub review from someone other than the author or a bot. Per Amplifying’s methodology, while bot-workflow counts are exact for that visible route, declared-attribution stock counts are estimates that can swing 30% to 40% between runs, and 17 of 47 tracked agents leave no detectable public signal.
Greptile’s Rise of the Overnight Agents (May 2026) used bot authorship, co-author footers, and branch prefixes to identify likely end-to-end agent pull requests across its reviewed population, finding those signals in 27.6% of merged pull requests in April 2026.

Both market sources describe public activity under their own attribution heuristics; neither measures task-level correctness or predicts performance on your backlog.

Bottom Line for Leaders: Treat executable test pass rate, repeat reliability, trajectory quality, maintainer acceptance, and total cost as separate columns on your scorecard. Never collapse them into a single opaque index.

Build Tasks That Represent Real Engineering

A credible evaluation task starts at a pinned repository commit with a request an engineer could reasonably attempt and observable behavior that a test suite can verify. The prompt must provide enough context to diagnose the issue without leaking the reference patch, and hidden tests must verify the requested behavior while tolerating valid implementation choices (SWE-bench Verified instructions).

Sample the Work You Expect to Delegate

Start with the backlog you actually plan to assign. Sample across repositories, languages, issue types, test maturity, dependency complexity, and risk levels:

Include both routine single-file fixes and multi-module integration work, and report their outcomes separately. A model that clears small bugs can still fail cross-module refactors, while a difficult-only suite will misstate everyday utility.
Maintain a strict held-out task set that remains unseen during prompt or SKILL.md tuning (Princeton SAgE survey).
Record repository membership for every task. Because tasks from the same repository share build tooling and conventions, uncertainty estimates must account for repository clustering (Adding Error Bars to Evals).

Three recent benchmark designs illustrate how sampling choices shape what a score actually measures:

Benchmark	What the Agent Receives	Strength for a Team Eval	Limitation to Carry Forward
DeepSWE	Short requests for 113 original tasks across 91 active repositories and 5 languages	New reference solutions kept out of upstream Git history block direct commit lookup	Fixed runner and functional tests do not measure maintainer acceptance
Senior SWE-Bench	Repository snapshots and short requests from recent PRs; 100 tasks (50 private)	Covers recent feature, bug, and performance work with adaptive validation	Private tasks and LLM-assisted judging require method and calibration disclosure
Terminal-Bench 2.0	Instructions, containers, final-state tests, and human solutions for 89 terminal tasks	Reproducible CLI environments test system tooling and environment recovery	Category mix extends beyond repository coding; aggregate scores require task-mix inspection

What Belongs in a Task Contract

Every task in your registry should have a machine- and human-readable contract recording:

Agent-visible request and base commit
Source cutoff (the latest project timestamp or state the agent is permitted to use)
Expected observable behavior and allowed interface changes
Pinned setup steps, fixtures, and dependency versions
Reference solution and hidden grading tests (stored strictly outside the agent’s workspace)

Manual prompt audits catch broken tasks before compute is wasted. In Demystifying LLM-Based Software Engineering Agents, Xia and colleagues reviewed 300 SWE-bench Lite issues and excluded 51 when constructing Lite-S (249 tasks) because the issue text exposed the exact patch, gave misleading solution directions, or omitted essential requirements. Similarly, when building RExBench’s 12 research-extension tasks, the authors reproduced the original paper results, had a domain expert implement a reference edit and a second author validate it, and revised instructions to supply missing setup details.

If you use human completion time to bucket task difficulty, always state the reference workers and the target success probability. In Measuring AI Ability to Complete Long Software Tasks (170 tasks), models’ 80%-success time horizons were 4x to 6x shorter than their 50%-success horizons, and 95% horizons could not be estimated reliably. A “two-hour 50% horizon” does not mean an agent can dependably clear two-hour tickets on your backlog.

Code-Level Walkthrough: How a Grader Can Over-Constrain a Task

Inspecting a real benchmark task down to its test file shows how specification wording and grader code can diverge. The Terminal-Bench 2 sanitize-git-repo task asks an agent to find and replace AWS, GitHub, and Hugging Face secrets in /app/dclm with four placeholder strings (<your-aws-access-key-id>, <your-aws-secret-access-key>, <your-github-token>, and <your-huggingface-token>) while leaving unrelated files untouched.

Reading the commit-pinned test_outputs.py grader reveals three test functions that define success in code:

test_removal_of_secret_information() checks that five hardcoded secret strings are absent (case-insensitively) across three target files (ray_cluster.yaml, process.py, and one tokenized JSON file).
test_correct_replacement_of_secret_information() asserts byte-for-byte string equality (assert contaminated_text == decontaminated_text) between each of those three working-tree files and a bundled reference file.
test_no_other_files_changed() runs repo.commit("d6987af002b122fef54bc0be402062c76488a4d9").diff(None) and raises a ValueError if any path outside those three files appears in the Git diff.

Notice the design tension: the README asks the agent to “find and sanitize sensitive API keys and tokens from a Git repository,” yet assert contaminated_text == decontaminated_text will reject any patch that differs by a single whitespace character or redacts an additional suspect token, and diffing working-tree files against a base commit does not verify whether secrets were purged from .git history. Before reusing any task, compare its visible prompt against every assertion in the test file and run a second, independently written valid solution through the grader.

Block Answer Leakage Without Breaking Task Setup

Historical repository tasks can leak their solutions at test time through Git reflogs, tags, package registries, GitHub search, or public discussion pages (NIST CAISI). At the same time, blanket network blocks can break legitimate setup if fixtures are not pre-staged (as in sanitize-git-repo, which clones its target repo during setup).

To balance isolation and solvability:

Separate network phases: SWE-Bench Pro V2 fetches dependencies during container setup, restricts the solving agent to its model inference endpoint, strips later Git history, and regrades patches in fresh containers (32 of its 642 tasks required network access during the grading phase to download grading dependencies). After auditing 731 older public tasks, the authors repaired or retired 480 tasks to release the 642-task V2 suite. In a separate experiment, 237 of 351 tasks solved by an open-network frontier run still passed after GitHub access was blocked, demonstrating that network rules materially shift scores.
Pin inputs and gate with reference runs: Where tasks require live package installation or service startup (as in Senior SWE-Bench’s allowlisted registries), pin dependencies during trusted setup first, then verify that setup, the reference solution, and the unchanged-repository check behave as expected under the scored runtime policy (Permission Denied / Boundary-Bench). Apply the egress policy to both shell commands and provider-hosted web tools (HarnessTax).
Account for training-time leakage: Test-time firewalls cannot prevent pretraining or fine-tuning memorization. AgentPack assembled public GitHub code changes co-authored by agents to fine-tune code-editing models, illustrating a direct pipeline from public repositories into model weights. Always version task and access changes so scores before and after a hardening pass are never silently pooled.

Protect Task Integrity and Grading (The Two-Sided Gate)

A trustworthy grader must pass a two-sided gate:

Sensitivity (No False Passes): An unchanged repository, incomplete patches, and adversarial shortcuts must fail.
Specificity (No False Rejections): Both the reference patch and an independently written valid alternative must pass.

Even automated task-mining pipelines require static workspace audits. In Skill Issue, reversing merged pull requests at a shared commit occasionally deleted a required import or left the original fix visible in dead code; a static check rejected 56 of 280 mined tasks in an intermediate koog pool, including 36 tasks that had already passed dynamic test gates.

What Empirical Audits Reveal About Grader Failures

Study / Audit	Scope & Method	Key Empirical Finding	Practical Lesson
TESTJACK	Generated trial-specific witness tests (passing the reference, failing the agent patch, and reviewed against the prompt by an LLM agent) across 8,862 trials	Refuted 1,545 of 4,487 (34.4%) passing trials across 5 coding benchmarks (method-specific refutation fraction)	Passing existing repository tests frequently leaves stated prompt requirements undone
Code Agents as Testers	Generated tests required to fail before a reference fix and pass after it on Python tasks	Filtering SWE-Agent patches yielded 45% precision and 10% recall (45% of selected patches were correct; caught 10% of all correct patches)	Generated reproducing tests help prioritize patches for review, but cannot serve as a standalone correctness verdict
Adversarial Hacker-Fixer Loops	Audited 1,968 tasks across 5 terminal benchmarks using an iterative Hacker, Fixer, and in-loop Solver cycle	Found 323 exploitable environments; reduced unhinted attack success on 77 Terminal-Bench tasks from 39.2% to 16.7%, but held-out benign solver passes dropped 10.9 points (76.1% to 65.2%)	Hardening a verifier against exploits easily breaks legitimate solutions if gated only on a single reference or LLM solver
OpenAI SWE-Bench Pro Audit	Audited the older 731-task public split of SWE-Bench Pro (July 2026)	Flagged 286 of 731 tasks on initial screen for tests that missed incomplete work or demanded unstated implementation details	Retain a regression bank of incomplete patches, empty diffs, and known exploits to replay after every grader edit
SWE-ABS	Adversarial test strengthening on SWE-bench Verified across 30 agents	Rejected 2,184 of 11,041 previously passing patches, but also uncovered reference-overfitted tests on 53 of 500 tasks	A lower score after test strengthening can reflect fixed false passes, overstrict assertions, or both
VerifierBench & SpecGuardBench	Evaluated LLM-generated graders across 222 tasks (4,993 valid/invalid cases) and 291 atomic specs	VerifierBench true-positive rates (accepting valid work) were 21.1% to 70.2% vs. 79.2% to 96.3% true-negative rates; SpecGuardBench rejected valid code on 15% to 27% of tasks	Generated verifiers are much better at rejecting broken code than accepting valid alternative implementations
CodeHacker & AInsteinBench	Competitive-programming checker testing and scientific repository expert review	Both demonstrate probing checkers with plausible wrong outputs alongside valid alternate implementations	Domain experts still find invalid patches passing and valid scientific approaches failing overly rigid assertions

Execute Security Claims and Audit Adaptive Graders

When evaluating security patches, do not substitute LLM judge agreement for code execution. In Fixing, Breaking, or Faking It?, two LLM judges evaluated 254 real Java CVE patches and flagged 28 and 43 functional regressions (“over-fixes”), whereas full-project Maven execution of the repository regression suites found only 1. Always execute proof-of-vulnerability and regression suites where behavior can be exercised, and keep stylistic code-quality rubrics separate from functional correctness.

When tasks permit open-ended architectural choices, adaptive validation (such as Senior SWE-Bench, where a validator agent uses a private expert recipe to write behavioral tests tailored to the submitted patch’s interface) can reduce false rejections. However, any generated test and judge rubric become part of the grader: archive them, test them against known valid and invalid patches, and always execute final grading in a clean container that reverts any test edits made inside the agent’s workspace (SWE-Bench Pro V2).

How to Tier Engineering Effort on a Budget: Writing a clean-room second implementation for every task in a 150-task suite can be prohibitively expensive. Tier your admission gate:

Tier 1 (Every task): Reference patch passes (3/3 runs), unchanged repo fails (3/3 runs), static workspace check confirms no broken imports or leaked dead-code fixes (Skill Issue), and setup works under the scored network policy (Boundary-Bench).

Tier 2 (Pilot tasks, hardened/adaptive graders, and disputed verdicts): Author an independent clean-room second solution and replay known incomplete and exploit patches (TESTJACK, Hacker-Fixer).

Measure Reliability Across Runs and Infrastructure

Coding agents are stochastic systems running on noisy infrastructure. An agent can solve a task on attempt one and fail it on attempt two, even at temperature=0.0. Repeating runs quantifies that variance, provided your task admission checks have already verified the grader.

Separate Oracle `pass@k` from Deployable Retry Loops

Define your success metric to match how the agent will actually operate in production:

pass@1: Expected success when the workflow produces a single submitted patch per task.
pass@k (Oracle vs. Deployed): Counts a task as solved if at least one of k independent attempts passes. In benchmark papers, pass@k is an oracle upper bound because the hidden test suite selects the winning attempt. In production, a retrying workflow only achieves that gain if an agent-visible check or ranker selects which attempt to submit without access to the hidden grader.
pass^k: Requires every one of k independent attempts to pass (measuring worst-case consistency).

In On Randomness in Agentic Evals, a single configuration on R2E-Gym (a repository coding suite) scored 34.4% on pass@1, 52.9% on oracle pass@5, and 15.5% on pass^5. Across 500 SWE-bench Verified tasks, running six model-runner pairs across two temperature settings (12 configurations total) 10 times each (60,000 trajectories) produced within-configuration single-run score spans of 2.2 to 6.0 percentage points, with the 6.0-point span occurring at temperature=0.0.

Two Denominators of Uncertainty: Run Variance vs. Task Sampling

Do not confuse rerun variance on a fixed suite with generalization across future engineering work:

Within-task repeat noise: How much the score moves when the agent retries the exact same ticket.
Between-task sampling error: How much the score moves when you sample new tickets from your backlog.

Running 1,000 repeats on one ticket measures that ticket’s pass probability precisely while teaching you nothing about the rest of your codebase. Conversely, running Terminal-Bench’s minimum of five trials per task (per its dataset card observed November 2, 2026) does not automatically power a small local comparison. In Do Context Files Help Coding Agents?, a 17-task, three-repository study with three repeats per task had only 57% statistical power to detect a massive 30-point effect.

How to compute paired intervals without statistical traps (Adding Error Bars to Evals):

Pair on task IDs and average repeats first: For each task, average each candidate’s score across its K independent repeats and compute the task-level paired difference. Never pool all n × K runs as if repeats of the same task were independent tasks; the variance of the task-level mean differences already captures both between-task variation and within-task repeat noise.
Handle repository clustering correctly: If you are evaluating future tasks within 2 to 5 named repositories, treat repositories as fixed strata and resample tasks with replacement within each repository (preserving each task’s repeat average and each repository’s weight), and restrict your claim to those named repositories. Resampling whole repositories to generalize to unseen codebases requires enough independent repositories (typically at least 10 to 20) to estimate between-repository variance reliably.
Predeclare your useful margin: Suppose your team decides a +5 percentage-point gain in accepted PRs justifies adoption. A paired 95% confidence interval of [+1, +7] points shows a positive difference but leaves the adoption threshold unresolved, whereas [+6, +10] clears the bar.

Pin Container Resources and Audit Trajectory Luck

Infrastructure ceilings directly alter agent pass rates. When Anthropic held the model, runner, and Terminal-Bench 2.0 tasks fixed while varying container resource limits, pass rates shifted by 6 percentage points between the strictest and most generous tiers (and infrastructure errors dropped from 5.8% to 0.5%), while a 227-task SWE-bench crossover showed a 1.54-point gain from 5x RAM. Similarly, Re-executing Coding-Agent Evaluations found that while aggregate rankings were stable in one archive replay, a reference-bound dependency cache rejected 6 of 15 valid alternative patches. Always record guaranteed CPU/RAM, hard kill limits, timeouts, container images, and provider outages, and interleave candidate runs across time blocks.

Finally, distinguish outcome reproducibility from process quality. AgentLens compared 1,136 passing OpenHands trajectories across 47 tasks against reference action graphs and labeled 122 (10.7%) as “Lucky” passes, though 15 of those used divergent but valid engineering approaches. Use trajectory anomaly scores to sample runs for human inspection, not as an automatic defect rate.

Compare Agents Under Controlled Conditions

Match the evaluated unit to the decision you are making:

For team adoption (Package Comparison): Compare the complete deployable system (model, native runner, tool permissions, prompts, SKILL.md / AGENTS.md, context policy, resource limits, retry rules, and human supervision).
For model selection inside your stack (Fixed-Harness Comparison): Pin the runner, tools, prompt, and budget so only the model weights vary. Where models and runners are compatible, a crossed design (testing each model across each runner on the same task IDs) separates model capability from harness fit.

How Non-Model Variables Swing Success and Cost

Variable Changed	Empirical Study	Measured Impact	Takeaway for Evaluation Design
Context window & harness safeguards	Same Model, Different Harness (169 SWE-bench Verified tasks)	Modifying context truncation, stall recovery, and command guards raised solves from 43 to 72 under a tight 20,480-token limit (near-zero difference under wide context)	Harness engineering changes outcomes dramatically under constrained context budgets
Model × Runner interaction	HarnessTax (7 models × 3 runners, 3 repeats on two 30-task sets)	Demonstrated substantial conditional success and cost shifts across runners for the same underlying model	Never attribute a benchmark win solely to model weights when runners differ
Runner token efficiency & turn caps	The Scaffold Effect (2 models × 3 runners on 50 Terminal-Bench Pro tasks)	OpenCode consumed ~40x more tokens per solved task than Goose (including failed runs and without the 40-turn cap applied to Goose/OpenHands-SDK), while pass-rate gaps were 0 to 8 points	Report total billed cost, tokens (including failed attempts), wall time, and human review labor per accepted patch
Repository skill files (`SKILL.md`)	Skill Issue (Claude Code + Sonnet 4.6 across 3 Kotlin repos, 20 to 26 held-out tasks/repo)	Automated prompt optimizers GEPA and SkillOpt averaged +4.9 and +0.1 points over an empty seed, but neither reached significance on a sign test (`p < 0.05`, best `p = 0.29`)	Version `SKILL.md` / `AGENTS.md` files, keep tuning tasks out of the test set, and verify gains on held-out work
Legacy rules in `AGENTS.md`	Lessons Outlive Their Failures (4 historical failure replays, 3 runs/arm)	3 of 4 historical failures never recurred with or without their old rule; only 1 (an outdated doc count) recurred (3/3) when its check was removed	Replay legacy instructions against their original triggering tasks and control system clock inputs
Execution permission tiers	Permission Denied (Boundary-Bench)	7 tasks were blocked by design under the strictest terminal policy before model capability could be measured	Test the reference solution under each permission tier and report policy-specific scores separately

Report What the Evidence Supports

A decision-grade evaluation report allows another engineer to audit every denominator, uncertainty interval, and limitation.

Keep Denominators Separate

Never add or conflate percentages that measure different stages of the lifecycle:

Study	Headline Figure	What It Actually Measures (Outcome & Denominator)
TESTJACK	34.4%	Trial-specific witness refutation fraction among 4,487 passing trials in its selected corpus
AgentLens	10.7%	Trajectory divergence label (“Lucky”) among 1,136 passing runs on 47 eligible tasks
METR	24.2-point gap	Adjusted gap between automated test pass rate and single-review maintainer acceptance across 3 repositories

Check Whether Close Leaderboard Ranks Are Actually Tied

When two candidates finish within a few percentage points, report the exact number of discordant tasks (tasks where one candidate passed and the other failed) and apply your paired uncertainty test. In Coding Agents Have Converged, an audit of published SWE-bench Verified results found that the top two entries differed on only 36 of 500 tasks, and no adjacent pair among the top 30 was statistically separated by an exact paired test (even before accounting for multi-run stochasticity). Similarly, when summarizing run consistency across diverse suites, show per-task repeat distributions and task composition rather than relying on a single summary number like intraclass correlation (Stochasticity in Agentic Evaluations).

Structure Human Review and Post-Merge Telemetry

Blinded Maintainer Review: Give qualified maintainers the agent-visible request and permitted repository context, blind them to which candidate produced the patch, and grade against a written rubric (correctness, regressions, security, scope discipline, test quality, and maintainability). Duplicate a subset of reviews to measure inter-reviewer agreement, and if your workflow allows an agent or human revision loop after review comments, measure the time and cost of that loop directly (METR).
Post-Merge Quality Tracking: If patches are merged into production, track reverts, follow-up bug fixes, and incidents over a defined window. Note that observational metrics (such as Greptile’s revert counts per 1,000 merged PRs or 7-day follow-up edits on commented files) are noisy proxies confounded by task difficulty and prior human review; use controlled cohort tracking on your own merged PRs instead.

The 8-Step Team Protocol for Trustworthy Agent Evals

While no single study validates this entire sequence end to end, each step combines proven empirical safeguards into an auditable engineering record:

Step	Action	Artifact Produced	Primary Failure Prevented
1. Write the Decision Contract	Define target repos, evaluated packages, autonomy tier, time/cost budgets, primary outcome, and minimum useful margin (DeepSWE, METR).	Signed Decision Contract	Moving the goalposts after seeing scores; testing the wrong unit
2. Build a Held-Out Task Registry	Sample real tickets; record base commit, source cutoff, prompt, expected behavior, fixtures, and exclusion reasons. Keep scored tasks out of prompt/`SKILL.md` tuning (Princeton SAgE).	Versioned Task Registry	Prompt overfitting; unrepresentative task mix
3. Admit Runnable Tasks	Pin dependencies during setup, apply the scored network policy, and verify that the reference passes and an unchanged repo fails across repeated runs (Boundary-Bench).	Container & Network Manifest	Test-time answer lookup; broken task environments
4. Challenge Both Grading Directions	Enforce reference, no-op, and workspace-integrity checks on all tasks (Senior SWE-Bench, Skill Issue). On pilot tasks, hardened/adaptive graders, and disputed verdicts, require an independent second solution to pass and incomplete/exploit patches to fail (TESTJACK, Hacker-Fixer).	Two-Sided Grader Audit Log	False credit (reward hacking) & false rejection (reference overfitting)
5. Pilot & Freeze the Design	Run a small pilot on separate tasks to estimate paired task variance and repeat noise. Size task count `n` and repeat count `K` for your target margin, then freeze the allocation and resource limits (Randomness, Context Files, Error Bars).	Frozen Pre-Registration Spec	Underpowered trials; infrastructure noise (Anthropic)
6. Run Paired Attempts	Execute candidates on the same task IDs from clean containers, interleave execution order, log all tokens/costs, and separate in-workflow retries from independent evaluation repeats `K` (Randomness, Error Bars).	Raw Trajectory & Cost Archive	Run-to-run luck; temporal API drift; hidden token inflation (Scaffold Effect)
7. Review Patches Independently	Apply each package’s agent-visible selection rule to produce its submitted patch per run. Have blinded maintainers score every submitted patch (or a predeclared random run per task) under a written rubric, counting missing submissions as unaccepted (METR, Error Bars).	Blinded Review Scorecard & Paired Interval	Merging unmaintainable or over-scoped code that happens to pass tests
8. Release & Maintain the Suite	Publish per-task and aggregate tables, denominators, paired intervals, cost per accepted PR, and grader repair diffs. Quarantine broken tasks and rotate in fresh held-out work (SWE-Bench Pro V2, OpenAI Audit, Senior SWE-Bench, Princeton SAgE).	Living Eval Release Notes	Benchmark saturation and training-time contamination

Opinionated Synthesis: Where Agent Evals Go Next

Public benchmarks answer narrower questions than team adoption: DeepSWE’s original tasks and fixed runner isolate model comparisons within its chosen suite, while Senior SWE-Bench’s recent tasks and adaptive validation cover a broader mix of feature, bug, and performance work. For an engineering organization deciding what to deploy, however, no public leaderboard predicts maintainer acceptance on its local backlog (METR, Hacker-Fixer). The decisive evidence is a task-paired trial on the work the team plans to assign: testing the deployable package under declared resource limits, measuring repeatability and total cost, and having maintainers review the resulting patches.

Sustaining a trustworthy agent evaluation requires treating tasks and graders as living software. Released tasks and trajectories eventually enter training corpora (AgentPack), while new agents discover shortcuts that earlier test suites missed (SWE-Bench Pro V2). The most important open problem in evaluation research is a matched study that scores the same patches across executable tests, fresh retries, trajectory audits (AgentLens), and blinded maintainer decisions to test whether any automated process signal reliably predicts human merge acceptance beyond a test pass.

Until that link is established, start small: write a decision contract, pilot 15 to 20 tasks from your own repositories with two-sided grader checks, and use the observed variance to size the comparison that will actually decide your rollout (On Randomness in Agentic Evals, Adding Error Bars to Evals).

References & Further Reading

Huang et al. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (July 2026).
METR. Many SWE-bench-Passing PRs Would Not Be Merged into Main (March 2026).
Yao et al. TESTJACK: Should You Trust the Results in Coding-Agent Benchmarks? (October 2026).
Zhong et al. Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops (June 2026).
Bjarnason, Silva, and Monperrus. On Randomness in Agentic Evals (February 2026).
Anthropic. Quantifying infrastructure noise in agentic coding evals (February 2026).
Merrill et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (January 2026).
Amplifying. State of the Coding Agent Market (November 2026 edition observed November 2, 2026; October 26 snapshot).
Greptile. Rise of the Overnight Agents (May 2026).
OpenAI. SWE-bench annotation instructions.
Snorkel AI. How Senior SWE-Bench works (June 2026, with July updates).
Harbor Framework. sanitize-git-repo task directory (April 2026).
Harbor Framework. sanitize-git-repo pinned grader (test_outputs.py).
NIST CAISI. Examples of cheating in CAISI’s agent evaluations.
Scale AI et al. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark (October 2026).
OpenAI. Separating signal from noise in coding evaluations (July 2026).
Yu et al. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark (March 2026).
Yin et al. VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents (October 2026).
Zhong et al. SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? (October 2026).
Deininger and Slany. Fixing, Breaking, or Faking It?, Journal of Cybersecurity and Privacy (September 2026).
Harbor Framework. Terminal-Bench 2.0 leaderboard dataset card (observed November 2, 2026).
Khatri. Do Context Files Help Coding Agents? (July 2026).
Ding et al. Re-executing Coding-Agent Evaluations (October 2026).
Sahoo et al. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation (October 2026).
Lewis. Same Model, Different Harness: Different Coding-Agent Results (August 2026).
Pan et al. HarnessTax: How Much Does the Harness Matter for Coding Agents? (observed November 2, 2026).
Kozyrev, Kozyrev, and Podkopaev. Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents (September 2026).
Davidovich et al. Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments (August 2026).
Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (November 2024).
Liu et al. Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead (September 2026).
Vats and Golev. The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation (July 2026).
Chen et al. Emerging Trends in AI Agent Evaluation (SAgE) (observed November 2, 2026).
Hassanzadeh Aliabadi. Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents (October 2026).
Zi et al. AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans (September 2026).
Amplifying. Coding agents methodology (observed November 2, 2026).
Shi et al. CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions (ACL 2026).
Edwards et al. RExBench: Can coding agents autonomously implement AI research extensions? (ACL 2026).
Xia et al. Demystifying LLM-Based Software Engineering Agents (PACMSE, 2025).
Mündler et al. Code Agents are State of the Art Software Testers (June 2024, v1).
Duston et al. AInsteinBench: Benchmarking Coding Agents on Scientific Repositories (December 2025).
Mustahsan et al. Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation (December 2025).
Kwa et al. Measuring AI Ability to Complete Long Software Tasks (NeurIPS 2025).

Explore the Operating Autonomous Fleets runbook · Return to Architecting Intelligent Businesses