The Blame Game: Model vs. Harness

Who actually made the move: the model, the prompt, the Python tool, or the runtime?

The benchmark score names the model, but the move may have come from the harness.

This post reports five pre-registered game-playing experiments. Each froze its hypothesis, seed set, and decision rule before launch, and later experiments followed what earlier audits surfaced. Together they show how a scaffold effect can be measured, localized to prompt text, stress-tested on a stronger model, and probed with an intervention. The durable contribution is harness forensics: a practical audit layer for agent systems.

Who moved the score

The scaffold around a model can swing its score. The same GPT-4 ranged from 2.7% to 28.3% on SWE-bench Verified depending on the scaffold it ran in.¹1 OpenAI, 2024. Introducing SWE-bench Verified: same model, four scaffolds. The same Claude Opus 4.5 scored 77.8% under one scaffold and 42.2% under another on CORE-Bench Hard.²2 Kapoor et al. (Princeton PLI), 2025. Holistic Agent Leaderboard: CORE-Bench Hard leaderboard data. Controlled work in coding has measured scaffold swings of 27 percentage points with the model held fixed, and a growing literature now argues that agent scores should be reported as model–scaffold configurations.

All of that evidence lives at the outcome level: a final number, compared across conditions. Outcome-level numbers cannot say which layer moved the score or why, because software-engineering environments have no per-decision ground truth. The spread stays a black box.

So we chose an environment where the black box opens. In games with a perfect-play solver, every decision carries exact ground truth. That makes decomposition possible: the scaffold effect can be measured per move, linked to a mechanism, and checked with interventions. The measurements below are scoped to this environment. The audit method is the part that transfers to noisier environments.

One terminology split matters throughout. The harness is the fixed runtime software that drives the agent: it feeds observations, executes tools, records moves. The scaffold is the model-facing package that varies per experimental arm: prompt text, tool access, execution rules, parsing, retries, timeouts, runtime permissions. Our question, fixed up front: when an agent’s behavior changes, which layer changed it?

Here is the audit method in miniature. For each arm, write down allowed tools; audit actual tool calls; inspect prompt text for policy prescriptions; inspect executed code for what the tool computed; prove paired environments were identical; guard against parse and contamination failures; score decisions where ground truth exists; track resource failures; rerun sensitivity exclusions; then put every public claim in a ledger. The ten controls below exist because one of these experiments needed it.

The laboratory

We picked two games that stress different skills so a scaffold effect had room to flip sign. Both games ran on DeepSeek v4 Flash, and the key ConnectFour result was repeated on DeepSeek v4 Pro.

The games

Minesweeper tests constraint bookkeeping. One player, no opponent; the model sees a partially revealed board and replies with one cell. Mines are placed after the first click, so pairing across arms forces the same first click.

ConnectFour tests adversarial depth. Two players drop discs; four in a row wins. Going first is a large advantage, so the two arms play head-to-head in seat-swapped pairs.

The arms

An arm is one experimental condition: a model plus scaffold combination being measured. raw is an answer-only prompt arm. repl is the same model wrapped in a scaffold: a prompt plus a Python execute tool to call before committing a move. Both arms ran inside the same real harness, with actual tool use audited from traces rather than assumed from labels (the payoff arrives in The tool surface).

What we measured

Minesweeper reports the share of boards solved and the mine-hit rate per move.

ConnectFour reports per-move regret from the Pascal Pons exact solver. The solver’s raw position score can be positive (the current player can force a win in N plies), 0 (a draw with perfect play), or negative (the current player is losing). Regret is different: it is the gap between the best available position score and the score after the move actually played (both from the mover’s perspective). Regret is always ≥ 0 — 0 for a perfect move, larger for a worse move. Blunder rate is the share of moves with regret above 0. A game yields one win/loss/draw; its roughly twelve plies yield about six scored decisions per player, so regret captures move-quality shifts that the final outcome can hide.

Because the two arms are opponents, the positions one arm creates are the positions the other arm faces, so their regrets are coupled. The paired bootstrap intervals respect this pairing without treating the two arms as independent draws.

The tests

Hypotheses and decision rules were fixed before the evidence runs:

H1 is evaluated with a McNemar test (p<0.05 on paired seeds, counting only pairs where the arms disagree). H2 is evaluated with a paired bootstrap confidence interval that excludes zero. Higher regret means worse play. Sign changes were declared interesting-not-failure in advance, and the failure condition was the registered test missing its threshold.

The protocol

Every experiment shared the same controls: pre-registration, paired seeds, seat swaps that cancel first-player advantage, solver regret instead of bare win/loss, a contamination guard that drops parse failures and illegal moves before scoring, prompt and executed-code audits, an arm-purity audit of actual tool_calls, and a sensitivity analysis that recomputes every headline claim after excluding contaminated games. Only claims that survive get published. Arm labels are claims to audit.

Pilot runs only debugged the machinery and never counted as evidence.

Prior work sits in three threads, all cited at the end: outcome-level score-spread evidence, harness-disclosure advocacy with model×harness configuration benchmarks, and a thin game-domain thread of modular harness ablations plus solver-scored game evaluation. We found no prior work combining decision-level scaffold decomposition, mechanism localization through executed-code forensics, a pre-registered scaffold intervention, and an arm-purity audit against the runtime’s exposed tool surface. Reference 8 reports 35 scaffold versions with a fixed coding model and no meaningful solve-rate change. Our setting differs because per-move regret exposes quality shifts that solve rates hide.

EXPT1: the scaffold helps

What this section shows: the same model plus a Python tool solves more Minesweeper boards, and the gain survives a purity check.

The first experiment asked the simplest version of the question: on truly identical minefields, does the Python-tool scaffold beat answer-only play?

The repl scaffold helped the model avoid mines when the safe move was knowable. Repl solved 86 boards against raw’s 60 [+26 percentage points, McNemar, paired]. The result survives the purity check: dropping the two contaminated seed pairs gives 86/98 for repl against 60/98 for raw [+26.5 percentage points, p < .001]. Mine hits where logic was available fell from 2.9% to 1.3% per move, while forced guesses stayed roughly equal.

Minesweeper rewards bookkeeping. Executed code helps because the scaffold can track constraints more reliably than a pure text answer.

EXPT2: the scaffold hurts

What this section shows: the same scaffold architecture flips sign on ConnectFour, and the harm shows up as move-level regret concentrated in winning positions.

The second experiment ran the mirror image: same model, same scaffold architecture (prompt plus Python execution before each move) on ConnectFour, where the skill is adversarial depth.

The repl scaffold blundered less often but catastrophically, and the catastrophe concentrated in positions it was winning. Raw won 126 decided games to repl’s 72 [126–72, seat-swapped pairs]. Per-arm mean regret was 2.348 for raw against 3.248 for repl, and the headline paired diff was +1.065 [95% CI [+0.777, +1.357], excludes zero]. A win-rate gap alone could be dismissed as variance. Regret adds a move-level check against perfect-play ground truth. Raw made more non-perfect moves overall [blunder rate 41.6% vs 35.6%], but repl’s bad tail was much worse [90th-percentile regret 19 vs 7].

That unconditioned rate is misleading: the positions a player faces are produced by both players jointly, and a losing player has little room to blunder. Conditioning on the solver’s evaluation of the position reverses the rate: in winning positions repl blundered more often [45.3% vs 37.7%], a scale-free comparison that position mix cannot explain. Winning positions also offer the most regret headroom, so part of the amplification is mechanical:

Position mix matters too: repl spent more of its moves in winning positions than raw did [66.2% vs 51.7% of scored moves (the shares sum past 100% because the arms are opponents)], and still lost 126–72. Of the 126 games repl lost, it held a winning position after the opening (ply ≥ 4) in 61.1%, while raw held one in 44.4% of its 72 losses. That signature matches a one-move policy: it misses the positions where a mistake is fatal. The raw arm was purity-clean [0/1406 raw turns].

Minesweeper is bookkeeping. ConnectFour is adversarial depth. A one-move routine can help one and damage the other.

The supported claim is narrower than “scaffold effects flip by task.” A scaffold whose prompt prescribed bookkeeping helped on Minesweeper. A scaffold whose prompt prescribed an unsuitable one-move policy hurt on ConnectFour. Any benchmark that reports a single tool-use number across task types risks averaging over prompt-task fit.

EXPT3: prompt mechanism

What this section shows: the repl arm’s executed code was a shallow one-move policy, and the prompt prescribed it.

Two results with opposite signs demanded a mechanism, and the audit protocol localizes in a fixed order: score → moves → executed code → prompt text.

The code

The executed code held the first clue. The strict template detector found immediate win → immediate block → center fallback code on 199/1,380 repl turns (14.4%). The rest were unclassified variants [misc: 1,029 of 1,380, 74.6%], execution errors [145, 10.5%], and one turn of heuristic evaluation without search, while explicit minimax or depth search appeared on only 6/1,380 turns (0.43%).

Can I win immediately? Can the opponent win immediately? Otherwise prefer center columns.

That is a one-move policy family. It misses two-move traps, fork threats, and tempo: precisely the skills ConnectFour exists to test. The strongest forensic claim is that deep search appeared on only 6/1,380 turns (0.43%).

The prompt

The second pass went one layer upstream, to prompt text. The ConnectFour repl scaffold was built with an explicit prescriptive prompt, a common harness pattern: the prompt offered Python and dictated the algorithm.

You MUST use the `execute` tool for every move.
In the tool, write a short Python snippet that:
1. Parses the ConnectFour board from the observation.
2. Checks for an immediate winning move, then an immediate block of the opponent's win.
3. Otherwise prefers center columns.
4. Prints the chosen column as [c].

The model was often executing the scaffold’s instructions, and those instructions encoded the shallow policy. The audit points to the prescriptive prompt as the damaging component.

The contrast case makes it sharper: the Minesweeper repl prompt prescribes bookkeeping (“tracks revealed cells and known constraints… identifies safe cells”), and it helped. Full prompt texts side by side: prompt excerpts.

The measurement stands: the repl scaffold made the model worse at ConnectFour. The attribution is narrower: the damaging component was one paragraph of prescriptive prompt text. Scaffolds must be audited as behavioral systems. The finding also creates a direct test: if the paragraph carries the policy, swapping the paragraph should move the outcome.

EXPT4: the scaffolded ceiling

What this section shows: the prescriptive prompt capped a stronger model near the weaker model’s scaffolded ceiling.

The mechanism suggests a stress test: if the prescriptive prompt caps both models near the same shallow policy, the stronger model has more baseline performance to lose. We pre-registered a follow-up on DeepSeek v4 Pro, a substantially stronger model, to check that pattern.

The prescriptive prompt capped the stronger model near the weaker model’s scaffolded level. Both prescriptive repl arms had median regret 0 and nearly identical 90th-percentile tails, while the unaided raw arms differed sharply by model strength.

The Pro paired diff was +3.002 [95% CI [+2.723, +3.302], excludes zero], about 2.8 times the Flash paired diff. Computed from per-arm means, the scaffold’s share of the Pro repl error:

The exact 69% figure uses the recorded DeepSeek v4 Pro raw reference, which had raw-arm bash leakage (full story in The tool surface, below). That leak makes raw look better, so it likely inflates the scaffold-share estimate. The sensitivity checks preserve the main paired comparison: excluding raw-contaminated games keeps the repl arm much worse [paired diff +2.950, 95% CI [+2.640, +3.260]].

Position conditioning localizes the ceiling further. The harm concentrates in winning positions, and the effect survives the forfeit check.

The balanced bucket flips sign between EXPT2 and EXPT4; only the winning-position concentration generalizes.

Part of the winning-position amplification is mechanical, since winning positions offer the most regret headroom, but the effect survives the forfeit check: the 13 forfeit-free pairs still show +5.858 [95% CI [+2.994, +9.509]]. The two models fail through different routes: on Flash, repl reached winning positions and threw them (held a post-opening winning position in 61.1% of its losses); on Pro, repl rarely developed one at all (18.2% of its losses). What generalizes is the winning-position bucket: under the prescriptive prompt, whatever regret an arm accrues concentrates in positions it was winning. One prompt, two failure modes.

A reader counting only wins would have missed this: raw’s record was 99 wins against 89 losses, 87 of the losses to its own stalls — raw lost only two games on the board — while the regret gaps rank the two experiments in the opposite order [+3.002 vs +1.065]. The outcome record was corrupted by a harness failure. The move-level scoring survived it.

This is a caveated but important claim: prompt-scaffold behavior can dominate the apparent model score in this setup. The prescriptive prompt put the weak and strong models in the same band, with Pro slightly worse under the scaffold despite being far better raw.

A forensic chain (executed code → prescriptive prompt) is still an inference. The stronger check is an intervention: if the prompt paragraph is load-bearing, replacing it with a depth-search paragraph should move the outcome. We pre-registered that comparison.

EXPT5: prompt intervention

What this section shows: swapping one prompt paragraph for a depth-search version reversed the sign in the same run.

The final experiment held everything identical (DeepSeek v4 Pro, ConnectFour, the same Python execute tool, the same audited protocol, seat-swapped pairs) except one paragraph of prompt text:

• Prescriptive prompt (the repl arm from EXPT4): check immediate win → block immediate win → prefer center. A one-move policy.
• Depth prompt (the new arm): search the game tree at least three plies deep, evaluate leaves with a simple heuristic. A search policy.

The prescriptive arm’s result was already on the books from EXPT4: mean regret 3.536, with the caveated 69% scaffold-share calculation above. The depth-prompted arm completed with 100 games [96 parse-clean, 50 seat-swapped pairs]. The prescriptive and depth arms were not run concurrently, so that cross-run comparison is suggestive. The within-run depth-versus-raw comparison is the defensible headline.

Same model, same tool, same game: the depth prompt beat raw by about 60% on scored moves. The tables below put the comparison in one place. The EXPT4 prescriptive column is a cross-run reference because the two prompts were not run concurrently.

The position split closes the mechanism loop: the depth prompt’s advantage is largest in the same winning-position bucket where the prescriptive prompt did its damage.

The first table strips the sample one exclusion at a time. The second table conditions on the solver’s evaluation of the position faced (a position-level slice, not a pair-level exclusion):

Poison and cure, same bucket, same units. Stated with attrition visible: the depth prompt beat raw by 1.198 regret on 39 clean pairs, 37 of which include a game raw lost to a prompt stall, while the two forfeit-free pairs point the same way (−2.866) but are too few to carry the claim alone.

Blunder rate (moves with regret above zero) tells the same story [37.5% raw → 21.2% depth-prompted repl], and the bad tail also shrank [90th-percentile regret 4 → 2].

The tool surface

What this section shows: answer-only arms still used the runtime’s bash tool, and the stronger model leaked more.

The finding most likely to transfer came from a design choice and a planned audit.

Every arm, including the answer-only raw arm, ran inside the same real harness. We skipped a separately sandboxed no-tool baseline for two reasons. Comparability: identical runtime conditions across arms. Realism: published baselines run inside tool-bearing harnesses everywhere. That harness exposes a native bash tool underneath the prompt. The prompt said “just answer.” The runtime said “bash is available.”

Our protocol never trusts an arm’s label. Every run gets an arm-purity check, an allowed-tool matrix compared against actual tool_calls. A sandboxed baseline would have hidden what this audit was built to catch. DeepSeek v4 Pro sometimes sided with the runtime.

A no-tool prompt is not a no-tool runtime. DeepSeek v4 Pro’s raw arm used native bash on 3.2–6.3% of turns [35/1088 turns in the EXPT4 replication, and 38/603 bash-leak turns plus one bomb in EXPT5], writing and running real minimax solvers, then quoting their output. DeepSeek v4 Flash used unintended tools far less often [typically 0–1.1%, and its ConnectFour raw arm was perfectly clean at 0/1406]. The leak rate was higher on the stronger model, consistent with a capability gradient but too little to establish one alone.

The finding is that an instruction in a prompt competes with the affordances of the runtime. As models get stronger, prompt-level restrictions may become a weaker control, and runtime-level isolation becomes the safer one.

Arm labels are intentions. Every agent eval needs an allowed-tool matrix and an actual-tool-use audit, because the runtime may expose more than the prompt describes.

Agent evals need arm-purity checks. Any “model-only baseline” that runs inside a tool-bearing harness is an assumption until the traces prove otherwise.

Harness forensics

The reusable contribution is the audit layer. Each item below was load-bearing in this program:

1. Allowed-tool matrix — what each arm is supposed to access.
2. Actual tool-call audit — what each arm actually used, from traces.
3. Prompt audit — whether prompt text prescribes policy, not just format.
4. Executed-code inspection — what the tool code actually computed.
5. Environment-pairing proof — whether paired seeds really produce identical conditions.
6. Parse/contamination guard — whether recorded moves faithfully reconstruct the game.
7. Decision-level ground truth — solver regret where available, not only wins.
8. Resource/process audit — timeouts, zombie processes, memory explosions.
9. Sensitivity analysis — exclude contaminated units and show which claims survive.
10. Claim ledger — separate confirmed, directional, caveated, and pending claims.

None of these are exotic. The reporting layer needs the same discipline: an earlier draft of this post cited an EXPT4 forfeit split (16 of 83 pairs) that existed nowhere in the analysis pipeline, and the crosscheck caught it because every number in the post must exist in the canonical metrics. The named failure modes these audits surfaced (rubber stamps, one-ply theater, trusted error output, unbounded-search bombs) live in the failure museum. The full checklist, with the motivating case for each item, is in the harness forensics checklist. Pre-registrations, seed manifests, prompt texts, and audit tables ship with the same repository.

Claim ledger

Machine-readable version with evidence links: claim ledger.

Limitations

The measurements are scoped to this environment: two games (Minesweeper and ConnectFour) and two models (DeepSeek v4 Flash and DeepSeek v4 Pro). We did not run a permissive-prompt arm, so we cannot separate “prescribing a strategy is harmful” from “this prescribed strategy was harmful,” and that missing arm is also the test that would clarify why reference 8 found little scaffold effect. We did not cross prompts between games. The prescriptive and depth arms were not run concurrently, and the Pro raw baseline moved between EXPT4 and EXPT5. The raw reference has bash leakage, which makes raw look better and likely inflates the 69% scaffold-share estimate, though sensitivity checks preserve the main paired comparisons. EXPT5 is forfeit-heavy: after exclusions, only 2 no-forfeit pairs remain. The leak pattern uses two models from one family, so it is consistent with a capability gradient rather than a measured trend. What transfers beyond this environment is the audit method.

What we would buy next, by value per dollar: extend the leak audit across more models, since it is cheap and transferable. Then a permissive-prompt ConnectFour arm on the weaker model. Then, if budget allows, rerun prescriptive and depth prompts concurrently with matched timeout budgets.

What we do claim:

In this audited setup, scaffold behavior was large enough to change task outcomes, dominate a strong model’s measured errors under a prescriptive prompt, and reveal prompt boundaries leaking into exposed harness tool surfaces.

Coda

Stop asking only which model scored? Ask: which layer made the move?

The model may have made it. Or the prompt. Or the Python tool. Or the parser. Or a runtime tool the prompt never described. In our ConnectFour runs, the honest answer was “one paragraph of prompt text”, and no leaderboard would have told you that.

That is the blame game: model vs. harness. The contribution is a way to open the agent stack and see where performance comes from. Once the audit exists, the findings keep coming, because the harness was dirtier than its labels claimed.

References

Score spreads and scaffold effects (outcome level).

1. OpenAI. Introducing SWE-bench Verified (2024): GPT-4 ranged 2.7% → 28.3% across scaffolds, same model. openai.com/index/introducing-swe-bench-verified/
2. Kapoor et al. Holistic Agent Leaderboard (HAL). arXiv:2510.11977: model–scaffold–benchmark reporting; CORE-Bench Hard: same Opus 4.5, 77.8% vs 42.2% across scaffolds.
3. Zheng et al. Claw-SWE-Bench. arXiv:2606.12344: 27.4pp Pass@1 swing with model fixed (coding).
4. Vats, Golev. The Scaffold Effect in Coding Agents. arXiv:2607.22585: harness choice as hidden variable; 40× token-efficiency spread.
5. Just a Wrapper? How Much Do Scaffolds Matter? (LessWrong, HAL data): scaffold explains more price-performance variation than model choice.

Harness disclosure and configuration reporting.

6. Zhang et al. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950.
7. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows. arXiv:2605.27922.
8. Don’t Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality. arXiv:2607.03691: 35 scaffold versions, fixed model, no significant solve-rate change (coding).

Game-domain harness work (thin).

9. Zhang, Yu, Hu, Jin, Zhang. General Modular Harness for LLM Agents in Multi-Turn Gaming Environments. ICML MAS Workshop 2025. arXiv:2507.11633: module ablations in games, one backbone; closest prior work.
10. Lou et al. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arXiv:2603.03329: harness synthesis in games, not controlled ablation.

Solver ground truth in games (separate thread).

11. Schultz et al. Mastering Board Games by External and Internal Planning with Language Models. ICML 2025. arXiv:2412.12119: ConnectFour per-move solver regret (Fhourstones); a planning study, not harness attribution.
12. CAST: Game Solvers as Turn-Level Teachers for LLM Agents. arXiv:2607.25308: turn-level solver signals on Minesweeper.

Platforms.

13. Guertler et al. TextArena. arXiv:2504.11442: game platform used in our runs.
14. Paglieri et al. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv:2411.13543.

About the author

Sagar Sarkale

Cite this post

@misc{sarkale2026blamegame,
  author = {Sarkale, Sagar},
  title  = {The Blame Game: Model vs. Harness},
  year   = {2026},
  month  = {August},
  url    = {https://www.cmul8.com/research/the-blame-game/}
}