The Blame Game: Model vs. Harness

Who actually made the move: the model, the prompt, the Python tool, or the runtime?

Read this as a detective story. An AI agent made moves in games. Some moves were good, some bad. The mystery is not only which model made them, but which layer caused them.

The benchmark score names the model, but the move may have come from the harness.

This post has two layers. If you want the story, read the bold claims, figures, and “what to remember” boxes. If you want the audit, read the tables, details blocks, linked artifacts, and claim ledger. Both layers tell the same story at different resolutions.

This is not a prompt-engineering story. It is agent forensics: tracing a score back through the move, code, prompt, tool surface, and runtime that produced it.

This post reports five pre-registered game-playing experiments. Each froze its hypothesis, seed set, and decision rule before launch, and later experiments followed what earlier audits surfaced. Together they show how a scaffold effect can be measured, localized to prompt text, stress-tested on a stronger model, and probed with an intervention. The durable contribution is harness forensics: a practical audit layer for agent systems.

Who moved the score

The scaffold around a model can swing its score. The same GPT-4 ranged from 2.7% to 28.3% on SWE-bench Verified depending on the scaffold it ran in.¹1 OpenAI, 2024. Introducing SWE-bench Verified: same model, four scaffolds. The same Claude Opus 4.5 scored 77.8% under one scaffold and 42.2% under another on CORE-Bench Hard.²2 Kapoor et al. (Princeton PLI), 2025. Holistic Agent Leaderboard: CORE-Bench Hard leaderboard data. Controlled work in coding has measured scaffold swings of 27 percentage points with the model held fixed, and a growing literature now argues that agent scores should be reported as model–scaffold configurations.

All of that evidence lives at the outcome level: a final number, compared across conditions. Outcome-level numbers cannot say which layer moved the score or why, because software-engineering environments have no per-decision ground truth. The spread stays a black box.

So we chose an environment where the black box opens. In games with a perfect-play solver, every decision carries exact ground truth. That makes decomposition possible: the scaffold effect can be measured per move, linked to a mechanism, and checked with interventions. The measurements below are scoped to this environment. The audit method is the part that transfers to noisier environments.

Cast of characters

• Model - the system producing text.
• Prompt - the instructions shaping what the model tries.
• Tool - code the agent can run before answering.
• Parser - the layer that turns text into a legal game move.
• Runtime - the process that exposes tools, permissions, timeouts, and logs.
• Harness - the fixed machinery that feeds observations, executes tools, records moves, and scores runs.
• Scaffold - the model-facing package that changes between arms: prompt text, tool access, execution rules, parsing, retries, timeouts, and permissions.

Most benchmark tables compress all of this into one model name. This post tries to uncompress it.

The key split is harness versus scaffold. The harness is the fixed runtime software that drives every arm. The scaffold is the model-facing package that varies by condition. Our question, fixed up front: when an agent’s behavior changes, which layer changed it?

Here is the audit method in miniature. For each arm, write down allowed tools; audit actual tool calls; inspect prompt text for policy prescriptions; inspect executed code for what the tool computed; prove paired environments were identical; guard against parse and contamination failures; score decisions where ground truth exists; track resource failures; rerun sensitivity exclusions; then put every public claim in a ledger. The ten controls below exist because one of these experiments needed it.

The laboratory

We picked two games that stress different skills so a scaffold effect had room to flip sign. Both games ran on DeepSeek v4 Flash, and the key ConnectFour result was repeated on DeepSeek v4 Pro.

The games

Minesweeper tests constraint bookkeeping. One player, no opponent; the model sees a partially revealed board and replies with one cell. Mines are placed after the first click, so pairing across arms forces the same first click.

ConnectFour tests adversarial depth. Two players drop discs; four in a row wins. Going first is a large advantage, so the two arms play head-to-head in seat-swapped pairs.

The arms

An arm is one experimental condition: a model plus scaffold combination being measured. raw is an answer-only prompt arm. repl is the same model wrapped in a scaffold: a prompt plus a Python execute tool to call before committing a move. Both arms ran inside the same real harness, with actual tool use audited from traces rather than assumed from labels (the payoff arrives in The tool surface).

What we measured

Minesweeper reports the share of boards solved and the mine-hit rate per move.

ConnectFour reports per-move regret from the Pascal Pons exact solver. The solver’s raw position score can be positive (the current player can force a win in N plies), 0 (a draw with perfect play), or negative (the current player is losing). Regret is different: it is the gap between the best available position score and the score after the move actually played (both from the mover’s perspective). Regret is never negative: 0 for a perfect move, larger for a worse move. Blunder rate is the share of moves with regret above 0.

Example: if the perfect move keeps a forced win, but the played move turns that position into a draw, regret is the size of that drop. If the played move is also perfect, regret is 0. Lower is better.

A game yields one win/loss/draw; its roughly twelve plies yield about six scored decisions per player, so regret captures move-quality shifts that the final outcome can hide.

Because the two arms are opponents, the positions one arm creates are the positions the other arm faces, so their regrets are coupled. A paired diff compares matched seeds or seat-swapped games rather than pretending the arms faced independent worlds. The paired bootstrap intervals respect that pairing.

The tests

Hypotheses and decision rules were fixed before the evidence runs:

H1 is evaluated with a McNemar test (p<0.05 on paired seeds, counting only pairs where the arms disagree). H2 is evaluated with a paired bootstrap confidence interval that excludes zero. Higher regret means worse play. Sign changes were declared interesting-not-failure in advance, and the failure condition was the registered test missing its threshold.

The protocol

Every experiment shared the same controls: pre-registration, paired seeds, seat swaps that cancel first-player advantage, solver regret instead of bare win/loss, a contamination guard that drops parse failures and illegal moves before scoring, prompt and executed-code audits, an arm-purity audit of actual tool_calls, and a sensitivity analysis that recomputes every headline claim after excluding contaminated games. Only claims that survive get published. Arm labels are claims to audit.

Pilot runs only debugged the machinery and never counted as evidence.

Prior work already shows that scaffolds can move outcome scores, and the references at the end cover those threads. What was missing was a way to decompose the effect per decision: inspect the move, the code, the prompt, and the runtime surface together. Reference 8 found little scaffold effect across coding scaffolds; our setting differs because solver-scored games expose move quality even when final solve rates would hide it.

EXPT1: the scaffold helps

What this section shows: the same model plus a Python tool solves more Minesweeper boards, and the gain survives a purity check.

The first scene looked simple: give the agent Python, see if it helps. In Minesweeper, it did. The tool acted like a bookkeeping aid.

The first experiment asked the simplest version of the question: on truly identical minefields, does the Python-tool scaffold beat answer-only play?

The repl scaffold helped the model avoid mines when the safe move was knowable. Repl solved 86 boards against raw’s 60 [+26 percentage points, McNemar, paired]. The result survives the purity check: dropping the two contaminated seed pairs gives 86/98 for repl against 60/98 for raw [+26.5 percentage points, p < .001]. Mine hits where logic was available fell from 2.9% to 1.3% per move, while forced guesses stayed roughly equal.

Minesweeper rewards bookkeeping. Executed code helps because the scaffold can track constraints more reliably than a pure text answer.

What to remember: Python helped when the task was bookkeeping. That does not mean Python helps every task.

EXPT2: the scaffold hurts

What this section shows: the same scaffold architecture flips sign on ConnectFour, and the harm shows up as move-level regret concentrated in winning positions.

Then the same kind of scaffold walked into a different room. ConnectFour does not reward bookkeeping alone; it punishes shallow plans.

Reminder: raw means answer-only; repl means prompt plus Python execution before the move.

The second experiment ran the mirror image: same model, same scaffold architecture (prompt plus Python execution before each move) on ConnectFour, where the skill is adversarial depth.

The short version: repl made fewer small mistakes in some places, but worse mistakes where mistakes mattered most.

The repl scaffold blundered less often but catastrophically, and the catastrophe concentrated in positions it was winning. Raw won 126 decided games to repl’s 72 across seat-swapped pairs. Mean regret was 2.348 for raw against 3.248 for repl; the headline paired diff was +1.065 [95% CI [+0.777, +1.357], excludes zero]. A win-rate gap alone could be dismissed as variance. Regret adds a move-level check against perfect-play ground truth. Raw made more non-perfect moves overall, but repl’s bad tail was much worse (90th-percentile regret 19 vs 7).

The overall blunder rate is misleading because a losing position often has little left to ruin, while a winning position can still be thrown away. So the cleaner comparison is inside position buckets. There, the reversal appears: in winning positions repl blundered more often [45.3% vs 37.7%]. Winning positions also offer the most regret headroom, so part of the amplification is mechanical:

Position mix matters too: repl spent more of its moves in winning positions than raw did [66.2% vs 51.7% of scored moves (the shares sum past 100% because the arms are opponents)], and still lost 126–72. Of the 126 games repl lost, it held a winning position after the opening (ply ≥ 4) in 61.1%, while raw held one in 44.4% of its 72 losses. That signature matches a one-move policy: it misses the positions where a mistake is fatal. The raw arm was purity-clean [0/1406 raw turns].

Minesweeper is bookkeeping. ConnectFour is adversarial depth. A one-move routine can help one and damage the other.

The supported claim is narrower than “scaffold effects flip by task.” A scaffold whose prompt prescribed bookkeeping helped on Minesweeper. A scaffold whose prompt prescribed an unsuitable one-move policy hurt on ConnectFour. Any benchmark that reports a single tool-use number across task types risks averaging over prompt-task fit.

What to remember: ConnectFour punished shallow one-move logic. Win/loss showed harm; per-move regret showed where harm came from.

EXPT3: prompt mechanism

What this section shows: the repl arm’s executed code was a shallow one-move policy, and the prompt prescribed it.

This is where outcome reporting stops and forensics starts. The score said repl was worse. The traces let us ask why.

Two results with opposite signs demanded a mechanism, and the audit protocol localizes in a fixed order: score → moves → executed code → prompt text.

The code

The executed code held the first clue. The strict template detector found immediate win → immediate block → center fallback code on 199/1,380 repl turns (14.4%). The rest were unclassified variants [misc: 1,029 of 1,380, 74.6%], execution errors [145, 10.5%], and one turn of heuristic evaluation without search, while explicit minimax or depth search appeared on only 6/1,380 turns (0.43%).

Can I win immediately? Can the opponent win immediately? Otherwise prefer center columns.

That is a one-move policy family. It misses two-move traps, fork threats, and tempo: precisely the skills ConnectFour exists to test. The strongest forensic claim is that deep search appeared on only 6/1,380 turns (0.43%).

The prompt

The second pass went one layer upstream, to prompt text. The ConnectFour repl scaffold was built with an explicit prescriptive prompt, a common harness pattern: the prompt offered Python and dictated the algorithm.

You MUST use the `execute` tool for every move.
In the tool, write a short Python snippet that:
1. Parses the ConnectFour board from the observation.
2. Checks for an immediate winning move, then an immediate block of the opponent's win.
3. Otherwise prefers center columns.
4. Prints the chosen column as [c].

The model was not independently inventing this bad strategy. The prompt handed it the strategy.

The model was often executing the scaffold’s instructions, and those instructions encoded the shallow policy. The audit points to the prescriptive prompt as the damaging component.

The contrast case makes it sharper: the Minesweeper repl prompt prescribes bookkeeping (“tracks revealed cells and known constraints… identifies safe cells”), and it helped. Full prompt texts side by side: prompt excerpts.

The measurement stands: the repl scaffold made the model worse at ConnectFour. The attribution is narrower: the damaging component was one paragraph of prescriptive prompt text. Scaffolds must be audited as behavioral systems. The finding also creates a direct test: if the paragraph carries the policy, swapping the paragraph should move the outcome.

What to remember: tool use was not magic. The scaffold mostly executed the strategy the prompt prescribed, and that strategy was wrong for ConnectFour.

EXPT4: the scaffolded ceiling

What this section shows: the prescriptive prompt capped a stronger model near the weaker model’s scaffolded ceiling.

If a bad prompt replaces the model’s own planning, a stronger model has more ability to lose. That makes the stress test simple: keep the bad paragraph, upgrade the model, and see whether the damage grows. We pre-registered that follow-up on DeepSeek v4 Pro.

In practical terms, the prompt did not merely hurt a weak model. It overrode much of what made the stronger model strong.

The prescriptive prompt capped the stronger model near the weaker model’s scaffolded level. Both prescriptive repl arms had median regret 0 and nearly identical 90th-percentile tails, while the unaided raw arms differed sharply by model strength.

The Pro paired diff was +3.002 [95% CI [+2.723, +3.302], excludes zero], about 2.8 times the Flash paired diff. Computed from per-arm means, roughly 69% of Pro repl’s measured error appeared only under the scaffold: 3.536 mean regret under repl against 1.112 under raw.

The exact 69% figure uses the recorded DeepSeek v4 Pro raw reference, which had raw-arm bash leakage (full story in The tool surface, below). That leak makes raw look better, so it likely inflates the scaffold-share estimate. The sensitivity checks preserve the main paired comparison: excluding raw-contaminated games keeps the repl arm much worse [paired diff +2.950, 95% CI [+2.640, +3.260]].

The same winning-position pattern reappears, and the effect survives the forfeit check.

The balanced bucket flips sign between EXPT2 and EXPT4; only the winning-position concentration generalizes.

Part of the winning-position amplification is mechanical, since winning positions offer the most regret headroom, but the effect survives the forfeit check: the 13 forfeit-free pairs still show +2.437. The two models fail through different routes: on Flash, repl reached winning positions and threw them (held a post-opening winning position in 61.1% of its losses); on Pro, repl rarely developed one at all (18.2% of its losses). One prompt, two failure modes.

A reader counting only wins would have missed this because EXPT4 included many raw prompt stalls. The outcome record was corrupted by a harness failure; move-level scoring survived it.

This is a caveated but important claim: prompt-scaffold behavior can dominate the apparent model score in this setup. The prescriptive prompt put the weak and strong models in the same band, with Pro slightly worse under the scaffold despite being far better raw.

A forensic chain (executed code → prescriptive prompt) is still an inference. The stronger check is an intervention: if the prompt paragraph is load-bearing, replacing it with a depth-search paragraph should move the outcome. We pre-registered that comparison.

What to remember: a bad scaffold can erase much of a stronger model’s advantage. The published score would blame the model, but the traces blame the prompt.

EXPT5: prompt intervention

What this section shows: swapping one prompt paragraph for a depth-search version reversed the sign in the same run.

The forensic chain was still an inference. So the final test changed the suspected weapon: same model, same game, same Python tool, different paragraph.

The final experiment held everything identical (DeepSeek v4 Pro, ConnectFour, the same Python execute tool, the same audited protocol, seat-swapped pairs) except one paragraph of prompt text:

• Prescriptive prompt (the repl arm from EXPT4): check immediate win → block immediate win → prefer center. A one-move policy.
• Depth prompt (the new arm): search the game tree at least three plies deep, evaluate leaves with a simple heuristic. A search policy.

The prescriptive arm’s result was already on the books from EXPT4: mean regret 3.536, with the caveated 69% scaffold-share calculation above. The depth-prompted arm completed with 100 games [96 parse-clean, 50 seat-swapped pairs]. The prescriptive and depth arms were not run concurrently, so that cross-run comparison is suggestive. The within-run depth-versus-raw comparison is the defensible headline.

In short: one paragraph was both poison and cure. It mattered because it specified a policy, not because wording magic fixed the model. The shallow paragraph hurt; the search paragraph helped.

Same model, same tool, same game: the depth prompt cut mean regret by about 60% on scored moves. The tables below put the comparison in one place. The EXPT4 prescriptive column is a cross-run reference because the two prompts were not run concurrently.

The position split closes the mechanism loop: the depth prompt’s advantage is largest in the same bucket where the prescriptive prompt did its damage.

The first table strips the sample one exclusion at a time. The second table conditions on the solver’s evaluation of the position faced (a position-level slice, not a pair-level exclusion):

Poison and cure, same bucket, same units. The caveat is attrition: the clean comparison still favors the depth prompt, but most clean pairs include a raw prompt-stall forfeit; the two forfeit-free pairs point the same way but are too few to carry the claim alone.

Blunder rate (moves with regret above zero) tells the same story [37.5% raw → 21.2% depth-prompted repl], and the bad tail also shrank [90th-percentile regret 4 → 2].

What to remember: changing one paragraph changed the sign of the scaffold effect. That is why prompt text is part of the evaluated system, not a footnote.

The tool surface

What this section shows: answer-only arms still used the runtime’s bash tool, and the stronger model leaked more.

This is the finding most likely to travel beyond games. A prompt can say “do not use tools,” while the runtime still quietly offers one.

It came from a design choice and a planned audit.

Every arm, including the answer-only raw arm, ran inside the same real harness. We skipped a separately sandboxed no-tool baseline for two reasons. Comparability: identical runtime conditions across arms. Realism: published baselines run inside tool-bearing harnesses everywhere. That harness exposes a native bash tool underneath the prompt. The prompt said “just answer.” The runtime said “bash is available.”

Our protocol never trusts an arm’s label. Every run gets an arm-purity check, an allowed-tool matrix compared against actual tool_calls. A sandboxed baseline would have hidden what this audit was built to catch. DeepSeek v4 Pro sometimes sided with the runtime.

A no-tool prompt is not a no-tool runtime. “Please do not use tools” is a request. Removing the tool is a control.

DeepSeek v4 Pro’s raw arm used native bash on 3.2–6.3% of turns [35/1088 turns in the EXPT4 replication, and 38/603 bash-leak turns plus one bomb in EXPT5], writing and running real minimax solvers, then quoting their output. DeepSeek v4 Flash used unintended tools far less often [typically 0–1.1%, and its ConnectFour raw arm was perfectly clean at 0/1406]. The leak rate was higher on the stronger model, consistent with a capability gradient but too little to establish one alone.

The finding is that an instruction in a prompt competes with the affordances of the runtime. As models get stronger, prompt-level restrictions may become a weaker control, and runtime-level isolation becomes the safer one.

Arm labels are intentions. Every agent eval needs an allowed-tool matrix and an actual-tool-use audit, because the runtime may expose more than the prompt describes.

Agent evals need arm-purity checks. Any “model-only baseline” that runs inside a tool-bearing harness is an assumption until the traces prove otherwise.

What to remember: arm labels are intentions. Tool traces are evidence.

Harness forensics

The reusable contribution is the audit layer. Each item below was load-bearing in this program:

None of these are exotic. The reporting layer needs the same discipline: an earlier draft of this post cited an EXPT4 forfeit split (16 of 83 pairs) that existed nowhere in the analysis pipeline, and the crosscheck caught it because every number in the post must exist in the canonical metrics. The named failure modes these audits surfaced (rubber stamps, one-ply theater, trusted error output, unbounded-search bombs) live in the failure museum. The full checklist, with the motivating case for each item, is in the harness forensics checklist. Pre-registrations, seed manifests, prompt texts, and audit tables ship in the same repository.

Claim ledger

Machine-readable version with evidence links: claim ledger.

Limitations

What transfers is the audit method, not the exact numbers. The evidence here covers two games, two DeepSeek models, and one harness. We did not run the missing permissive-prompt arm, so we cannot claim that prescribing strategy is harmful in general - only that this prescribed strategy was harmful here. The prescriptive and depth prompts were not run concurrently; EXPT5 is forfeit-heavy; and the Pro raw reference leaked bash, likely inflating the 69% scaffold-share estimate. Sensitivity checks preserve the main paired comparisons.

Next best runs: leak audit across more models; permissive-prompt ConnectFour on the weaker model; then concurrent prescriptive-vs-depth prompts with matched timeout budgets.

What we do claim:

In this audited setup, scaffold behavior was large enough to change task outcomes, dominate a strong model’s measured errors under a prescriptive prompt, and reveal prompt boundaries leaking into exposed harness tool surfaces.

Final takeaway

Stop asking only which model scored. Ask: which layer made the move?

The model may have made it. Or the prompt. Or the Python tool. Or the parser. Or a runtime tool the prompt never described. In our ConnectFour runs, the honest answer was “one paragraph of prompt text”, and no leaderboard would have told you that.

That is the blame game: model vs. harness. The contribution is a way to open the agent stack and see where performance comes from. Once the audit exists, the findings keep coming, because the harness was dirtier than its labels claimed.

Code and data

Everything behind this post is open source, at github.com/CMUL8/the-blame-game.

• Code - the harness that ran the arms and the pipeline that produced every number: prompts, seed manifests, the env wrappers, the exact-solver regret scoring, the purity audit, and the clustering.
• Data - turn-level tables, one row per turn, published as a release asset. Each row carries the arm, the pairing key, the move played, its solver regret, the policy cluster of whatever code the model executed, and the raw model reply and tool calls.
• The crosscheck - scripts/pipeline/crosscheck_blog.py recomputes every headline value in this post from that data and reports any that drift. It is the control described under Harness forensics; the published state is zero mismatches.

Download the data, unzip it beside the code, and the whole chain from raw turns to the numbers above runs end to end.

Artifacts for technical readers

References

Score spreads and scaffold effects (outcome level).

1. OpenAI. Introducing SWE-bench Verified (2024): GPT-4 ranged 2.7% → 28.3% across scaffolds, same model.
2. Kapoor et al. Holistic Agent Leaderboard (HAL). arXiv:2510.11977: model-scaffold-benchmark reporting; CORE-Bench Hard: same Opus 4.5, 77.8% vs 42.2% across scaffolds.
3. Zheng et al. Claw-SWE-Bench. arXiv:2606.12344: 27.4pp Pass@1 swing with model fixed (coding).
4. Vats, Golev. The Scaffold Effect in Coding Agents. arXiv:2607.22585: harness choice as hidden variable; 40× token-efficiency spread.
5. Just a Wrapper? How Much Do Scaffolds Matter? (LessWrong, HAL data): scaffold explains more price-performance variation than model choice.

Harness disclosure and configuration reporting.

6. Zhang et al. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950.
7. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows. arXiv:2605.27922.
8. Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality. arXiv:2607.03691: 35 scaffold versions, fixed model, no significant solve-rate change (coding).

Game-domain harness work (thin).

9. Zhang, Yu, Hu, Jin, Zhang. General Modular Harness for LLM Agents in Multi-Turn Gaming Environments. ICML MAS Workshop 2025. arXiv:2507.11633: module ablations in games, one backbone; closest prior work.
10. Lou et al. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arXiv:2603.03329: harness synthesis in games, not controlled ablation.

Solver ground truth in games (separate thread).

11. Schultz et al. Mastering Board Games by External and Internal Planning with Language Models. ICML 2025. arXiv:2412.12119: ConnectFour per-move solver regret (Fhourstones); a planning study, not harness attribution.
12. CAST: Game Solvers as Turn-Level Teachers for LLM Agents. arXiv:2607.25308: turn-level solver signals on Minesweeper.

Platforms.

13. Guertler et al. TextArena. arXiv:2504.11442: game platform used in our runs.
14. Paglieri et al. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv:2411.13543.

About the author

Sagar Sarkale

Cite this post

@misc{sarkale2026blamegame,
  author = {Sarkale, Sagar},
  title  = {The Blame Game: Model vs. Harness},
  year   = {2026},
  month  = {August},
  url    = {https://www.cmul8.com/research/the-blame-game/}
}