When Five Agents Behave Like One

The world is going multi-agent. Coding agents fan out into sub-agents that review each other’s diffs. Medical and legal assistants only surface an answer when several models concur. Eval harnesses grade responses by how many judges agree. The fastest-rising committee of all is the LLM-as-a-judge^1,2:1  Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.2  Liu et al., 2023. G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. EMNLP 2023. models grading models went from evaluation shortcut to the field’s default measuring instrument in under three years, and its newest form is a panel of diverse judges whose consensus is read as ground truth.³3  Verga et al., 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv 2024. Orchestration frameworks ship committee patterns as first-class primitives, and every serious AI product roadmap now has a box labeled “multiple models vote.” Which means agent selection, routing, collaboration, and communication are quietly becoming some of the deepest engineering problems of our time: who should be on the team, how many seats should it have, how should they talk, and when should you believe them?

Almost every deployed answer to that last question is the same one: agreement equals confidence. And the most common way to build the committee is the cheapest one: sample five completions of the same model at temperature 0.7 and take the majority vote.^4,54  Wang et al., 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.5  Li et al., 2024. More Agents Is All You Need. TMLR 2024. We measured what that construction actually buys: 11,250 frontier-model completions on MMLU-Pro across law, medicine, and science, plus 1,980 more on GPQA-Diamond as a harder, contamination-controlled replication. The answer is the title of this post. Five temperature seeds of one model are worth 1.24–1.37 effective independent voters; even the most diverse roster money can buy stops at 1.77–1.81. That measurement is the work’s primary contribution. A corollary falls out of taking it seriously: a small calibrate-once heuristic — DEFT, for Design-Effect-guided Frugal Teams — that turns the measured correlation into a committee. Held out, it is always cheaper than the five-seed default, and its accuracy gains track the domain: +10.8 points in law and +14.4 on GPQA-Diamond, +1.7 in medicine, −1.1 in science. And in the domain where the default is weakest, the default loses to a single deterministic call to its own base model.

The recipe at the end of this post is deliberately boring: two numbers per candidate model on about 100 calibration questions, then arithmetic. No prompt engineering, no debate rounds, no learned router. That is the point. The interesting object is not the recipe but the mispricing the measurement exposes: multi-agent committees everywhere are running head-count math on voters who are secretly the same voter. Anything that prices the redundancy correctly starts the race ahead.

Your committee is mispriced because its votes are photocopies

Agreement-as-confidence rests on a theorem with a load-bearing word in it. If five agents err independently, unanimous convergence on the same wrong answer is essentially impossible — the naive posterior in our tables reads 100.0% at unanimity. That is Condorcet’s jury theorem, from 1785,⁶6  Condorcet, 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Paris, 1785. and it is the reason “two models concurred” feels like evidence. If the agents’ errors are correlated, the same arithmetic is fiction.

Survey statisticians solved the corrupted version of this problem seventy years ago. Poll five people from one household and you did not collect five opinions. Kish (1965)⁷7  Kish, 1965. Survey Sampling. John Wiley & Sons. called the fix the design effect, and it converts N correlated voters into the number of independent voters they are actually worth:

N_eff = N / (1 + (N − 1) · ρ)

Here ρ is the pairwise error correlation: how often two models are wrong on the same questions. ρ = 0 is a true second opinion. ρ = 1 is a photocopy. The formula sanity-checks at both extremes: at ρ = 0 you get N_eff = N, and at ρ = 1 you get N_eff = 1, five copies of one voter.

Correlation itself is no longer news to researchers. Kim, Garg, Peng & Garg document substantial error correlation across 350+ models, rising with accuracy and shared provider,⁸8  Kim, Garg, Peng & Garg, 2025. Correlated Errors in Large Language Models. ICML 2025. and Kohli prices a fixed panel of nine frontier judges at roughly two effective votes.⁹9  Kohli, 2026. Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels. arXiv 2026. But almost nobody deploying committees measures ρ, and nobody had priced the construction practitioners actually ship — temperature seeds of a single model — or watched how the answer moves across diversity regimes and knowledge domains. So we measured it, at scale: 8 frontier models, 750 MMLU-Pro questions¹⁰10  Wang et al., 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. NeurIPS 2024. across science, medicine, and law (about ten answer options per question, which keeps lucky-guess collisions rare), and three committee designs with five seats each. Design one (D1) is the industry default, five seeds of DeepSeek V4 at temperature 0.7. Design two (D2) is one model each from five different Chinese labs (DeepSeek V4, Kimi K2, Qwen 3 235B, GLM 4.6, ByteDance Seed 1.6) at temperature 0. Design three (D3) crosses cultures: DeepSeek V4, Kimi K2, Claude Sonnet 4.6, GPT-5-mini, and Gemini 2.5 Pro, also at temperature 0. And because MMLU-Pro predates every one of these models’ training cutoffs, the two headline conditions get a replication on 198 GPQA-Diamond items,¹¹11  Rein et al., 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv 2023. a harder benchmark built with stronger contamination controls. After filtering to questions where all five members produced a definite answer, 9,549 MMLU-Pro responses survive. Every ρ estimate carries a 500-iteration item-clustered bootstrap confidence interval, and every cell also reports the co-failure rate β, the fraction of items where all agents fail simultaneously, with an exact Clopper–Pearson interval.¹²12  Chen, 2026. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models. arXiv 2026.

Two facts about committees follow immediately. First, temperature is not a personality transplant. Seeds phrase differently, sometimes even reason differently, and then fail on the same questions anyway. Second, “just add more agents” runs out almost instantly: at ρ ≈ 0.5–0.65, the eighth seat adds roughly 0.02 effective agents while still casting a full, correlated vote that can tip a plurality onto a shared wrong answer. More seats can actively make the committee worse, and in our enumeration they do.

The collapse is not an artifact of MMLU-Pro. On GPQA-Diamond — harder questions, stricter contamination controls — five DeepSeek V4 seeds measure ρ = 0.769 and N_eff = 1.227, and the cross-family committee measures ρ = 0.439 and N_eff = 1.813: materially less correlated, still nowhere near five.

Why seats saturate

How fast do extra seats stop paying? Run the measured correlations through the Kish formula and the answer is the study’s central picture:

Walk medicine’s staircase by hand and the economics get visceral. Seat one is a full expert: N_eff = 1.00. Seat two brings the committee to 1.23. Seat three: 1.33. Seat four: 1.38. Seat eight: 1.48. The second hire brings a quarter of an expert, the eighth brings two hundredths of one, and every one of them costs full price and casts a full, correlated vote. Hiring correlated models is buying the same newspaper twice: ρ̂ measures how much they are the same newspaper, and N_eff counts the distinct newspapers you actually hold.

What agreement is actually worth

Now the expensive question. Production systems ship answers because k models concurred; what is that concurrence worth? For every item, take the largest same-answer cluster among the five agents and call its size k. The sharpest test sits at the bottom of the scale, k = 2 — a bare plurality, two agents against three scattered dissenters — so here is every committee cell in the study at exactly that agreement level, naive prediction against measured reality:

Nor is unanimity a safe harbor. Observed accuracy under unanimous five-way agreement is 86–99%, and every complete-five cohort we studied, on both benchmarks, contains at least one unanimously wrong item; in law the error rate under unanimity is roughly 14%. So in a deployed committee, the corrected posterior, not the vote count, should decide whether an answer ships or goes to a human — with two honest caveats. The k_eff substitution over-corrects at k = 1–2; a principled fix needs a beta-binomial or copula model of correlated votes. And a system with no explicit confidence threshold anywhere in it gains nothing from the discount: the correction matters exactly where agreement feeds an auto-ship rule, a probability shown to a user, or an escalation gate.

Structured missingness

Nearly every committee paper quietly conditions on this measurement without reporting it. Models abstain or return unparseable answers on 5.3–31.8% of MMLU-Pro items, and the abstentions are not random. Same-model seeds fail to parse the same items together (ρ_miss = 0.36–0.46) while cross-family abstentions are quasi-independent (0.00–0.19). On GPQA-Diamond the problem is dramatic: DeepSeek V4 returned a parseable answer on 45.8% of items and GLM 4.6 on 34.8%. Treating abstentions as errors would conflate two different correlations, so all primary numbers here use the strict cohort where all five members answered, and missingness is reported separately — an accounting choice we think any honest ρ analysis has to make.

Temperature sampling is not ensembling

The finding that indicts the default construction directly: in law, the five-seed committee scores 0.585, below the 0.611 of a single temperature-0 call to the same model. Temperature 0.7 introduced errors faster than majority voting could absorb them — the errors are correlated (ρ = 0.755) while the added noise is not independent — so the vote launders one model’s systematic mistakes into false confidence. In our hardest domain the pattern is not just overconfident; it actively degrades accuracy at five times the inference cost. Medicine and science stay above their single-call baselines, where accuracy headroom leaves more independent error signal for the vote to absorb — the domain-dependence lesson in miniature.

Temperature sampling plus voting is not ensembling. When a model’s errors are systematic, sampling five noisy copies and voting launders one model’s biases into false confidence, at five times the price.

The corollary: DEFT

Taking the measurement seriously yields a recipe, and we are deliberate about what it is not: not a mathematically optimal algorithm, but a structured checklist for pruning redundant API calls — the design-effect formula run in reverse. Calibrated once per domain, applied at inference time with no further measurement, it decides who to hire, how many, and how much to trust the room:

CALIBRATE  (once, ~100 items, K·|D_cal| model calls)
    for each candidate model m:   p̂_m  ← accuracy on D_cal
    for each pair (i, j):         ρ̂_ij ← corr. of error indicators

FLOOR      F ← models within δ = 0.10 of the best p̂
SELECT     enumerate every non-empty subset S ⊆ F (≤ 31 of them),
           score each on D_cal (no new API calls);
           keep the most accurate, ties broken by N_eff
PRICE      per item: k of |S*| agree → k_eff = k · N_eff/|S*| →
           corrected posterior q; ship iff q ≥ τ, else escalate

Each line is licensed by a measured failure of something cleverer. The accuracy floor comes first because minimizing ρ outright is a trap: drop the weakest model from a committee and the measured ρ̄ rises by 0.09–0.18 — a weak agent’s scattered errors fake independence because it disagrees with everyone, including when everyone is right. We call this the competence confound, and the pool contains a live demonstration: GLM 4.6, at 66.5% roughly 26 points below the best model, violates the floor, and excluding it raises measured ρ in law and science.

Selection is exhaustive because greedy rules lose. N_eff is not monotone in composition — in law the marginal seat adds +0.33, +0.04, then +0.16 effective agents, and a threshold rule that stops at the first shallow step misses the best committee, a quartet at 0.791 against the greedy stop at 0.763. The floor leaves at most five candidates, hence at most 31 committees, every one scoreable on calibration responses already collected — zero new API calls. A matched-size ablation splits the credit: committee size contributes +5.7 points of the +6.7-point law advantage over always-taking-the-two-cheapest; member identity contributes +1.0. N_eff tells you how many seats to buy, not which models to seat.

And there is no dream roster to learn, because rosters don’t transfer: cross-domain accuracy and ρ̄ rankings correlate at r = −0.54 to +0.19. The honest comparisons are the cheaper ones. Filling DEFT’s committee size with the cheapest floor-qualifying models — no ρ computed at all — trails by just +1.0 point in law, +0.1 in medicine, +0.9 in science; the best single model matches or exceeds DEFT in medicine and science at 57–78% fewer tokens, and in law trails by 1.2 points at half the token cost. Price the one-model committee first. DEFT is compression of the five-seed default people actually deploy, not a universal accuracy winner — and at inference time it simply reuses Figure 4’s correction, discounting the vote count by N_eff/|S| before anything ships or escalates.

The scoreboard

Strict held-out evaluation, 10 seeds × 5 folds. “Five seeds” is the deployed default, five temperature-0.7 runs of DeepSeek V4 with majority voting. “Top-5” is a curated committee of the five most accurate models. “Random-N” matches DEFT’s committee size with random floor-qualified members.

Read it honestly, because we did. Against the deployed default, DEFT wins every row — +12.0 points in law, +14.4 on GPQA-Diamond — usually with two or three seats instead of five (on GPQA it selects the same frugal pair, ByteDance Seed 1.6 + Qwen3-235B, in all 50 folds). Against a curated top-accuracy five-model committee it is a wash: −0.9 in law, −0.8 in science, +0.8 in medicine, at a fraction of the calls. Held-out validation over 20 random calibration splits draws the boundary of the claim precisely: law +10.8 ± 0.6 points with a bootstrap CI excluding zero, medicine +1.7 ± 0.4 (19 of 20 seeds), science −1.1 ± 0.4 — null to negative, because a domain sitting at 98% accuracy leaves a 50-item calibration set no signal to work with. DEFT is a compression heuristic with a validity domain, not a universal accuracy winner.

What this means if you’re building multi-agent systems

The findings compress into rules you can apply this week:

• Measure ρ before you scale N. One hundred calibration questions per domain is enough — and in our data, committee composition only stabilizes at ≥100 items, so don’t skimp. If you aren’t measuring error correlation, your committee size is a guess and your confidence signal is a fiction.
• Stop defaulting to five seeds of one model. In our hardest domain that construction lost to a single deterministic call. If the budget only allows one model family, one temperature-0 call plus an escalation rule beats a seed jury.
• Price the one-model committee first. The best single model matched or beat our full protocol in two of three domains at 57–78% fewer tokens. Inspect the size-1 and size-2 options before you commit to a multi-model configuration.
• Small committees are not a compromise, they are the optimum. At realistic correlations the second and third seats carry almost all the evidence you will ever get. Spend the saved tokens on better members or on escalation.
• Never trust the raw vote count. Discount agreement by N_eff/N before it touches a confidence threshold, an auto-ship rule, or a user-facing probability. The correction is one multiplication — and if your system has no explicit threshold anywhere, the vote count was never evidence to begin with.
• Don’t calibrate against a ceiling. In a domain where your best model already scores ≈98%, a small calibration set carries no signal and committee optimization is noise. Our own science results are null to negative; that is the boundary of the method, stated plainly.
• Report missingness separately from error. Abstentions correlate too — strongly for same-model seeds, barely for cross-family rosters. Folding them into the error rate corrupts both measurements.
• Re-calibrate per domain. Rosters, accuracy ranks, and even the correlation structure are domain-specific. The protocol transfers; its inputs do not.

Where this sits in the literature

The premise — LLM committees are correlated — is by now established territory. Kim et al. documented it pairwise across 350+ models, and showed the correlation rises with capability;⁸ Goel et al. showed model mistakes grow more alike as models get better, which undermines AI oversight schemes built on cross-checking;¹³13  Goel et al., 2025. Great Models Think Alike and this Undermines AI Oversight. ICML 2025. Ding found agreement is a positive but weak predictor of correctness on GPQA-Diamond and AIME, and especially unreliable for frontier models — their item-clustered bootstrap methodology informed ours;¹⁴14  Ding, 2026. When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals. arXiv 2026. and Norman et al. showed at scale that raw agreement metrics systematically overstate independence in LLM-as-a-judge pipelines.¹⁵15  Norman, Rivera & Hughes, 2026. Reliability Without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. arXiv 2026. Kohli converted the diagnosis into an effective-size statement for a fixed panel of nine distinct judges, and showed that cleverer aggregation recovers at most 11% of the lost information — the bottleneck is the panel, not the vote-counting.⁹ DEFT starts where that diagnosis ends. If the panel is the bottleneck, the question worth solving is which panel to buy.

Two papers get closest to that question, and we engage both with experiments rather than arguments. Turkmen, Buyukates & Bastopcu select a size-k committee by greedy mutual information, and prove that under correlation, picking the top-k models by accuracy is not optimal.¹⁶16  Turkmen, Buyukates & Bastopcu, 2026. Don’t Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection. arXiv 2026. We ran their selection head-to-head against DEFT over twenty calibration splits, same floor, shared test pool. The verdict cuts both ways: DEFT is marginally more accurate (under 1 point on average) but consistently more expensive — their mutual-information objective more often picks cheaper models at comparable accuracy, so for a cost-constrained practitioner it is the more efficient choice, and DEFT’s advantage is interpretability as a design-effect audit. Chen bounds what any routing, voting, or mixture policy can gain via the co-failure rate β, and proves that pairwise ρ cannot identify that floor for committees of three or more.¹² That critique deserved data, not a rebuttal, so every cell in our tables now reports β directly, with exact Clopper–Pearson intervals. On our multiple-choice data β is small — 0.01 to 0.16 — the regime where ρ-based pricing is valid; β still caps the ceiling, and P(correct | 3 of 5 agree) depends on the joint structure below that ceiling, which is exactly what the corrected posterior calibrates.

The seeds-versus-diversity question is genuinely contested. Self-MoA showed that sampling only the single best model can beat mixing different LLMs at equal sample counts;¹⁷17  Li, Lin, Xia & Jin, 2025. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? ICML 2025. Chen’s matched-quality comparison finds the reverse once member quality is equalized.¹² We think the deployment-relevant axis is neither counts nor quality but dollars, and the design effect explains both results inside one framework: a seed committee is cheap per seat and worth ~1.3 votes, a diverse committee is expensive per seat and worth ~1.8, and which one wins depends on the price of the seats — which is what the frugal baselines make explicit.

What remains ours after that accounting: the measured effective size of the temperature-seed construction, replicated across two benchmarks; the culture-regime contrast, now honestly powered and flagged exploratory; the sized-and-priced committee selection with a calibrated posterior as its output; the cross-domain transfer test; and the structured-missingness accounting.

What this doesn’t show

A measurement paper earns trust by drawing its own boundaries, so here are ours. Filtering to items where all five members answered biases the retained set toward easier questions — accuracy on it runs 15–30 points above the full sweep — which compresses the correlation coefficient anti-conservatively; the GPQA-Diamond replication on a gated benchmark is the mitigation, and the 1,701 filtered rows ship alongside the full data. Science’s same-model cell sits at 97.4% accuracy, which leaves so few errors to correlate that its ρ interval spans [0.17, 0.92]; we mark it unreliable everywhere it appears. All committee conditions fix N = 5, and exhaustive SELECT is only cheap because the floor caps the pool at five candidates; larger pools need something smarter. The domains are three English multiple-choice slices plus one English MCQ benchmark — no claim here extends to open-ended tasks or other languages. And we checked our own estimator choices: tetrachoric correlations did not improve selection over plain Pearson φ (−2.8 points in law, −1.9 in medicine), and committee composition only stabilizes with ≥100 calibration items in law, remaining unstable in medicine and science — tight-margin domains want a pre-registered held-out set.

The data is yours

All 11,250 MMLU-Pro completions and 1,980 GPQA-Diamond responses are packaged with the analysis code (ρ estimation, bootstrap CIs, DEFT selection, 176 tests) under MIT and CC-BY-4.0 licences with SHA-256 checksums. Cost is reported throughout as measured output tokens per item, because dollar prices depend on provider routing; the GPQA-Diamond run additionally ships per-attempt billing metadata. The repository accompanies the paper version of this work, currently under peer review, and goes public at acceptance.

Where this goes next

Three follow-ups would buy the most new information. A D4 all-Western committee cell and a non-English variant to settle the law reversal, which currently needs about six times more data to detect reliably. An N-ablation past five seats, where ΔN_eff is provably non-monotone and exhaustive search stops being free. And the pricing upgrade: the Kish scaling is the crudest correction that could possibly work and it over-corrects at low agreement, so a Dawid-Skene-style aggregator¹⁸18  Dawid & Skene, 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics 28(1). with an explicit correlation term is the principled successor, sharpest exactly at the low agreement levels where the money is.

The open questions we would most love the community to attack:

• Is ρ rising with every model generation? Frontier models increasingly train on each other’s outputs and the same synthetic pipelines — shared training-data coverage and shared alignment recipes are the likely mechanism behind the correlations we measure, an instance of algorithmic monoculture.^19,2019  Kleinberg & Raghavan, 2021. Algorithmic Monoculture and Social Welfare. PNAS 118(22).20  Bommasani et al., 2022. Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization? NeurIPS 2022. If the independence collapse is deepening, the staircases get flatter every year, committees get smaller, and correlation becomes the scaling bottleneck for collective AI. Phylogenies reconstructed from output similarity already show how densely the model family tree is intertwined.²¹21  Yax, Oudeyer & Palminteri, 2024. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks. ICLR 2025.
• Can reasoning traces substitute for seats? Models that agree on answers may disagree on reasoning paths. If trace-level disagreement carries independent evidence, it is a diversity source that costs no extra seats.
• What is the ρ-aware version of debate, routing, and delegation? We priced the simplest collaboration primitive, the vote. Every richer primitive in the multi-agent stack has the same unmeasured correlation sitting inside it.

Coda

The multi-agent era is being built on a borrowed intuition: that many minds are safer than one. The intuition is right, and the accounting is wrong. Independence is not a default you get by calling the API five times; it is a scarce, measurable, domain-specific resource, and today almost every committee in production is spending it like it’s free. The design effect is one formula pointed at that mistake: measure ρ, let it size the room, spend the savings on better seats, and discount the applause by the number of distinct voters in it. The teams that internalize this will run cheaper, smaller, sharper committees than the teams that count heads. The vote count was never the evidence. The independence was.

@misc{katte2026fiveagents,
  author = {Katte, Abhijeet and Ghosh, Basab and Jain, Arjun},
  title  = {When Five Agents Behave Like One: Measuring Effective
            Committee Size in Multi-Agent LLM Systems},
  year   = {2026},
  month  = {July},
  url    = {https://www.cmul8.com/research/when-five-agents-behave-like-one/}
}