DEFT: Forming Efficient Multi-Agent Committees

The world is going multi-agent. Coding agents fan out into sub-agents that review each other’s diffs. Medical and legal assistants only surface an answer when several models concur. Eval harnesses grade responses by how many judges agree. The fastest-rising committee of all is the LLM-as-a-judge^1,2:1  Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.2  Liu et al., 2023. G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. EMNLP 2023. models grading models went from evaluation shortcut to the field’s default measuring instrument in under three years, and its newest form is a panel of diverse judges whose consensus is read as ground truth.³3  Verga et al., 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv 2024. Orchestration frameworks ship committee patterns as first-class primitives, and every serious AI product roadmap now has a box labeled “multiple models vote.” Which means agent selection, routing, collaboration, and communication are quietly becoming some of the deepest engineering problems of our time: who should be on the team, how many seats should it have, how should they talk, and when should you believe them?

Almost every deployed answer to that last question is the same one: agreement equals confidence. And the most common way to build the committee is the cheapest one: sample five completions of the same model at temperature 0.7 and take the majority vote.^4,54  Wang et al., 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.5  Li et al., 2024. More Agents Is All You Need. TMLR 2024. We measured what that construction actually buys, across 11,250 frontier-model completions in three domains, and then built the protocol the measurements demand. It is called DEFT, for Design-Effect-guided Frugal Teams: a calibrate-once recipe that decides who sits on the committee, how many seats it gets, and what its agreement is worth. It wins every head-to-head we ran. In law it scores 0.667 against the default’s 0.585 at a third of the cost. In medicine it scores 0.832 against 0.817 at a ninth of the cost. In science, 0.891 against 0.867 at a fifth. And it usually does this while buying fewer seats, not more.

DEFT is deliberately boring. You measure two numbers per candidate model on about 100 calibration questions, and from there arithmetic makes every decision. No prompt engineering, no debate rounds, no learned router. This post explains why something that simple beats current practice, and the answer is not that DEFT is clever. The answer is that the default is badly mispriced: multi-agent committees everywhere are running head-count math on voters who are secretly the same voter. Any protocol that prices the redundancy correctly starts the race ahead.

Your committee is mispriced because its votes are photocopies

Agreement-as-confidence rests on a theorem with a load-bearing word in it. If five agents with 80% accuracy err independently, the chance they unanimously converge on the same wrong answer is about one in three hundred thousand. That is Condorcet’s jury theorem, from 1785,⁶6  Condorcet, 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Paris, 1785. and it is the reason “two models concurred” feels like evidence. If the agents’ errors are correlated, the same arithmetic is fiction.

Survey statisticians solved the corrupted version of this problem seventy years ago. Poll five people from one household and you did not collect five opinions. Kish (1965)⁷7  Kish, 1965. Survey Sampling. John Wiley & Sons. called the fix the design effect, and it converts N correlated voters into the number of independent voters they are actually worth:

N_eff = N / (1 + (N − 1) · ρ)

Here ρ is the pairwise error correlation: how often two models are wrong on the same questions. ρ = 0 is a true second opinion. ρ = 1 is a photocopy. The formula sanity-checks at both extremes: at ρ = 0 you get N_eff = N, and at ρ = 1 you get N_eff = 1, five copies of one voter.

Correlation itself is no longer news to researchers. Kim, Garg, Peng & Garg document substantial error correlation across 350+ models, rising with accuracy and shared provider,⁸8  Kim, Garg, Peng & Garg, 2025. Correlated Errors in Large Language Models. ICML 2025. and Kohli prices a fixed panel of nine frontier judges at roughly two effective votes.⁹9  Kohli, 2026. Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels. arXiv 2026. But almost nobody deploying committees measures ρ, and nobody had priced the construction practitioners actually ship — temperature seeds of a single model — or watched how the answer moves across diversity regimes and knowledge domains. So we measured it, at scale: 8 frontier models, 750 MMLU-Pro questions¹⁰10  Wang et al., 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. NeurIPS 2024. across science, medicine, and law (about ten answer options per question, which keeps lucky-guess collisions rare), and three committee designs with five seats each. Design one is the industry default, five seeds of DeepSeek V4 at temperature 0.7. Design two is five models from five different Chinese labs (DeepSeek, Kimi, Qwen, GLM, ByteDance Seed) at temperature 0. Design three crosses cultures: DeepSeek, Kimi, Claude, GPT-5-mini, and Gemini, also at temperature 0. After filtering to questions where all five members produced a definite answer, 9,549 responses survive, and every ρ estimate carries a 500-iteration bootstrap confidence interval.

Two facts about committees follow immediately. First, temperature is not a personality transplant. Seeds phrase differently, sometimes even reason differently, and then fail on the same questions anyway. Second, “just add more agents” runs out almost instantly: at ρ ≈ 0.5–0.65, the eighth seat adds roughly 0.02 effective agents while still casting a full, correlated vote that can tip a plurality onto a shared wrong answer. More seats can actively make the committee worse, and in our enumeration they do.

The protocol

DEFT is the design-effect formula run in reverse. Instead of diagnosing a committee you already have, it decides who to hire, when to stop hiring, and how much to trust the room once it votes.

CALIBRATE  (once, ~100 items, K·|D_cal| model calls)
    for each candidate model m:   p̂_m  ← accuracy on D_cal
    for each pair (i, j):         ρ̂_ij ← corr. of error indicators

FLOOR      Q ← models within δ = 0.10 of the best p̂
SIZE       N* ← grow N while the marginal hire adds ≥ ε = 0.10
           effective agents at the pool’s measured ρ̂ (Kish)
FILL       S ← the N* cheapest members of Q within budget
PRICE      per item: k of |S| agree → k_eff = k · N_eff/|S| →
           corrected posterior q; accept iff q ≥ τ, else escalate

Everything after calibration is pure arithmetic. It re-runs per domain for free, it is deterministic given the calibration set, and it is monotone in its constraints: shrinking the budget or the seat cap never increases cost. The interesting question is why these four moves and not others, and this is where the measurements earn their keep. Every line of DEFT is licensed by a measured failure of something greedier we tried first.

FLOOR, because minimizing ρ hires incompetence

The obvious protocol, and the first one we tried, goes: if correlation is the disease, select the roster that minimizes ρ. This is a trap, and it is the most instructive negative result in the paper. Because all eight models answered all 750 questions at temperature 0, every subset of the pool is a scoreable committee, so we enumerated all 219 of them. Within every domain, committee ρ̄ correlates positively with plurality accuracy (Spearman +0.13 to +0.36). Chasing low ρ recruits bad models.

The mechanism first showed up as an ablation surprise: drop the weakest model from a committee and the measured ρ rises by 0.09–0.18. A weak agent’s scattered errors look statistically independent, because it disagrees with everyone, including when everyone is right. We call this the competence confound, and it is why DEFT consults accuracy first and ρ only afterwards. The floor removes the models whose “diversity” is really just noise, before the diversity math is allowed to run.

SIZE, because the marginal hire adds a fiftieth of an expert

With the floor applied, how many seats should you buy? The enumeration gives a blunt answer: small committees saturate. The best size-3 committee matches or beats the best committee of any size in every domain, and the full 8-model pool is strictly worse (in law, 0.715 for the best trio versus 0.699 for all eight). That is exactly what the design effect predicts, so DEFT lets the measured ρ̂ draw the staircase and stops climbing when the next step gets too shallow:

Walk medicine’s staircase by hand and the economics get visceral. Seat one is a full expert: N_eff = 1.00. Seat two brings the committee to 1.21. Seat three: 1.31. Seat four: 1.36. Seat eight: 1.45. The second hire brings a fifth of an expert, the eighth brings a fiftieth, and every one of them costs full price and casts a full, correlated vote. Hiring correlated models is buying the same newspaper twice: ρ̂ measures how much they are the same newspaper, N_eff counts the distinct newspapers you actually hold, and ε says stop buying when the next copy is 90% repeat.

FILL, because rosters don’t transfer

Floor set, size chosen: which specific models get the seats? Here is the negative result that killed our fancier ideas. Nothing about roster identity survives domain shift. Across our three domains, the accuracy rankings of the 56 size-5 committees correlate at Spearman −0.46 to +0.38, and the ρ̄ rankings at −0.54 to +0.19. Even the correlation structure is domain-specific. There is no “generally diverse” dream team waiting to be discovered; a roster lovingly tuned on one domain is noise on the next.

So DEFT refuses to roster-optimize at all. Any model that survived the floor is good enough, and the remaining degree of freedom goes to the only thing that reliably transfers across domains: the invoice. Seat the cheapest qualifying models and stop. This is where the 3–9× savings come from, and it is a feature born directly from a failure: we spent the roster-search budget because the data proved roster search is fitting noise.

PRICE, because naive consensus overpays by 23–75 points

The committee is hired. Now, what is its agreement worth? This is the question production systems answer implicitly every time they ship an answer because k models concurred, and it is the question with the most expensive wrong answer. For every item, take the largest same-answer cluster among the five agents, call its size k, and compare what independence-assuming Condorcet math predicts against what actually happens:

This is why PRICE exists and why its last word is “escalate.” In a deployed committee, the corrected posterior, not the vote count, should decide whether an answer ships or goes to a human. And the correction is free at inference time: it reuses the same ρ̂ the SIZE step already bought during calibration.

The scoreboard, and one sentence to steal

DEFT wins every domain at 3–9× lower cost, usually with fewer, better, cheaper seats. Read the medicine row again: the protocol looked at the measured correlation, decided a third seat wasn’t worth hiring, and beat the five-seat default with a committee of two at a ninth of the price. And buried in the law column is the cleanest sentence we can offer a practitioner: five temperature-0.7 seeds of one model score 0.585, below the 0.611 of a single temperature-0 call to the same model.

Temperature sampling plus voting is not ensembling. When a model’s errors are systematic, sampling five noisy copies and voting launders one model’s biases into false confidence, at five times the price.

What this means if you’re building multi-agent systems

The findings compress into rules you can apply this week:

• Measure ρ before you scale N. One hundred calibration questions per domain is enough. If you aren’t measuring error correlation, your committee size is a guess and your confidence signal is a fiction.
• Stop defaulting to five seeds of one model. In our hardest domain that construction lost to a single deterministic call. If the budget only allows one model family, one temperature-0 call plus an escalation rule beats a seed jury.
• Small committees are not a compromise, they are the optimum. At realistic correlations the second and third seats carry almost all the evidence you will ever get. Spend the saved tokens on better members or on escalation.
• Never trust the raw vote count. Discount agreement by N_eff/N before it touches a confidence threshold, an auto-ship rule, or a user-facing probability. The correction is one multiplication.
• Re-calibrate per domain. Rosters, accuracy ranks, and even the correlation structure are domain-specific. The protocol transfers; its inputs do not.

Where this sits in the literature

The premise — LLM committees are correlated — is by now established territory. Kim et al. documented it pairwise across 350+ models;⁸ Kohli converted it into an effective-size statement for a fixed panel of nine distinct judges, and showed that cleverer aggregation recovers at most 11% of the lost information — the bottleneck is the panel, not the vote-counting.⁹ DEFT starts where that diagnosis ends. If the panel is the bottleneck, the question worth solving is which panel to buy.

Two papers get closest to that question. Turkmen, Buyukates & Bastopcu select a size-k committee from equally priced candidates by greedy mutual information, and prove that under correlation, picking the top-k models by accuracy is not optimal.¹¹11  Turkmen, Buyukates & Bastopcu, 2026. Don’t Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection. arXiv 2026. DEFT solves the priced version of their problem: seats cost dollars, not counts, replicas of cheap models are legal moves, the output is a calibrated posterior rather than a subset, and the correlation structure that drives selection gets a cross-domain transfer test instead of a within-dataset one. Chen bounds what any routing, voting, or mixture policy can gain via the co-failure rate β — the fraction of queries where every model in the pool is wrong — and proves that pairwise ρ cannot identify that floor.¹²12  Chen, 2026. When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models. arXiv 2026. Since DEFT is ρ-centric, that critique deserves a direct answer: β caps the ceiling, but it does not determine what intermediate agreement is worth — P(correct | 3 of 5 agree) depends on the joint error structure below the all-wrong cell, which is exactly what PRICE calibrates. And on multiple-choice, Chen’s own measurements put β near zero: the regime where ρ-based pricing is valid. We will report β alongside ρ for every cell in v2 regardless.

The seeds-versus-diversity question is genuinely contested. Self-MoA showed that sampling only the single best model can beat mixing different LLMs at equal sample counts;¹³13  Li, Lin, Xia & Jin, 2025. Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? ICML 2025. Chen’s matched-quality comparison finds the reverse once member quality is equalized.¹² We think the deployment-relevant axis is neither counts nor quality but dollars, and the design effect explains both results inside one framework: a seed committee is cheap per seat and worth ~1.3 votes, a diverse committee is expensive per seat and worth ~1.8, and which one wins depends on the price of the seats — which is what FILL computes.

What remains ours after that accounting: the measured effective size of the temperature-seed committee, the culture-regime contrast (prior work slices diversity by provider family, not lab region), the priced committee knapsack with replicas, the calibrated posterior as the protocol’s output, and the cross-domain transfer test.

Where this goes next

Version 2 is already scoped, and it is aimed at making the claims bigger, not safer. The head-to-head gets a held-out replication on GPQA-Diamond, so the scoreboard graduates from demonstration to deployment guidance. An all-Western committee cell and a non-English variant will pin down one of the most provocative results in the data: that in US common law, an all-Chinese-lab committee out-diversifies a culturally mixed one, suggesting diversity failure is a property of domains organized around a single culture’s framework. And the pricing formula gets an upgrade: the Kish scaling is the crudest correction that could possibly work, and a Dawid-Skene-style aggregator¹⁴14  Dawid & Skene, 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics 28(1). with an explicit correlation term should sharpen it at low agreement levels, where the money is. Two robustness columns join the tables in v2: the co-failure floor β for every committee cell, and tetrachoric correlation estimates alongside the Pearson ones.

The open questions we would most love the community to attack:

• Is ρ rising with every model generation? Frontier models increasingly train on each other’s outputs and the same synthetic pipelines. If the independence collapse is deepening, the staircases get flatter every year, committees get smaller, and correlation becomes the scaling bottleneck for collective AI. Phylogenies reconstructed from output similarity already show how densely the model family tree is intertwined.¹⁵15  Yax, Oudeyer & Palminteri, 2024. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks. ICLR 2025.
• Can reasoning traces substitute for seats? Models that agree on answers may disagree on reasoning paths. If trace-level disagreement carries independent evidence, it is a diversity source that costs no extra seats.
• What is the ρ-aware version of debate, routing, and delegation? We priced the simplest collaboration primitive, the vote. Every richer primitive in the multi-agent stack has the same unmeasured correlation sitting inside it.

Coda

The multi-agent era is being built on a borrowed intuition: that many minds are safer than one. The intuition is right, and the accounting is wrong. Independence is not a default you get by calling the API five times; it is a scarce, measurable, domain-specific resource, and today almost every committee in production is spending it like it’s free. DEFT is one formula pointed at that mistake: measure ρ, let it size the room, spend the savings on better seats, and discount the applause by the number of distinct voters in it. The teams that internalize this will run cheaper, smaller, sharper committees than the teams that count heads. The vote count was never the evidence. The independence was.

@misc{katte2026deft,
  author = {Katte, Abhijeet and Ghosh, Basab},
  title  = {DEFT: Forming Efficient Multi-Agent Committees},
  year   = {2026},
  month  = {July},
  url    = {https://www.cmul8.com/research/deft/}
}