Capping Conformity: Rating-Only Channels for Diverse LLM Search Teams

When several different language models work together on a problem a program can score, such as packing circles or finding short tours, letting them read each other's full answers makes the team worse. They copy the first plausible answer and lose the variety that made the team useful. Letting them work in complete isolation avoids the copying, but it also gives up the iterative improvement that evolutionary search provides. This proposal tests a middle channel taken from recommender systems. Agents do not exchange solutions. They exchange explicit ratings: a verifier score, a calibrated 1–5 opinion of a peer's candidate, and at most a short flag naming a violated constraint. A structural cap then guarantees that no single peer, or tight cluster of peers, ever supplies half or more of the influence an agent receives.

The cap comes from opinion dynamics. In the weighted-median model of how opinions spread over a network, an agent copies a neighbor outright once that neighbor carries more than half of its incoming weight. Groups that direct at least half their weight inward become sealed echo chambers. Our hypothesis is that the copying collapse seen in LLM teams is an instance of this mechanism. If it is, keeping every agent's largest incoming weight below one half should keep the team diverse while still letting good signals flow. Put in two sentences: diverse LLM teams fail because each agent ends up dominated by one peer's answer. Replace answer-sharing with sparse, calibrated ratings and cap any source's influence below one half, and the team should get the benefits of interaction without paying for it in lost diversity.

Motivation and Prior Work

The phenomenon motivating this work is documented in The Interaction Tax: When Communication Erases Diversity in Multi-Agent Teams. The paper runs teams of Claude 3.5 Sonnet, GPT-4o and Gemini 1.5 Flash on 11 verifier-scored optimization tasks and defines Marginal Interaction Gain (MIG): the score of an interactive team minus the score of the same agents working independently. Its main findings are these:

  • For diverse-model Debate, MIG falls to −0.078.
  • In a 2x2 factorial analysis, model diversity contributes +0.188 to performance, while synthesis contributes almost nothing.
  • When a synthesizer model is asked to merge proposals, it copies the best proposer verbatim more than 80% of the time.
  • Critique helps only when an error is local and easy to point to, as in Knapsack. It hurts when the constraint is global, as in 3AP-Free.

The paper examines only full-solution sharing and leaves low-bandwidth channels open. That open question is what we answer here.

Two other results constrain the design by ruling out "just isolate the agents." Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models benchmarks LLM heuristic search. On TSP, standalone sampling reaches a 9.64% gap, a greedy single-candidate search called (1+1)-EPS reaches 3.87%, and population-based EoH reaches 0.12%. Iterated information sharing is therefore necessary on rugged problems. Multiple-Frequencies Population-Based Training reaches a similar conclusion for reinforcement-learning hyperparameter search. Greedy Population-Based Training collapses to an interquartile-mean reward of 16,171 on Brax Humanoid at 1 billion steps, below Random Search at 17,713. Its asymmetric migration scheme recovers 23,793. Its lesson is that the direction and strength of information flow matter more than whether information flows at all. It hard-codes this with a two-tier hierarchy; we want a general rule over arbitrary communication graphs.

The mechanism draws on two sources:

The work closest to ours in spirit is CALM: Co-evolution of Algorithms and Language Model for Automatic Heuristic Design. CALM escapes stagnation by wiping out the whole population when progress stalls. It reacts after diversity is already gone and treats the pool as unstructured. Multi-Objective Evolution of Heuristic Using Large Language Model (MEoH) keeps diversity by selecting for code dissimilarity measured on abstract syntax trees, but its parents still pass full programs to the model that recombines them. We differ from both by preventing conformity at the communication layer, before it happens. We also differ from The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward (DPH-RL). That paper addresses collapse inside a single policy's training objective by replacing mode-seeking reverse-KL with mass-covering divergences; our problem is collapse between agents at inference time, with no weight updates.

Core Claim

Conformity in diverse LLM teams is governed by how concentrated each agent's incoming influence is, not by communication as such. Consider a channel with three properties: it carries only calibrated explicit ratings and localized flags; it opens a full-solution edge only on strong rated evidence; and it holds every agent's maximum incoming weight from any peer cluster below a cap $c < 1/2$. Such a channel should turn diverse-team MIG from negative to positive on the Interaction Tax suite, and it should beat both isolation and full sharing at equal query budget.

Research Program

Each agent $i$ maintains its own candidate lineage and refines it with its own model. After each round, every agent scores a random subset of peers' current best candidates. It sees only each candidate's verifier score and a short description it generates itself from a fixed probe. We then compute:

  • Calibrated opinion. Agent $i$'s raw rating of peer $j$ is z-scored against $i$'s own rating history to give a calibrated opinion $o_{ij}$.
  • Edge weight. Influence weights are $$\alpha_{ij} = \min\!\left(c,\ \mathrm{softmax}_j\big(\beta\, o_{ij} + \gamma\, s_j\big)\right).$$ Here $s_j$ is peer $j$'s normalized verifier score, $\beta$ and $\gamma$ are fixed mixing temperatures, and $c$ is the cap. Before clipping, weights are summed over clusters of peers whose candidates are near-duplicates (abstract-syntax-tree distance for code, embedding distance otherwise), so a clique of clones cannot evade the cap. Mass removed by clipping goes to a self-loop, meaning the agent keeps its own lineage.

  • Messages on the channel. What crosses an edge depends on its weight. Low-weight edges carry only the peer's score and the receiver's own rating of it. Mid-weight edges add a one-line constraint flag, such as "capacity exceeded on item 17." Only edges above a threshold transmit the full candidate. The agent samples which peer to read in proportion to $\alpha_{ij}$, so no peer is read with probability $c$ or more.

We vary three things: the cap $c$ (from 1 down to 0.2), the bandwidth level (scores only, scores plus flags, gated full solutions), and whether calibration is used. The prediction from weighted-median dynamics is a sharp drop in first-round conformity once $c$ falls below about 0.5. LLMs are not weighted-median agents, so this threshold is a falsifiable hypothesis rather than an assumption. We also predict that flags help on locally verifiable tasks such as Knapsack and that only scores help on globally constrained tasks such as 3AP-Free, matching the local/global asymmetry reported in the Interaction Tax paper.

Baselines

Independent parallel sampling. The same three models run in isolation at the same total query budget, and the team reports the best result. This is the reference point MIG is defined against, so any interaction scheme must beat it to justify itself. No exact per-task scores for this condition are reported on Lacuna, so we will re-run it ourselves with the Interaction Tax protocol of 5 seeds per condition.

Mixture-of-Agents (MoA). Independent proposers feed a synthesizer. It is the strongest existing interactive configuration in the Interaction Tax study because it was the one that avoided diversity collapse, although its synthesizer copies more than 80% of the time.

Multi-agent Debate. Full-solution sharing among the agents. Its reported diverse-team MIG of −0.078 in The Interaction Tax is the failure we aim to reverse.

Island-model evolutionary search. Separate sub-populations that occasionally migrate their best candidates, run through the llm-eps implementations of FunSearch and EoH described in Understanding the Importance of Evolutionary Search. Each island uses a different model, and the query budget is matched. This is the strongest established competitor. The EoH TSP gap of 0.12% reported there used a different setup from ours, so we will not compare against that number directly.

Ablations. The proposed channel is run without the cap ($c=1$), without calibration, and with a scores-only global leaderboard broadcast that has no graph structure. These isolate which ingredient does the work.

Fair comparison. All methods share the same models, decoding settings, verifier-call budget and wall-clock budget. Each receives the same small grid for its own temperature and population parameters.

What counts as success. Beating the baselines means positive diverse-team MIG on a majority of the 11 tasks, together with a higher mean normalized score than island-model search at equal budget, with seed-level confidence intervals that do not overlap.

Why not just…?

Why not just run the models independently and take the best? The Interaction Tax data favor isolation over Debate, but the heuristic-design benchmarks show that isolated sampling stalls far short of iterative search (9.64% versus 0.12% gap on TSP). Isolation is our zero-bandwidth endpoint. If it matches our method at equal budget, the core claim fails, and the bandwidth sweep will show exactly that.

Why not just use island-model migration, which already limits communication? This is the real threat. Migration is a coarse low-bandwidth channel. The difference is that islands copy whole candidates on a fixed schedule with no cap on how much a dominant migrant takes over. We will measure conformity under island migration directly. If islands already keep the largest incoming share below one half and match our scores, our contribution reduces to a diagnosis rather than a method, and we will report it that way.

Why not just broadcast a scalar leaderboard? This tests whether graph structure and calibration matter at all. A shared leaderboard makes every agent's top-weighted source the same agent, which we predict recreates conformity indirectly. The leaderboard ablation tests this directly.

Why not just add a dissimilarity penalty, as MEoH does? Selecting for dissimilarity after generation does not stop a model from imitating a candidate it has already read. We will add an abstract-syntax-tree dissimilarity term to Debate as an extra arm to check whether post-hoc selection alone recovers positive MIG.

Proposed Experiments

The main testbed is the 11-task verifier-scored suite from the Interaction Tax paper. It includes Erdős Overlap, Difference Bases, Flat Polynomials, 3AP-Free, Circle Packing, TSP-50/100, Molecule QED, Lennard-Jones with 41 atoms, and Knapsack-50, all run with diverse-model teams and 5 seeds. A secondary testbed is TSP and Online Bin Packing through llm-eps, to check that the effect holds in program-evolution search. We measure:

  • MIG for every method;
  • normalized verifier score;
  • first-round conformity rate (the fraction of agents whose next candidate falls within a small distance of a peer's);
  • pairwise solution diversity over time;
  • copying rate.

Three checks decide the outcome. First, conformity plotted against the cap $c$ should show a threshold near 0.5. A smooth or absent threshold would refute the opinion-dynamics explanation even if scores improve. Second, diverse-team MIG under the capped channel should be positive where Debate's is negative, at matched budget. Third, the benefit of constraint flags should split by task type, positive on Knapsack and neutral or negative on 3AP-Free, as the Interaction Tax local/global finding predicts.