When a group of AI agents discusses a complex problem, the resulting transcript can be massive. To make a final decision, we often compress that discussion into a "memory" or summary. But what should that memory prioritize?
A new paper, "Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions," reveals that our go-to strategies for summarizing AI group talk are surprisingly fragile. By introducing a new diagnostic benchmark called CrossProfile, the researchers demonstrate that LLMs are prone to "social proof" biases and that the most common ways we filter information can actually lead models toward the wrong answer.
TL;DR
Compression heuristics like prioritizing "rare" or "relevant" facts often fail in multi-agent decision-making because they don't account for the balance of evidence. The authors find that LLMs suffer from a "social proof" bias—overweighting facts mentioned by multiple agents—and that the biggest failure often happens "upstream" during the discussion itself, before memory is even created.
What Problem Does This Paper Address?
The authors focus on hidden-profile tasks. These are scenarios where a group collectively holds all the information needed to make a correct choice, but individual members only have pieces of the puzzle. Typically, information held by everyone (shared info) points to a "decoy" (a plausible but wrong choice), while the critical information needed to find the truth is "private" (held by only one person).
When we compress these discussions to fit into an LLM's context window, we usually rely on two heuristics:
- Rarity: Keep the facts that only one or two agents mentioned (assuming they are the "private" keys to the truth).
- Relevance: Keep facts that relate to the decision criteria.
The paper investigates why these heuristics fail and how the way we attribute information (who said what) biases the final decision-maker.
Methods
The researchers developed a pipeline to test how different "memory policies" affect decision accuracy across several LLMs (Qwen2.5-7B, Llama-3.1-8B, and DeepSeek models).
They compared several memory policies:
- R1 (Rarest-first): Prioritizes facts mentioned by the fewest agents.
- B4 (First-mention): Prioritizes facts based on when they first appeared in the transcript.
- R1-rel (Relevance-first): Filters for facts relevant to the criteria, then applies the rarity rule.
- B3 (Summary): An LLM-generated summary of the discussion.
Datasets
The study uses two main benchmarks:
- HiddenBench: An existing benchmark for multi-agent reasoning.
- CrossProfile (New): A diagnostic benchmark of 60 templates (480 tasks total). Unlike previous benchmarks, CrossProfile systematically "crosses" fact ownership. It takes the same set of facts and varies whether the "support" facts (pro-correct) or "decoy" facts (pro-wrong) are shared or private. This allows researchers to see if a model is actually following a rarity rule or just getting lucky because the rare facts happened to be the right ones.
Key Insights and Ideas
- Rarity is a Gamble: Prioritizing rare facts only works if the rare facts are the useful ones. If the "decoy" facts are the ones held privately, the rarity heuristic leads the model straight to the wrong answer.
- Relevance is Blind to Utility: Decoys are, by definition, relevant to the decision. A relevance filter cannot distinguish between a helpful fact and a misleading one that is "on topic."
- The "Social Proof" Bias: LLMs exhibit a version of the human "common-knowledge effect." They give extra weight to facts attributed to many agents (e.g., marked with tags like
[A1, A2, A3, A4]). This bias persists even when the "shared" facts are just redundant assignments, not independent observations. - The Upstream Bottleneck: In natural, unscripted discussions, agents often fail to even mention their private information. If the evidence never makes it into the transcript, no memory policy—no matter how clever—can recover it.
Key Results
- Performance Reversal: On CrossProfile, the Rarest-first (R1) and First-mention (B4) policies are mirror images. R1 hits ~95% accuracy when support is private, but drops to ~20% when decoys are private.
- Social Proof Impact: Removing attribution tags from a memory containing all facts increased accuracy by 13 to 24 points in "classic" hidden-profile setups. This proves the model was being misled by the number of "speakers" associated with a fact, rather than the fact's content.
- Natural Discussion Failure: In unscripted tests, the full transcript (REF) actually yielded lower accuracy than an empty memory. The models were effectively talking themselves into the wrong answer because they failed to disclose their private evidence.
Limitations and Future Work
The authors acknowledge that their study used a fixed group size (four agents) and focused on scripted vs. natural discussion gaps.
Future work should move away from simple prevalence-based or topic-based filtering. Instead, memory designs for agent groups should aim to preserve the balance of evidence between competing options. Furthermore, researchers suggest that system prompts should be carefully designed to ensure agents disclose private evidence before they start making recommendations, helping to solve the "upstream" silence that dooms many group decisions.
Summary updated