TL;DR
Multi-agent document assessment in Retrieval-Augmented Generation (RAG) is powerful but computationally ruinous. This paper reveals a surprising dichotomy: smaller 7B–9B models don't actually benefit from agentic "scoring" capabilities, but rather from the structural side-effect of document isolation. By diagnosing this behavior label-free, the authors introduce MADARA, a routing architecture that achieves peak RAG performance while slashing unnecessary multi-agent compute overhead by up to 4×.
What Problem Does This Paper Address?
Deploying multiple LLM agents to evaluate, filter, or debate retrieved documents significantly improves RAG accuracy. However, this multi-agent paradigm multiplies inference calls exponentially, leading to prohibitive latency and costs in production.
To save money, practitioners often swap out giant models for smaller, deployable ones (7B–9B parameters). But do these smaller models actually possess the sophisticated reasoning required to score documents reliably? If their performance gains stem merely from bypassing long-context confusion rather than substantive reasoning, we are paying a massive "compute tax" for redundant processing. This paper aims to understand what truly drives multi-agent RAG gains in smaller models and how to optimize these pipelines dynamically.
Methods
The authors first conduct controlled ablations of a standard three-agent RAG pipeline using training-free interventions. To systematically classify model behavior, they introduce two core innovations:
- Reasoning-Score Coupling (RSC): A label-free diagnostic probe. It applies three levels of increasing perturbation to a model's Chain-of-Thought (CoT) reasoning—Shuffled (reordered steps), Contradicted (negated steps), and Random (completely irrelevant context)—and measures if the model's document scores degrade monotonically. If they do, the model is "quality-ordered"; if not, its scoring is "stochastic."
- MADARA (Model-Adaptive Document Assessment Routing Architecture): An offline-online pipeline. It profiles a model-task pair using RSC and its baseline No-Filter (NF) accuracy, then dynamically routes queries to one of four tailored treatments:
- Per-Document Extraction (PDE): Processes documents in strict isolation to eliminate multi-document context confusion.
- Score Distribution Alignment (SDA): Normalizes stochastic scores into percentile ranks.
- CoT De-polarization: Forces explicit reasoning to prevent models from assigning only extreme (0 or 5) scores.
- Adaptive Threshold Filtering (ATF): Dynamically filters out low-scoring documents.
Datasets
The framework was evaluated across diverse single-hop and multi-hop QA benchmarks:
- CONFLICTS (CFL): An adversarial QA dataset featuring inter-document contradictions via entity substitution.
- FEVER (FVR): A binary fact-verification benchmark representing a high-baseline regime.
- TriviaQA (TQA): A standard factoid QA benchmark used as a held-out evaluation set.
- MuSiQue: A complex multi-hop QA dataset used to test the boundaries of document isolation.
Key Insights and Ideas
- The Isolation vs. Scoring Asymmetry: For weaker models, multi-agent RAG gains do not come from high-quality scoring. Instead, they come from structural isolation. Smaller models suffer from severe multi-document context confusion ("lost-in-the-middle" syndrome). Forcing them to look at documents one by one cures this bottleneck.
- Assessment is Redundant for Weak Baselines: Astoundingly, a variant called PDE-Random—which completely bypasses multi-agent assessment and just selects documents at random for isolated generation—matches the performance of the full, computationally heavy multi-agent pipeline for weaker models.
- Scoring is a Dynamic Interaction: A model's ability to score is not a fixed trait. Models can be highly "quality-ordered" on simple binary tasks (like FEVER) but completely "stochastic" on complex adversarial tasks (like CONFLICTS).
Key Results
- Massive Accuracy Boosts: For weak baseline models like Llama-3.1-8B on the CONFLICTS dataset, switching to Per-Document Extraction (PDE) boosted Exact Match (EM) accuracy by +36.3 percentage points over standard RAG. On TriviaQA, it yielded a +49.8 percentage point gain.
- 4× Compute Reduction: Because PDE-Random matches full multi-agent performance on weaker baselines, practitioners can entirely eliminate multi-agent scoring loops, reducing inference calls by roughly 4× without sacrificing accuracy.
- MADARA Superiority: MADARA successfully routes model-task pairs zero-shot across four unseen model families, cutting the overall performance gap to the theoretical "oracle" strategy down to just 0.45 percentage points.
Limitations and Future Work
- The Multi-Hop Boundary: Document isolation (PDE) is structurally incapable of solving multi-hop reasoning tasks (like MuSiQue) because it prevents the model from synthesizing information across different documents. On multi-hop tasks, the optimal strategy flips back to agentic scoring.
- Static Thresholds: MADARA currently relies on a static baseline threshold ($\tau_{NF} = 30\%$) derived from a pilot model. While empirically robust, highly heterogeneous retrieval pipelines will require future engineering to dynamically adapt this threshold based on retrieval quality.
Summary updated