TL;DR
RE-PRESENT is a training-free framework that boosts the performance of Large Language Model (LLM) ensembles by "triangulating" evidence. By selectively re-presenting disputed multiple-choice questions in alternative formats—such as reversing the order of options—it extracts more reliable signals than standard confidence scores, achieving high accuracy with a fraction of the computational cost required by multi-agent debate.
What Problem Does This Paper Address?
When you combine different LLMs into an ensemble, you hope their individual strengths will cover each other's weaknesses. However, even when at least one model in a group knows the right answer (the "oracle" case), current systems often fail to pick it.
The authors identify two major hurdles:
- The Oracle Gap: There is a massive performance ceiling (up to 12.6 percentage points) that standard voting or "routing" methods fail to reach.
- Unreliable Signals: Traditional signals like a model’s self-reported confidence or simple agreement between models are surprisingly poor at predicting if an answer is actually correct. High-confidence errors are a persistent plague.
Methods
The researchers propose a two-stage framework called RE-PRESENT. Unlike complex "routers" that require supervised training, RE-PRESENT is entirely training-free.
- Stage 1: Single-Round Aggregation. All models in the ensemble answer the question once. Their probabilities are combined using log-linear pooling.
- Selective Trigger: If the models mostly agree, the process stops to save cost. If there is significant disagreement (a "disputed" question), Stage 2 is triggered.
- Stage 2: Alternative Views. The disputed question is re-presented to the models in two new ways:
- Top-Two View: The models are only shown the two most likely candidates from Stage 1.
- Reversed View: All original options are shown, but their order is reversed.
- Triangulation: The framework aggregates the evidence from all three views. The core idea is that a correct answer will remain "stable" across these different presentations, while incorrect guesses are likely to "flip" when the context changes.
Datasets
The authors evaluated their method across seven diverse multiple-choice reasoning benchmarks, totaling 7,000 test questions:
- Medical: MedQA, MedMCQA, HeadQA.
- General Knowledge/Science: MMLU, ARC-Challenge (ARC-C), OpenBookQA.
- Mathematics: MathQA.
They tested these on two heterogeneous ensembles: a 7–9B parameter ensemble (including Llama 3.1 and Gemma 2) and a 14–32B parameter ensemble (including Qwen 3 and Ministral).
Key Insights and Ideas
- Stability is the Best Signal: The "aha moment" of the paper is that prediction stability is a much better indicator of truth than raw confidence. A model that picks "C" when the options are A-B-C-D and still picks "C" when they are D-C-B-A is significantly more likely to be right.
- Learned Routers are Overrated: The authors found that sophisticated, trained "routers" often just learn to favor the globally strongest model rather than picking the best model for a specific hard question. Simple log-linear pooling actually outperformed many learned methods.
- Triangulation over Interaction: While "multi-agent debate" (where models talk to each other) can improve accuracy, it is incredibly expensive in terms of tokens. RE-PRESENT achieves similar gains by simply looking at the same question from different angles, avoiding the overhead of long conversational contexts.
Key Results
RE-PRESENT consistently outperformed strong single-round baselines and provided a superior accuracy-to-cost ratio compared to interactive methods.
- Accuracy Gains: It improved average accuracy from 80.31% to 81.51% for the 7–9B ensemble and from 86.67% to 87.46% for the 14–32B ensemble.
- Efficiency: As shown in the trade-off analysis, RE-PRESENT achieves competitive accuracy while using vastly fewer tokens than interactive baselines like MAD-M² (Multi-Agent Debate).
- Disputed Questions: On the hardest questions (where models disagree), RE-PRESENT pooling reached 58.9% accuracy, significantly higher than the 54.9% achieved by standard single-round pooling.
Limitations and Future Work
- Multiple-Choice Only: The current framework relies on candidate-level probabilities and re-ordering options, making it specific to multiple-choice tasks. Extending this to open-ended generation (like coding or essay writing) is a major future challenge.
- Fixed Views: The study used two specific alternative views (Top-Two and Reversed). The authors suggest that future work could develop adaptive strategies to choose the most informative "view" based on the specific type of question or model disagreement.
- Inference Cost: While more efficient than debate, it still requires more tokens than a single-round pass. Finding the optimal "trigger" to minimize these extra calls remains an open area for optimization.
Summary updated