TL;DR
This paper explores whether simple, training-free "self-assessment" techniques—where models estimate their own confidence—are enough to build effective ensembles of Large Vision Language Models (LVLMs). The researchers find that while these simple methods can outperform individual models, their success is surprisingly fragile and highly dependent on the specific combination of datasets, model types, and prompt wording.
What Problem Does This Paper Address?
Large Vision Language Models (LVLMs) like Qwen or GPT-4V are incredibly capable, but they aren't perfect. Often, one model might excel at math while another is better at OCR. To get the best of all worlds, researchers use ensembles—groups of models working together.
However, traditional ensembling usually requires a "meta-model" that needs to be trained to aggregate answers, or complex, task-specific pipelines. This is expensive and limits the ensemble's ability to generalize to new tasks. The authors ask: Can we just let models "grade" their own answers and pick the one with the highest score? This would be a "training-free" approach, but it faces a massive hurdle: calibration. If Model A is "90% sure" and Model B is "80% sure," does that actually mean Model A is more likely to be right, or is it just more "overconfident" by nature?
Methods
The authors built a comprehensive testbed to evaluate candidate response selection. Instead of generating a new answer from scratch, the ensemble simply looks at the answers generated by several different models and picks the "best" one based on a confidence score.
They tested 34 different confidence estimation methods, which generally fall into two buckets:
- Token-level measures: Looking at the raw probabilities the model assigned to the words it produced (e.g., Mean Log Confidence).
- Verbalized confidence: Asking the model a follow-up question like, "Is your previous answer correct? Answer True or False." (referred to as P(True)).
They applied these methods across 24 ensemble configurations, ranging from "Best-of-N" (sampling the same model multiple times) to "Diverse" (mixing entirely different architectures).
Datasets
The study is impressively broad, evaluating performance across 20 different multimodal datasets. These include:
- General Reasoning: MMMU, MMBench, MMStar.
- Specialized Tasks: OCRBench (text recognition), MathVista (mathematics), ScienceQA.
- Safety & Hallucination: MMSafetyBench, POPE, HallusionBench.
Key Insights and Ideas
- Self-assessment is surprisingly competitive: Simple token-level confidence scores can actually perform as well as complex ensembling methods that require training data. This is a big win for "plug-and-play" systems.
- The "Joint Optimization" Trap: You cannot optimize one variable at a time. A confidence measure that works for one model on one dataset might fail miserably if you change the prompt or the model mix. Everything—the models, the prompt, and the task—must be considered together.
- Diversity is a double-edged sword: While combining different models should help, it often introduces calibration errors that outweigh the benefits. The authors found that Best-of-N (using multiple samples from a single strong model) is often more reliable because the "scores" are naturally calibrated against each other.
- P(True) is powerful but finicky: Asking a model if it's right (P(True) Search) was one of the top-performing methods, but it's highly sensitive to how the model formats its response. If a model adds a bit of "thinking" text before its answer, standard confidence checks often fail.
Key Results
The researchers used a metric called Relative Gain, which measures how much better the ensemble performs compared to the single best model in that group.
- Ensembles can win: In many cases, the ensemble achieved a positive gain (up to ~6% in some configurations), proving that training-free selection is viable.
- The Qwen Advantage: Using strong base models like the Qwen 3.5 series in a Best-of-N configuration provided some of the most consistent results, minimizing the "worst-case" performance dips often seen in more diverse ensembles.
- Fragility: As seen in the performance heatmaps, even "good" approaches can result in negative gains (performing worse than a single model) if the deployment conditions aren't perfectly matched.
Limitations and Future Work
The authors are transparent about several limitations:
- Inference Cost: Running an ensemble of three or more LVLMs is computationally expensive. For many users, fine-tuning a single model might be more efficient if the task is well-defined.
- Calibration Gaps: While they showed you can do without explicit calibration, they admit that relative calibration between different model families remains a major hurdle for diverse ensembles.
- Prompt Sensitivity: The fact that ensembling success depends so heavily on prompt wording (e.g., asking for a "short" vs. "normal" answer) suggests that these systems are not yet truly "set and forget."
Future research could look into more robust ways to align confidence scores across different model architectures without requiring massive training sets.
Summary updated