Is model self-assessment enough for diverse LVLM ensembles?

Michael Smith, Frank P. FerriearXiv 2026 · arXiv

TL;DR

This paper explores whether simple, training-free "self-assessment" techniques—where models estimate their own confidence—are enough to build effective ensembles of Large Vision Language Models (LVLMs). The researchers find that while these simple methods can outperform individual models, their success is surprisingly fragile and highly dependent on the specific combination of datasets, model types, and prompt wording.

What Problem Does This Paper Address?

Large Vision Language Models (LVLMs) like Qwen or GPT-4V are incredibly capable, but they aren't perfect. Often, one model might excel at math while another is better at OCR. To get the best of all worlds, researchers use ensembles—groups of models working together.

However, traditional ensembling usually requires a "meta-model" that needs to be trained to aggregate answers, or complex, task-specific pipelines. This is expensive and limits the ensemble's ability to generalize to new tasks. The authors ask: Can we just let models "grade" their own answers and pick the one with the highest score? This would be a "training-free" approach, but it faces a massive hurdle: calibration. If Model A is "90% sure" and Model B is "80% sure," does that actually mean Model A is more likely to be right, or is it just more "overconfident" by nature?

Methods

The authors built a comprehensive testbed to evaluate candidate response selection. Instead of generating a new answer from scratch, the ensemble simply looks at the answers generated by several different models and picks the "best" one based on a confidence score.

Figure 1. An overview of the testbed, showing the variables V1 to V4 that can affect how one example question from the MMVet dataset is processed through to evaluation. See Sec. 4.1 for details and Sec. 4.2 for a description of the variables.

They tested 34 different confidence estimation methods, which generally fall into two buckets:

  1. Token-level measures: Looking at the raw probabilities the model assigned to the words it produced (e.g., Mean Log Confidence).
  2. Verbalized confidence: Asking the model a follow-up question like, "Is your previous answer correct? Answer True or False." (referred to as P(True)).

They applied these methods across 24 ensemble configurations, ranging from "Best-of-N" (sampling the same model multiple times) to "Diverse" (mixing entirely different architectures).

Datasets

The study is impressively broad, evaluating performance across 20 different multimodal datasets. These include:

  • General Reasoning: MMMU, MMBench, MMStar.
  • Specialized Tasks: OCRBench (text recognition), MathVista (mathematics), ScienceQA.
  • Safety & Hallucination: MMSafetyBench, POPE, HallusionBench.

Key Insights and Ideas

  • Self-assessment is surprisingly competitive: Simple token-level confidence scores can actually perform as well as complex ensembling methods that require training data. This is a big win for "plug-and-play" systems.
  • The "Joint Optimization" Trap: You cannot optimize one variable at a time. A confidence measure that works for one model on one dataset might fail miserably if you change the prompt or the model mix. Everything—the models, the prompt, and the task—must be considered together.
  • Diversity is a double-edged sword: While combining different models should help, it often introduces calibration errors that outweigh the benefits. The authors found that Best-of-N (using multiple samples from a single strong model) is often more reliable because the "scores" are naturally calibrated against each other.
  • P(True) is powerful but finicky: Asking a model if it's right (P(True) Search) was one of the top-performing methods, but it's highly sensitive to how the model formats its response. If a model adds a bit of "thinking" text before its answer, standard confidence checks often fail.

Key Results

The researchers used a metric called Relative Gain, which measures how much better the ensemble performs compared to the single best model in that group.

Figure 3. Relative Gain (%) over the best constituent model, cho- sen optimally for each dataset, prompt and seed combination. The best possible approach for each ensemble configuration, dataset, and prompt is shown. This shows the best performance that can be obtained compared to an individual model, if we are able to fully characterize the deployment environment. This also serves as a breakdown of the right side of Fig. 2. Results are averaged across both prompt types (except CMMMU and MMSafetyBench which are only Normal) and all seeds.
  • Ensembles can win: In many cases, the ensemble achieved a positive gain (up to ~6% in some configurations), proving that training-free selection is viable.
  • The Qwen Advantage: Using strong base models like the Qwen 3.5 series in a Best-of-N configuration provided some of the most consistent results, minimizing the "worst-case" performance dips often seen in more diverse ensembles.
  • Fragility: As seen in the performance heatmaps, even "good" approaches can result in negative gains (performing worse than a single model) if the deployment conditions aren't perfectly matched.
Figure 4. Mean performance of different approaches by ensemble configuration, across all datasets, prompt types and seeds. Green and red shading represents the Relative Gain (%) over the best constituent model (whose mean performance is shown in blue), chosen optimally for each dataset, prompt and seed combination.

Limitations and Future Work

The authors are transparent about several limitations:

  • Inference Cost: Running an ensemble of three or more LVLMs is computationally expensive. For many users, fine-tuning a single model might be more efficient if the task is well-defined.
  • Calibration Gaps: While they showed you can do without explicit calibration, they admit that relative calibration between different model families remains a major hurdle for diverse ensembles.
  • Prompt Sensitivity: The fact that ensembling success depends so heavily on prompt wording (e.g., asking for a "short" vs. "normal" answer) suggests that these systems are not yet truly "set and forget."

Future research could look into more robust ways to align confidence scores across different model architectures without requiring massive training sets.

Summary updated

Abstract

Ensembles of diverse Large Vision Language Models have demonstrated performance beyond that of single models across multiple datasets and tasks. However, the necessary aggregation process usually relies on training, parameter fitting, or complex task-specific pipelines. Furthermore, evaluation is limited, making a fair comparison difficult. We create a unified pipeline where we perform candidate response selection: we build an ensemble by selecting the best answer from a set provided by different models. We comprehensively evaluate across 34 different means of confidence estimation, 24 ensemble configurations, 20 datasets, and two prompt specifications. We show that simple training and parameter-free approaches can be sufficient, even for ensembles of different architectures. These simple self-assessment approaches rely solely on each constituent model's confidence, without explicit calibration, yet are able to perform better than said constituent models. They are also competitive with more involved alternatives on standard metrics such as accuracy, including on open-ended problems. However, we also discover that dataset, ensemble composition, and prompt wording are key confounding variables insufficiently explored by prior work that significantly affect ensemble performance, such that the ensemble can perform worse than its best constituent model. These must be optimized for to get the most out of ensembles, at the cost of limiting their generalizability. We recommend that practitioners consider training-free ensembles only if deployment conditions can be precisely characterized, with Best-of-N configurations of strong models being a good starting point.
BibTeX
@misc{smith2026is,
  author = {Michael Smith and Frank P. Ferrie},
  title = {{Is model self-assessment enough for diverse LVLM ensembles?}},
  year = {2026},
  eprint = {2611.22956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22956},
}
Figure