Abstract
Recently, multimodal large language models (MLLMs) have received much attention for their impressive capabilities. The evaluation of MLLMs is becoming critical to analyzing attributes of MLLMs and providing valuable insights. However, current benchmarks overlook the problem of prompt sensitivity - minor prompt variations may lead to significant performance fluctuations. Thus, inappropriate prompts may obscure the models' capabilities, underestimating the models' performance. Moreover, different models have different preferences for different prompts, and thus, using the same prompt for all models will cause evaluation bias. This paper analyzes this deficiency in existing benchmarks and further introduces a new evaluation framework named TP-Eval, which introduces a prompt customization method to reduce evaluation biases and tap models' potential. TP-Eval will rewrite the original prompts to different customized prompts for different models. In particular, we propose some well-designed modules for prompt customization tailored to the scenario of MLLM evaluation. Extensive experiments demonstrate the effectiveness of our approach to uncovering models' capabilities, and TP-Eval should benefit the community in developing more comprehensive and convincing MLLM evaluation benchmarks.
The Problem: Prompt Sensitivity and Evaluation Bias
In the rapidly evolving field of Multimodal Large Language Models (MLLMs), standard benchmarks like MMT-Bench and MMMU serve as the primary yardsticks for progress. However, these benchmarks typically rely on static, human-designed prompt templates applied uniformly across all models. This "one-size-fits-all" approach introduces two critical flaws: underestimation and evaluation bias.
Underestimation occurs when a model possesses the underlying capability to solve a task but fails because the specific phrasing of the prompt does not align with its internal processing. For instance, a minor change in how a question about visual cues is framed can cause a model's accuracy to fluctuate by over 20%. Evaluation bias arises because different model architectures have distinct "preferences." A prompt optimized for one model might be suboptimal for another, leading to unfair rankings that reflect prompt compatibility rather than raw reasoning or perceptual ability.
The TP-Eval Framework
To address these deficiencies, we introduce TP-Eval, an automated framework designed to uncover a model's true potential by finding its "optimal" prompt for any given task. TP-Eval operates through an iterative optimization loop that customizes prompts for specific models.
Core Components
The framework consists of three primary modules working in a feedback loop:
- The Optimizer ($M_O$): Utilizing a high-capacity model (such as GPT-4o mini), this module generates and refines candidate prompts based on a high-level "Meta-Prompt."
- The Scorer: This module evaluates candidate prompts on a small validation set ($D_{few}$). It assigns a score based on task accuracy and semantic similarity to the original prompt, ensuring the task's core meaning remains intact.
- Answer Analyzer ($M_A$): This is the "secret sauce" of TP-Eval. Unlike standard optimizers that only see numerical scores, the Answer Analyzer performs Error Introspection. It examines why a model failed a specific example and provides a natural language explanation (e.g., "the model ignored the background context") back to the Optimizer. This allows for targeted, intelligent prompt adjustments.
To prevent overfitting to the small validation set, TP-Eval employs a decaying edit distance constraint, limiting the number of words the optimizer can change in a single iteration.
Experimental Setup and Results
We evaluated TP-Eval using three prominent MLLMs—LLaVA-1.5-7B, DeepSeek-VL-7B, and Mini-InternVL-Chat-4B-V1-5—across diverse benchmarks including MMT-Bench (MMT-S) and MMMU.
Key Findings
- Performance Gains: TP-Eval consistently improved performance across all models. LLaVA-1.5-7B's accuracy on MMT-S rose from 50.4% to 54.4%, while DeepSeek-VL-7B saw an increase from 55.2% to 57.3%. In specific tasks like "spot similarity," customization nearly doubled accuracy compared to the original benchmark prompts.
- Model "Personalities": Our experiments confirmed that prompt sensitivity is highly model-specific. Prompts optimized for LLaVA often performed poorly when transferred to DeepSeek, proving that there is no universal "best" prompt.
- The Power of Introspection: Ablation studies showed that without the Answer Analyzer's introspection, the optimization process was significantly less effective. Understanding the reason for failure is crucial for effective prompt engineering.
- Zero-Shot Transfer: Optimized prompts can serve as effective In-Context Learning (ICL) examples, improving performance on new tasks without additional training iterations.
Conclusion and Future Work
TP-Eval demonstrates that current MLLM benchmarks are likely under-reporting model capabilities due to prompt sensitivity. By automating prompt customization, we can achieve fairer and more comprehensive evaluations. Future work will focus on improving "semantic integrity" to ensure the optimizer does not inadvertently simplify tasks, and reducing the computational overhead of the iterative optimization process. TP-Eval represents a significant step toward more robust and insightful MLLM evaluation methodologies.
Summary updated