TL;DR
VHELM (Holistic Evaluation of Vision Language Models) is a new, standardized benchmarking framework that evaluates 22 prominent vision-language models (VLMs) across nine critical dimensions, including reasoning, bias, and safety. The study reveals that while frontier models like GPT-4o dominate in raw capabilities, no single model excels across all aspects—particularly when it comes to societal factors like fairness and robustness.
What Problem Does This Paper Address?
As Vision-Language Models (VLMs)—AI systems that "see" images and "write" text—become integrated into everything from robotics to medical QA, our methods for testing them have lagged behind.
Current benchmarks suffer from three main issues:
- Narrow Scope: Most focus purely on "perception" (what is in the image?) or "reasoning" (logic puzzles), neglecting "societal" factors like bias, toxicity, multilinguality, and safety.
- Inconsistency: Different benchmarks use varying prompts and inference parameters, making "apple-to-apple" comparisons between models like GPT-4o and Claude 3.5 nearly impossible.
- Evaluation Cost: Many benchmarks require expensive human annotation or complex setups, making them difficult to run frequently as new models are released.
VHELM addresses these by extending the "Holistic Evaluation" (HELM) philosophy to the vision domain, providing a unified, automated, and multi-dimensional leaderboard.
Methods and Framework
The authors developed a standardized evaluation pipeline consisting of four core components:
- Aspects: Nine evaluative dimensions: Visual Perception, Knowledge, Reasoning, Bias, Fairness, Multilinguality, Robustness, Toxicity, and Safety.
- Scenarios: 21 existing datasets mapped to these aspects.
- Adaptation: Standardized zero-shot prompting and inference parameters to ensure fairness.
- Metrics: A shift toward automated metrics to keep evaluations "cheap and fast." This includes Exact Match for multiple-choice, Perspective API for toxicity, and Prometheus-Vision—a specialized VLM used as a "judge" to grade the quality of open-ended responses on a scale of 1–5.
The core result is a Win Rate, calculated as the probability that a specific model outperforms another randomly selected model across all scenarios.
Datasets
VHELM aggregates 21 diverse datasets to cover its nine aspects. Key datasets include:
- Perception/Knowledge: Flickr30k, VQAv2, MMMU (expert-level tasks), and VizWiz.
- Reasoning: MathVista, GQA, and Mementos (sequential reasoning).
- Societal/Safety: Hateful Memes (toxicity), PAIRS (gender/racial bias), FairFace (fairness), and MM-SafetyBench (harmful instructions).
- Multilingual: Crossmodal-3600 and translated versions of A-OKVQA.
- Robustness: Unicorn (out-of-distribution sketches) and Bingo (interference/hallucination).
Key Insights and Ideas
- The "No King" Finding: There is no "perfect" VLM. While GPT-4o (0513) has an unparalleled 100% win rate in robustness and leads in reasoning, it performs significantly worse than other models in bias and safety.
- The Efficiency-Bias Tradeoff: "Lite" or efficiency-focused models (like Claude 3 Haiku or Gemini 1.5 Flash) perform comparably to their "Pro/Opus" counterparts on reasoning tasks but are significantly more biased on the PAIRS benchmark.
- The Instruction-Following Gap: Open-weight models (like IDEFICS or PaliGemma) often struggle not because they lack visual understanding, but because they fail to follow formatting instructions (e.g., providing a long sentence when asked for a single letter).
- English-Centricity: Even the best models see a massive performance drop (up to 33%) when prompted in languages other than English. Spanish and Chinese generally perform better than Hindi or Swahili.
- Distribution Fragility: Models are surprisingly robust to typos, but they struggle with "distribution shifts"—for example, GPT-4o's accuracy drops when looking at abstract sketches (Unicorn dataset) compared to real photos.
Key Results
The benchmark evaluated 22 models. Some standout numbers from the "Win Rate" leaderboard:
- Top Overall: GPT-4o (0513) led with a mean win rate of 0.793, followed by its newer iteration (0806) at 0.766.
- Closed vs. Open: Closed-API models (GPT, Gemini, Claude) significantly outperformed open-weight models. The best open-weight model, IDEFICS2-8B, trailed with a win rate of 0.344.
- Bias Accuracy: Top closed models achieved ~95% accuracy on bias probes, while open-weight models were as low as 0% to 48%, indicating a major area for improvement in open-source fine-tuning.
- Toxicity: Detecting "mean" memes remains difficult; the best model (IDEFICS 2) only managed 62.2% accuracy on the Hateful Memes dataset.
Limitations and Future Work
The authors acknowledge several hurdles:
- Thin Coverage: The "Safety" and "Toxicity" aspects currently rely on a small number of datasets and could be expanded.
- Metric Imperfection: While automated "VLM-as-a-judge" (Prometheus-Vision) is efficient, it may still have biases or miss nuances that a human would catch.
- Data Contamination: As with all benchmarks, there is a risk that models were trained on the test data. The authors hope to integrate "fresh" real-world data in future iterations.
- Living Benchmark: VHELM is intended to be a "living" project, with plans to add more models and datasets as the field evolves.
Summary updated