VHELM: A Holistic Evaluation of Vision Language Models

Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, Percy Liang

TL;DR

VHELM (Holistic Evaluation of Vision Language Models) is a new, standardized benchmarking framework that evaluates 22 prominent vision-language models (VLMs) across nine critical dimensions, including reasoning, bias, and safety. The study reveals that while frontier models like GPT-4o dominate in raw capabilities, no single model excels across all aspects—particularly when it comes to societal factors like fairness and robustness.

What Problem Does This Paper Address?

As Vision-Language Models (VLMs)—AI systems that "see" images and "write" text—become integrated into everything from robotics to medical QA, our methods for testing them have lagged behind.

Current benchmarks suffer from three main issues:

  1. Narrow Scope: Most focus purely on "perception" (what is in the image?) or "reasoning" (logic puzzles), neglecting "societal" factors like bias, toxicity, multilinguality, and safety.
  2. Inconsistency: Different benchmarks use varying prompts and inference parameters, making "apple-to-apple" comparisons between models like GPT-4o and Claude 3.5 nearly impossible.
  3. Evaluation Cost: Many benchmarks require expensive human annotation or complex setups, making them difficult to run frequently as new models are released.

VHELM addresses these by extending the "Holistic Evaluation" (HELM) philosophy to the vision domain, providing a unified, automated, and multi-dimensional leaderboard.

Methods and Framework

The authors developed a standardized evaluation pipeline consisting of four core components:

  • Aspects: Nine evaluative dimensions: Visual Perception, Knowledge, Reasoning, Bias, Fairness, Multilinguality, Robustness, Toxicity, and Safety.
  • Scenarios: 21 existing datasets mapped to these aspects.
  • Adaptation: Standardized zero-shot prompting and inference parameters to ensure fairness.
  • Metrics: A shift toward automated metrics to keep evaluations "cheap and fast." This includes Exact Match for multiple-choice, Perspective API for toxicity, and Prometheus-Vision—a specialized VLM used as a "judge" to grade the quality of open-ended responses on a scale of 1–5.
Overview of the 9 VHELM aspects, example prompts, and metrics.

The core result is a Win Rate, calculated as the probability that a specific model outperforms another randomly selected model across all scenarios.

Datasets

VHELM aggregates 21 diverse datasets to cover its nine aspects. Key datasets include:

  • Perception/Knowledge: Flickr30k, VQAv2, MMMU (expert-level tasks), and VizWiz.
  • Reasoning: MathVista, GQA, and Mementos (sequential reasoning).
  • Societal/Safety: Hateful Memes (toxicity), PAIRS (gender/racial bias), FairFace (fairness), and MM-SafetyBench (harmful instructions).
  • Multilingual: Crossmodal-3600 and translated versions of A-OKVQA.
  • Robustness: Unicorn (out-of-distribution sketches) and Bingo (interference/hallucination).

Key Insights and Ideas

  • The "No King" Finding: There is no "perfect" VLM. While GPT-4o (0513) has an unparalleled 100% win rate in robustness and leads in reasoning, it performs significantly worse than other models in bias and safety.
  • The Efficiency-Bias Tradeoff: "Lite" or efficiency-focused models (like Claude 3 Haiku or Gemini 1.5 Flash) perform comparably to their "Pro/Opus" counterparts on reasoning tasks but are significantly more biased on the PAIRS benchmark.
  • The Instruction-Following Gap: Open-weight models (like IDEFICS or PaliGemma) often struggle not because they lack visual understanding, but because they fail to follow formatting instructions (e.g., providing a long sentence when asked for a single letter).
  • English-Centricity: Even the best models see a massive performance drop (up to 33%) when prompted in languages other than English. Spanish and Chinese generally perform better than Hindi or Swahili.
  • Distribution Fragility: Models are surprisingly robust to typos, but they struggle with "distribution shifts"—for example, GPT-4o's accuracy drops when looking at abstract sketches (Unicorn dataset) compared to real photos.

Key Results

The benchmark evaluated 22 models. Some standout numbers from the "Win Rate" leaderboard:

  • Top Overall: GPT-4o (0513) led with a mean win rate of 0.793, followed by its newer iteration (0806) at 0.766.
  • Closed vs. Open: Closed-API models (GPT, Gemini, Claude) significantly outperformed open-weight models. The best open-weight model, IDEFICS2-8B, trailed with a win rate of 0.344.
  • Bias Accuracy: Top closed models achieved ~95% accuracy on bias probes, while open-weight models were as low as 0% to 48%, indicating a major area for improvement in open-source fine-tuning.
  • Toxicity: Detecting "mean" memes remains difficult; the best model (IDEFICS 2) only managed 62.2% accuracy on the Hateful Memes dataset.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Thin Coverage: The "Safety" and "Toxicity" aspects currently rely on a small number of datasets and could be expanded.
  • Metric Imperfection: While automated "VLM-as-a-judge" (Prometheus-Vision) is efficient, it may still have biases or miss nuances that a human would catch.
  • Data Contamination: As with all benchmarks, there is a risk that models were trained on the test data. The authors hope to integrate "fresh" real-world data in future iterations.
  • Living Benchmark: VHELM is intended to be a "living" project, with plans to add more models and datasets as the field evolves.

Summary updated

Abstract

Current benchmarks for assessing vision-language models (VLMs) often focus on their perception or problem-solving capabilities and neglect other critical aspects such as fairness, multilinguality, or toxicity. Furthermore, they differ in their evaluation procedures and the scope of the evaluation, making it difficult to compare models. To address these issues, we extend the HELM framework to VLMs to present the Holistic Evaluation of Vision Language Models (VHELM). VHELM aggregates various datasets to cover one or more of the 9 aspects: *visual perception*, *knowledge*, *reasoning*, *bias*, *fairness*, *multilinguality*, *robustness*, *toxicity*, and *safety*. In doing so, we produce a comprehensive, multi-dimensional view of the capabilities of the VLMs across these important factors. In addition, we standardize the standard inference parameters, methods of prompting, and evaluation metrics to enable fair comparisons across models. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast. Our initial run evaluates 22 VLMs on 21 existing datasets to provide a holistic snapshot of the models. We uncover new key findings, such as the fact that efficiency-focused models (e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark but not when evaluated on the other aspects. For transparency, we release the raw model generations and complete results on our website at https://crfm.stanford.edu/helm/vhelm/v2.0.1. VHELM is intended to be a living benchmark, and we hope to continue adding new datasets and models over time.
BibTeX
@inproceedings{lee2024vhelm,
  author = {Tony Lee and Haoqin Tu and Chi Heem Wong and Wenhao Zheng and Yiyang Zhou and Yifan Mai and Josselin Somerville Roberts and Michihiro Yasunaga and Huaxiu Yao and Cihang Xie and Percy Liang},
  title = {{VHELM: A Holistic Evaluation of Vision Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=TuMnKFKPho},
}
Figure