BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, Alex C. Kot

Abstract

Large Multimodal Models (LMMs) such as GPT-4V and LLaVA have shown remarkable capabilities in visual reasoning with common image styles. However, their robustness against diverse style shifts, crucial for practical applications, remains largely unexplored. In this paper, we propose a new benchmark, BenchLMM, to assess the robustness of LMMs against three different styles: artistic image style, imaging sensor style, and application style, where each style has five sub-styles. Utilizing BenchLMM, we comprehensively evaluate state-of-the-art LMMs and reveal: 1) LMMs generally suffer performance degradation when working with other styles; 2) An LMM performs better than another model in common style does not guarantee its superior performance in other styles; 3) LMMs' reasoning capability can be enhanced by prompting LMMs to predict the style first, based on which we propose a versatile and training-free method for improving LMMs; 4) An intelligent LMM is expected to interpret the causes of its errors when facing stylistic variations. We hope that our benchmark and analysis can shed new light on developing more intelligent and versatile LMMs.

Introduction

Current Large Multimodal Models (LMMs), including state-of-the-art systems like GPT-4V and LLaVA, demonstrate impressive visual reasoning on "common" images—typically high-quality photographs prevalent on the internet. However, the practical utility of these models depends on their versatility across diverse real-world domains. A robust AI assistant must interpret medical MRIs, security X-rays, or artistic sketches with the same proficiency it applies to standard photos.

Despite their success, there is a significant gap in our understanding of how LMMs handle distribution shifts in visual style. Most existing benchmarks prioritize textual diversity while relying on a narrow range of photographic imagery. BenchLMM addresses this deficiency by systematically evaluating models across 15 diverse sub-styles to determine if their reasoning capabilities remain stable when visual inputs deviate from the norm.

The BenchLMM Framework

BenchLMM is structured around three primary pillars, each containing five distinct sub-styles:

  1. Artistic Style: This category includes Cartoon, Painting, Sketch, Handmade, and Tattoo styles. These images require the model to recognize objects and scenes despite non-photorealistic textures and abstractions. Example of Artistic Style (Sketch) from the BenchLMM dataset. Source, p. 1
  2. Sensor Style: This category focuses on specialized imaging hardware, including Infrared (surveillance), Low-energy X-ray (security), and medical imaging (High-energy X-ray, MRI, and CT scans). Example of Sensor Style (Security X-ray) from the BenchLMM dataset. Source, p. 1
  3. Application Style: This category covers domain-specific contexts such as Remote Sensing (satellite), Autonomous Driving, Domestic Robotics, Open Game environments (e.g., Minecraft), and Industrial Defect Detection.
Example of Artistic Style (Sketch) from the BenchLMM dataset.
Example of Sensor Style (Security X-ray) from the BenchLMM dataset.

Style Prompt Enhancement (SPE)

A key contribution of this research is Style Prompt Enhancement (SPE), a training-free prompting strategy inspired by human cognition. Humans often identify the context or medium of an image (e.g., "this is a medical scan") before analyzing its contents. By prepending the instruction "Determine the style of the image before answering the question," the authors force the LMM to leverage stylistic cues as a precursor to reasoning.

Empirical results show that SPE provides a significant "free" performance boost. For LLaVA-1.5-13B, SPE improved reasoning accuracy by 5.8% in artistic styles and 4.5% in sensor styles. This suggests that LMMs often possess the latent knowledge required for a task but fail because they do not correctly frame the visual context. However, SPE was less effective in the "Application" category (+1.4%), indicating that stylistic framing cannot fully substitute for specialized domain knowledge.

Key Insights and Results

The comprehensive evaluation of state-of-the-art LMMs yielded several critical insights:

  • The "Style Gap" is Significant: Even GPT-4V, the current leader, experienced a performance drop from 81.5% on common photos to 62.6% on artistic styles, and as low as 50% on security X-rays.
  • Common Style Performance is Deceptive: High scores on standard photographic benchmarks do not guarantee robustness. For instance, InstructBLIP-13B scored lower than its 7B version on "Common" style images but outperformed it across all artistic benchmarks.
  • Reflection as a Metric for Intelligence: The study utilized "Error-Reflection Analysis," asking models to explain their mistakes after being shown the ground truth. GPT-4V demonstrated superior intelligence by providing logical, scene-aware explanations, whereas many open-source models merely rephrased the correct answer without showing true self-correction capability.

Conclusion and Future Work

BenchLMM highlights that while LMMs are powerful, their cross-style robustness remains a major hurdle for deployment in specialized fields. While GPT-4V leads in complex reasoning, it is still vulnerable to style shifts and is often limited by safety filters in medical contexts. Future research should focus on integrating specialized domain knowledge into general-purpose models and expanding benchmarks to include even more diverse modalities, such as thermal or microscopic imaging.

Summary updated

Abstract

Large Multimodal Models (LMMs) such as GPT-4V and LLaVA have shown remarkable capabilities in visual reasoning with common image styles. However, their robustness against diverse style shifts, crucial for practical applications, remains largely unexplored. In this paper, we propose a new benchmark, BenchLMM, to assess the robustness of LMMs against three different styles: artistic image style, imaging sensor style, and application style, where each style has five sub-styles. Utilizing BenchLMM, we comprehensively evaluate state-of-the-art LMMs and reveal: 1) LMMs generally suffer performance degradation when working with other styles; 2) An LMM performs better than another model in common style does not guarantee its superior performance in other styles; 3) LMMs' reasoning capability can be enhanced by prompting LMMs to predict the style first, based on which we propose a versatile and training-free method for improving LMMs; 4) An intelligent LMM is expected to interpret the causes of its errors when facing stylistic variations. We hope that our benchmark and analysis can shed new light on developing more intelligent and versatile LMMs.
BibTeX
@inproceedings{cai2024benchlmm,
  author = {Rizhao Cai and Zirui Song and Dayan Guan and Zhenhao Chen and Xing Luo and Chenyu Yi and Alex C. Kot},
  title = {{BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=kUVzIBrS7B},
}
Figure