Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies

Kazuki Hayashi, Shintaro Ozaki, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

This paper, titled "Diagnosing Vision Language Models’ Perception by Leveraging Human Methods for Color Vision Deficiencies," investigates a critical but overlooked gap in multimodal AI: the assumption of a "standard" visual experience. While Large-scale Vision-Language Models (LVLMs) are increasingly used in assistive technologies and navigation, they may be fundamentally blind to the perceptual diversity of the human users they aim to serve.

TL;DR

LVLMs possess extensive factual knowledge about color blindness but fail to simulate the actual perceptual experience of it. Even when explicitly prompted to "see" like a person with Color Vision Deficiency (CVD), models default to normative vision, highlighting a significant barrier to inclusive and accessible AI.

What Problem Does This Paper Address?

Most LVLMs are trained on massive web-scale datasets that implicitly assume "typical" human vision. However, roughly 8% of males and 0.5% of females experience some form of Color Vision Deficiency (CVD), such as red-green color blindness.

If an AI navigation assistant tells a color-blind user to "follow the green line" on a map where red and green look identical, the system has failed. This paper explores whether current state-of-the-art models can account for these perceptual differences or if they are "hard-wired" to a single, normative way of seeing the world.

Methods

The authors repurpose the Ishihara Test—the gold-standard clinical tool consisting of plates with colored dots—as a diagnostic probe for AI. They evaluate model behavior across three dimensions:

  1. Generation Level: Can the model output the specific digit that a person with a particular CVD would see?
  2. Confidence Level: Using Perplexity, the authors measure how "confused" or uncertain a model is when forced to predict answers under simulated CVD conditions.
  3. Internal Representation Level: Using a technique called LogitLens, they probe the model's internal transformer layers to see if the latent representations actually distinguish between different vision types before the final output is generated.

They tested several leading models, including GPT-4o, Llama-3.2, Qwen2.5-VL, and mPLUG-Owl3, using various prompting strategies (e.g., providing linguistic descriptions of the deficiency or few-shot visual examples).

Datasets

  • Ishihara Test Plates: The primary evaluation set consisted of 25 plates requiring numeral responses.
  • Synthetic Ishihara Dataset: To test if pattern recognition was the bottleneck, the authors used a dataset of 10,000 synthetic Ishihara-style images to fine-tune a specialized version of Qwen2.5-VL.
  • Medical Data: They also evaluated MedVLM-R1, a model fine-tuned on clinical medical images, to see if domain expertise improved performance.

Key Insights and Ideas

  • Factual Knowledge $\neq$ Perceptual Grounding: Models can perfectly define "Protanopia" in text, but they cannot apply that definition to an image. They "know" the theory but lack the "experience."
  • The Normative Bias: Models have a massive "normative vision" prior. When asked what a color-blind person sees, they almost always default to what a person with typical vision sees.
  • Tritanopia as a Control: Since the Ishihara test doesn't target blue-yellow (Tritanopia) deficiencies, the "correct" answer for Tritanopia is the same as Normal vision. Models performed well here, proving they understood the task instructions but specifically failed at the red-green perceptual simulation.
  • Training Doesn't Fix It: Neither medical fine-tuning nor training on dot patterns enabled models to simulate the CVD experience. This suggests the issue lies deeper in how models integrate visual and linguistic information.

Key Results

  • Simulation Failure: While GPT-4o achieved ~90% accuracy for normal vision, no model exceeded 24% accuracy when asked to simulate CVD-specific responses.
  • High Uncertainty: Perplexity scores were significantly higher for red-green CVD conditions than for normal vision. This indicates that even when a model outputs a "normal" response, it is internally more "confused" by the CVD prompt.
  • Internal Inconsistency: LogitLens analysis showed that internal representations for different vision types were nearly identical across layers, suggesting the models lack a mechanism to represent alternative perceptual states.

Limitations and Future Work

The study is limited by the small scale of the Ishihara test and its focus primarily on red-green deficiencies. The authors also note that their prompting space, while varied, was not exhaustive.

Future Work: The authors call for the development of LVLMs that can explicitly represent and reason about diverse perceptual experiences. This might involve new architectural components or training objectives that move beyond the "standard observer" assumption, ultimately leading to AI that is truly inclusive and accessible for all users.

Summary updated

Abstract

Large-scale Vision Language Models (LVLMs) are increasingly being applied to a wide range of real-world multimodal applications, involving complex visual and linguistic reasoning. As these models become more integrated into practical use, they are expected to handle complex aspects of human interaction. Among these, color perception is a fundamental yet highly variable aspect of visual understanding. It differs across individuals due to biological factors such as Color Vision Deficiencies (CVDs), as well as differences in culture and language. Despite its importance, perceptual diversity has received limited attention. In our study, we evaluate LVLMs' ability to account for individual level perceptual variation using the Ishihara Test, a widely used method for detecting CVDs. Our results show that LVLMs can explain CVDs in natural language, but they cannot simulate how people with CVDs perceive color in image based tasks. These findings highlight the need for multimodal systems that can account for color perceptual diversity and support broader discussions on perceptual inclusiveness and fairness in multimodal AI.
BibTeX
@inproceedings{hayashi2026diagnosing,
  author = {Kazuki Hayashi and Shintaro Ozaki and Yusuke Sakai and Hidetaka Kamigaito and Taro Watanabe},
  title = {{Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.356},
}

Loading related content...

Figure