Abstract: Large-scale Vision Language Models (LVLMs) are increasingly being applied to a wide range of real-world multimodal applications, involving complex visual and linguistic reasoning. As these models become more integrated into practical use, they are expected to handle complex aspects of human interaction. Among these, color perception is a fundamental yet highly variable aspect of visual understanding. It differs across individuals due to biological factors such as Color Vision Deficiencies (CVDs), as well as differences in culture and language. Despite its importance, perceptual diversity has received limited attention. In our study, we evaluate LVLMs' ability to account for individual level perceptual variation using the Ishihara Test, a widely used method for detecting CVDs. Our results show that LVLMs can explain CVDs in natural language, but they cannot simulate how people with CVDs perceive color in image based tasks. These findings highlight the need for multimodal systems that can account for color perceptual diversity and support broader discussions on perceptual inclusiveness and fairness in multimodal AI.
Introduction: The Blind Spot in Multimodal AI
As Large-scale Vision-Language Models (LVLMs) become central to assistive technologies, navigation, and complex visual reasoning, a critical assumption remains largely unchallenged: that there is a "standard" way of seeing the world. Most AI training datasets are built on web-scale images and captions that implicitly reflect typical human vision. However, this ignores the approximately 8% of males and 0.5% of females who experience Color Vision Deficiency (CVD), such as red-green color blindness.
If an AI navigation assistant instructs a color-blind user to "follow the green line" on a map where red and green appear identical, the system has fundamentally failed in its role as an inclusive assistant. This research investigates whether state-of-the-art models can account for these perceptual differences or if they are "hard-wired" to a single, normative visual experience.
Methodology: Probing Perception with the Ishihara Test
To diagnose this gap, the authors repurposed the Ishihara Test—the clinical gold standard for detecting CVD—as a diagnostic probe for AI. The test consists of plates made of colored dots that form numerals; people with different types of vision see different numbers or none at all.
The study evaluated leading models, including GPT-4o, Llama-3.2, Qwen2.5-VL, and mPLUG-Owl3, across three levels of analysis:
- Generation Level: Assessing if the model can output the specific digit a person with a particular CVD (like Protanopia or Deuteranopia) would see.
- Confidence Level: Using Perplexity to measure the model's internal uncertainty when forced to predict answers under simulated CVD conditions.
- Internal Representation Level: Applying LogitLens to probe the model's internal transformer layers, checking if latent representations distinguish between vision types before the final output.
Key Insights: Factual Knowledge vs. Perceptual Grounding
The study revealed a striking disconnect: Factual knowledge does not equal perceptual grounding. While models can accurately define "Protanopia" in text, they fail to apply that definition to visual inputs. They "know" the theory of color blindness but lack the "experience" to simulate it.
This stems from a massive Normative Bias. Models have such a strong prior for typical vision that even when explicitly prompted to "see" like a person with CVD, they default to what a person with standard vision sees. Interestingly, models performed well on Tritanopia (blue-yellow deficiency) controls. Since the Ishihara test doesn't target Tritanopia, the "correct" answer for it is the same as for normal vision. This proves the models understood the instructions but specifically failed at the red-green perceptual simulation.
Results: The Simulation Failure
The empirical results highlight a significant barrier to inclusive AI. While GPT-4o achieved approximately 90% accuracy for normal vision plates, no model exceeded 24% accuracy when tasked with simulating CVD-specific responses.
Furthermore, Perplexity scores were significantly higher for red-green CVD conditions than for normal vision. This indicates that even when a model manages to output a "normal" response, it is internally more "confused" by the CVD-specific prompt. LogitLens analysis confirmed this internal inconsistency, showing that representations for different vision types remained nearly identical across layers.
Conclusion and Future Directions
The failure of medical fine-tuning (MedVLM-R1) and specialized training on synthetic dot patterns to fix this issue suggests the problem is deeply rooted in how models integrate visual and linguistic information.
The authors call for a shift away from the "standard observer" assumption. Future work must focus on developing LVLMs that can explicitly represent and reason about diverse perceptual experiences. Achieving true inclusivity in multimodal AI requires systems that don't just know about human diversity, but can actually perceive the world through the diverse eyes of their users.
Summary updated