Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision

Rosiana Natalie, Wenqian Xu, Ruei-Che Chang, Rada Mihalcea, Anhong GuoarXiv 2025 · DOI · arXiv

TL;DR

Researchers at the University of Michigan investigated whether Vision Language Models (VLMs) like GPT-4o can simulate the visual experiences of people with low vision. They found that while VLMs suffer from a "super-vision" bias—often seeing more than a low-vision person would—few-shot prompting with specific examples can significantly improve their ability to act as a proxy for accessibility testing.

The Challenge: Overcoming "Super-Vision"

Inclusive design for the 2.2 billion people globally with vision impairments is often limited by the difficulty of recruiting participants for early-stage testing. While VLMs are adept at role-playing, they struggle to "unlearn" the high-resolution data they were trained on. This "super-vision" bias leads models to infer details that a person with conditions like Glaucoma or Retinitis Pigmentosa would realistically miss. The researchers asked: Can we prompt a VLM to effectively simulate these visual constraints?

Methods and Datasets

The study utilized a three-stage approach to build and evaluate "simulated agents":

  1. Human Ground Truth: 40 low-vision participants provided medical diagnoses, personal descriptions of their vision, and responses to 25 images.
  2. Agent Construction: GPT-4o agents were created using various prompts, from simple medical labels to detailed personal vision profiles.
  3. Evaluation: The researchers measured the "agreement score"—how often the VLM's response matched the human's actual perception.
An example task from the benchmark dataset featuring an image and corresponding multiple-choice questions for object detection, color recognition, and counting.

The resulting benchmark includes 40 detailed profiles, over 700 open-ended descriptions, and 4,170 multiple-choice responses across diverse scenarios like e-commerce and social networking.

Key Insights: Showing vs. Telling

The study revealed that simply "telling" a model about a disability is insufficient. Providing a medical diagnosis or a text description (e.g., "I see blurry shapes") did little to improve simulation fidelity. The model could not translate abstract descriptions into visual constraints.

However, "showing" the model examples through few-shot prompting was highly effective. Providing just one example of how a specific user previously described an image and answered a question allowed the model to quickly adopt that user's "perceptual style."

Key Results

The baseline agreement for agents with minimal prompting was low, as they achieved near-perfect accuracy on images that humans struggled with (human accuracy averaged 60.4%). Conversely, prompting a model to act "blind" resulted in 0% accuracy, failing to capture the residual vision many low-vision individuals possess.

Mean agreement scores across various prompting strategies, highlighting the significant improvement when combining vision profiles with few-shot examples.

As shown in the results, the highest mean agreement reached was 0.70. This peak was achieved by combining detailed vision profiles with "few-shot" examples that included both open-ended descriptions and multiple-choice answers. This combination provided a statistically significant boost (p < 0.0001) over using either format in isolation.

Conclusion: A Complementary Tool

With a top agreement score of 0.70, VLMs are "not there yet" as a total replacement for human testing. A 30% discrepancy remains too high for final accessibility audits. However, these models show great promise as complementary tools for early-stage design, helping developers identify obvious accessibility hurdles before moving to human trials. Future work will explore whether newer reasoning models can better internalize the logic of visual impairment.

Summary updated

BibTeX
@misc{natalie2025not,
  author = {Rosiana Natalie and Wenqian Xu and Ruei-Che Chang and Rada Mihalcea and Anhong Guo},
  title = {{Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision}},
  year = {2025},
  eprint = {2508.10972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10972},
}
Figure