Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe

Jiawei ChenarXiv 2026 · arXiv

TL;DR

Prompting vision-language models (VLMs) with personas like "You have low vision" fails to simulate actual human perception; instead, models simply parrot back stated facts or default to stereotypes. The study demonstrates that while VLMs can estimate how visual impairments affect others, they cannot embody those impairments through text alone—true simulation requires modifying the visual input itself.

What Problem Does This Paper Address?

As AI models become more capable, researchers and designers are tempted to use them as "synthetic participants" in user studies. For example, instead of recruiting 50 people with low vision to test if a new street sign is legible, one might prompt a VLM: "You have a visual acuity of 20/165. Can you read this sign?"

This paper investigates whether this "persona prompting" actually works. Does a textual description of a disability change how a model processes an image, or does it just change what the model says it sees? The authors argue that there is a fundamental gap between knowing facts about a viewer and answering as that viewer.

Methods

The researchers evaluated 14 different VLMs (including frontier models like Gemini, Claude, and GPT-4o—referred to as "Sol" in the paper) across five distinct visual and auditory tasks. They compared three types of model responses:

  1. Persona Answer: The model is given a clean image and a persona prompt (e.g., "You are color blind").
  2. Changed-Image Answer: The model is given no persona but sees a filtered image (e.g., blurred or color-shifted) that physically simulates the condition.
  3. Estimate: The model is asked to predict, as a third party, how a specific group of humans would perform on the task.

Crucially, they compared persona performance against a task-specific control: a simple clinical rule or textbook answer that uses only the facts stated in the persona (e.g., "if acuity is X, the reading limit is Y").

Datasets

The study utilized several established human benchmarks to ground the model performance:

  • Low-vision reading: Data from 56 readers using the MNREAD test.
  • Color blindness: The Farnsworth D-15 color arrangement test.
  • Subtitle preferences: The SURE study, which tracks how deaf, hard-of-hearing, and hearing viewers prefer subtitle line breaks and speeds.
  • Object recognition under blur: A model-vs-human benchmark using 160 photos at eight blur levels.
  • Visual Illusions: A diagnostic set of ten common illusions (e.g., Müller-Lyer, Ponzo).

Key Insights and Ideas

  • Telling ≠ Seeing: Describing a visual limit in text does not remove information from the model's visual processing pipeline. A model told it has severe blur still identifies objects with near-perfect accuracy, whereas a model shown a blurred image fails just like a human would.
Figure 1. Telling a model about blur does not make it see blur. Four sighted people and Gemini name objects in photos at eight blur levels. Top: Gemini is either shown the blurred photo, or given the clean photo and told about the blur by a persona that states the matching acuity. Bottom: shown the blur, Gemini loses accuracy as people do; told about it, Gemini stays at 98–99%. The photos illustrate the blur levels; the test photos are greyscale.
  • The "Acuity Rule" Ceiling: In reading tasks, personas rarely outperformed a simple mathematical rule based on the stated acuity. They failed to capture the nuances of specific diagnoses, like macular disease, which makes reading harder than a simple acuity score suggests.
  • Internal Transparency: By using linear probes (simple classifiers trained on the model's internal hidden states), the authors showed that a persona prompt adds almost no "blur signal" to the model's internal representation of the image. The information is still there; the model just chooses to ignore it or parrot a stereotype.
  • Knowledge vs. Experience: Models are rarely "fooled" by visual illusions in their own answers. However, they are surprisingly good at estimating the percentage of humans who would be fooled. This suggests models have "textbook knowledge" of human fallibility without sharing those fallibilities.

Key Results

  • Blur Recognition: In all 14 models tested, a text description of blur had zero effect on recognition accuracy. In contrast, physically blurring the image caused 7 models to match human-like performance drops.
  • Reading Accuracy: No persona predicted a reader's limit better than a simple clinical rule. The models tended to "clip" their performance exactly to the acuity number provided in the prompt, ignoring other diagnostic information like contrast sensitivity.
Figure 2. A reading persona simply copies the stated letter-chart acuity, missing the variance of real human readers.
  • Color Blindness: Personas for red-blindness (protanopia) often gave the "textbook" answer for green-blindness (deuteranopia), failing to distinguish between the two. Filtering the colors of the image itself resulted in much higher alignment with human data.
  • Effect Sizes: Across all models, the "persona effect" was negligible (near 0), while the "blurred image effect" was robust, often exceeding 1.0 (meaning the model was more affected by blur than humans).
Figure 5. The model's accuracy loss divided by people's loss. Text personas (red) show no effect, while blurred images (blue) show significant impact.

Limitations and Future Work

The authors acknowledge that they tested relatively short, template-based personas. While richer, more narrative descriptions are a natural next step, the internal probing results suggest that text alone is unlikely to fundamentally alter visual perception in current architectures.

Furthermore, some human reference groups were small (e.g., only 4 observers for the blur test), and the models were tested on static frames rather than the full video/audio experience humans had in the subtitle studies.

The primary takeaway for the research community is clear: If you want to simulate how a person with a visual impairment sees, you must change the image, not the prompt.

Summary updated

Abstract

Vision-language models (VLMs) are increasingly prompted with personas, such as "You have low vision", to stand in for people in user studies. We ask whether a persona makes a model see like the person it describes. We compare persona answers with measured people with low vision, color blindness or hearing loss, and with simple rules that use only the facts the persona states. The persona never beats these rules, and it erases the real differences between groups and individuals. Describing blur leaves recognition unchanged in all 14 models we test, while blurring the photo lowers it as it does for people. Inside open models, the description changes neither the image information nor the answer probabilities. Yet the models know the effects they fail to show: they can estimate how blur and illusions affect people. Telling is not seeing: a persona makes a model say what it knows about people, not see what they see.
BibTeX
@misc{chen2026telling,
  author = {Jiawei Chen},
  title = {{Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe}},
  year = {2026},
  eprint = {2610.39331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39331},
}
Figure