Persona Lenses: What Makes a Vision-Language Model See Through Its Persona's Eyes?

Yoonho ShinarXiv 2026 · arXiv

TL;DR

Vision-Language Models (VLMs) can simulate user gaze on web pages, but standard text prompts often fail to make the model truly "see" through a persona's specific interests. This paper introduces persona lenses—textual, visual, and weight-based interventions—that significantly boost a model's ability to mimic persona-specific attention, revealing that while larger models follow personas better naturally, smaller models can be "steered" effectively using these novel techniques.

What Problem Does This Paper Address?

Understanding where users look on a web page is the "holy grail" of UI/UX design, helping designers optimize layouts before they even launch. While human eye-tracking studies are the gold standard, they are too expensive and slow to run for every possible user group and page iteration.

Recent Vision-Language Models (VLMs) like Qwen-VL offer a promising alternative: simulating gaze by giving the model a persona (e.g., "a sports fanatic") and a screenshot. However, the authors found a "reach" problem: a persona in a text prompt only partially influences the simulated gaze. A 4B-parameter model might acknowledge the persona in text but still look at the page like a generic user. This paper asks: How can we make a VLM truly see through its persona’s eyes?

Methods

The authors propose and compare three "persona lenses" that sit between the model and the page:

  1. Text Lens (Two-Step Prompting): Instead of one jump to gaze prediction, the model first writes a "note" describing what the persona would likely find interesting on the specific page. This note is then fed back into the prompt for the final gaze simulation.
  2. Pixel Lens (Visual Prompting): This lens modifies the image itself. Using a SigLIP2 image-text encoder, the system identifies regions matching the persona's interests and highlights them with red boxes (for likes) or translucent gray (for dislikes) before the VLM sees the page.
  3. Weight Lens (Reinforcement Learning): The authors used Group Relative Policy Optimization (GRPO) to train LoRA adapters. The model is rewarded for "dwelling" longer on sections that align with its persona, while a "naturalness gate" ensures the resulting scanpath still looks like human-like browsing rather than a robotic jump to a single target.
Figure 1. Overview. A vision–language model receives a page screenshot and one of two persona versions that differ in one interest, and returns a simulated scanpath. The measure is the target-section dwell share of the Passionate version minus that of the Averse version. Three persona lenses and model scale change this gaze. The text lens (dashed) calls the same model twice on the screenshot, first for a note on what this persona would look at, then for the scanpath with the note added to the base prompt. The pixel lens encodes the page and each interest with SigLIP2, finds the interest’s region as a heat-map blob and draws the marker of a liked region in red. The weight lens is a reward on target dwell. The lenses act only on the scanpath prediction, and the note is always written without the pixel lens.

Datasets

  • Personas: 30 synthetic profiles drawn from the MatrAIx corpus, each defined by ~23 attributes.
  • Stimuli: 1280 × 800 screenshots with Set-of-Mark (SoM) tags.
  • Generated Pages: 12 controlled pages (portals, shops, news) with clearly labeled sections.
  • Live Pages: A "Live-page evaluation set" consisting of 87 cells across 61 screenshots of 19 real-world sites (e.g., BBC, MSN, Amazon).
  • Human Baseline: The UEyes dataset was used to evaluate the "human-likeness" of the simulated scanpaths.

Key Insights and Ideas

  • The Scale Effect: Larger models are naturally better "persona followers." A 32B model with a simple prompt outperforms a 4B model using a text lens.
  • The "Reach" of Text: The Text Lens is powerful but "brittle." It can cause the model to react to gaze-irrelevant attributes (e.g., changing gaze patterns based on a persona's sleep schedule or knitting hobby, which shouldn't affect how they look at a sports section).
  • Visual Robustness: The Pixel Lens is the most robust. Because it only highlights regions based on explicit interests, it isn't distracted by irrelevant persona text. However, its effectiveness is capped by the quality of the underlying region-finding encoder.
  • Internal vs. External Attention: Interestingly, at the 4B scale, the Text Lens changes the output (the reported dwell time) without significantly changing the model's internal late-layer attention. It’s as if the model "knows" it should say it's looking at sports, even if its internal "eyes" are still wandering.

Key Results

The primary metric is the pole difference: the gap in dwell time share on a target section between a "Passionate" persona and an "Averse" persona.

  • Base Performance: On sports content, the 4B model had a tiny pole difference of .076, while the 32B model reached .296 with the same prompt.
  • Text Lens Impact: Adding the Text Lens to the 4B model boosted its pole difference to .39, effectively making it "punch above its weight class" and behave more like the 32B model.
  • Weight Lens Transfer: The Weight Lens (RL-trained adapter) showed impressive generalization. Adapters trained on certain interests successfully transferred their "persona-following" behavior to entirely unseen interests (like fashion and travel) held out from training.
Figure 4. The lenses on one live page (BBC, sports), each alone on the base prompt. The weight lens is a relaxed-gate adapter whose reward covers sports and whose training uses these 30 personas, so panel (d) is not held out (Tab. 3). Color: Passionate minus Averse dwell share per element (30 personas, 2 seeds); ∆: target-section pole difference.

Limitations and Future Work

  • Ground Truth Gap: There is currently no public dataset that labels human gaze with detailed personas. Consequently, "human-likeness" can only be checked against generic, unlabeled viewers.
  • Encoder Dependency: The Pixel Lens is only as good as the SigLIP2 encoder's ability to find regions. It struggled with "technology" sections compared to "sports" or "finance."
  • Caricature Risk: Strong reinforcement learning (the Weight Lens) can sometimes cause "scanpath collapse," where the model becomes a caricature of a persona, staring obsessively at one element and losing the natural flow of human browsing. Future work could focus on better "naturalness" rewards to prevent this.
Figure 5. Scale on generated pages. Each panel is one interest, and the bars give the pole difference (persona effect) at 4B, 9B and 32B without a lens (gray) and the increment the pixel lens adds (blue, starred at one-sided paired p < .05). The increment is significant in 8 of 9 entries and does not grow from 4B to 32B.

Summary updated

Abstract

Vision-language models have been used to simulate where a user looks on a web page. However, a persona in the prompt reaches the simulated gaze only partly. We measure this reach as the difference in one model's gaze between two MatrAIx personas that differ in a single attribute. We introduce persona lenses, which place the persona between the model and the page: a text lens (a two-step prompt in which the model first writes what the persona would look at), a pixel lens (marks on the region matching the persona's interests) and a weight lens (a dwell reward gated to keep the scanpath natural). Using Qwen3-VL (4B and 32B) and Qwen3.5-9B on generated and live pages, we compare the lenses with one another and with scale, the 32B model with no lens. It has been confirmed that the text lens separates the personas strongly but is not robust to profile changes that should not move gaze. The pixel lens separates the personas less but is unaffected by attributes unrelated to gaze. At 4B and 9B, the weight lens separates the personas on interests held out from training, with no extra step on a new page. The 32B model separates the personas with the base prompt alone. Therefore, the choice of a lens depends on cost and model size.
BibTeX
@misc{shin2026persona,
  author = {Yoonho Shin},
  title = {{Persona Lenses: What Makes a Vision-Language Model See Through Its Persona's Eyes?}},
  year = {2026},
  eprint = {2611.25508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25508},
}
Figure