TL;DR
Vision-Language Models (VLMs) can simulate user gaze on web pages, but standard text prompts often fail to make the model truly "see" through a persona's specific interests. This paper introduces persona lenses—textual, visual, and weight-based interventions—that significantly boost a model's ability to mimic persona-specific attention, revealing that while larger models follow personas better naturally, smaller models can be "steered" effectively using these novel techniques.
What Problem Does This Paper Address?
Understanding where users look on a web page is the "holy grail" of UI/UX design, helping designers optimize layouts before they even launch. While human eye-tracking studies are the gold standard, they are too expensive and slow to run for every possible user group and page iteration.
Recent Vision-Language Models (VLMs) like Qwen-VL offer a promising alternative: simulating gaze by giving the model a persona (e.g., "a sports fanatic") and a screenshot. However, the authors found a "reach" problem: a persona in a text prompt only partially influences the simulated gaze. A 4B-parameter model might acknowledge the persona in text but still look at the page like a generic user. This paper asks: How can we make a VLM truly see through its persona’s eyes?
Methods
The authors propose and compare three "persona lenses" that sit between the model and the page:
- Text Lens (Two-Step Prompting): Instead of one jump to gaze prediction, the model first writes a "note" describing what the persona would likely find interesting on the specific page. This note is then fed back into the prompt for the final gaze simulation.
- Pixel Lens (Visual Prompting): This lens modifies the image itself. Using a SigLIP2 image-text encoder, the system identifies regions matching the persona's interests and highlights them with red boxes (for likes) or translucent gray (for dislikes) before the VLM sees the page.
- Weight Lens (Reinforcement Learning): The authors used Group Relative Policy Optimization (GRPO) to train LoRA adapters. The model is rewarded for "dwelling" longer on sections that align with its persona, while a "naturalness gate" ensures the resulting scanpath still looks like human-like browsing rather than a robotic jump to a single target.
Datasets
- Personas: 30 synthetic profiles drawn from the MatrAIx corpus, each defined by ~23 attributes.
- Stimuli: 1280 × 800 screenshots with Set-of-Mark (SoM) tags.
- Generated Pages: 12 controlled pages (portals, shops, news) with clearly labeled sections.
- Live Pages: A "Live-page evaluation set" consisting of 87 cells across 61 screenshots of 19 real-world sites (e.g., BBC, MSN, Amazon).
- Human Baseline: The UEyes dataset was used to evaluate the "human-likeness" of the simulated scanpaths.
Key Insights and Ideas
- The Scale Effect: Larger models are naturally better "persona followers." A 32B model with a simple prompt outperforms a 4B model using a text lens.
- The "Reach" of Text: The Text Lens is powerful but "brittle." It can cause the model to react to gaze-irrelevant attributes (e.g., changing gaze patterns based on a persona's sleep schedule or knitting hobby, which shouldn't affect how they look at a sports section).
- Visual Robustness: The Pixel Lens is the most robust. Because it only highlights regions based on explicit interests, it isn't distracted by irrelevant persona text. However, its effectiveness is capped by the quality of the underlying region-finding encoder.
- Internal vs. External Attention: Interestingly, at the 4B scale, the Text Lens changes the output (the reported dwell time) without significantly changing the model's internal late-layer attention. It’s as if the model "knows" it should say it's looking at sports, even if its internal "eyes" are still wandering.
Key Results
The primary metric is the pole difference: the gap in dwell time share on a target section between a "Passionate" persona and an "Averse" persona.
- Base Performance: On sports content, the 4B model had a tiny pole difference of .076, while the 32B model reached .296 with the same prompt.
- Text Lens Impact: Adding the Text Lens to the 4B model boosted its pole difference to .39, effectively making it "punch above its weight class" and behave more like the 32B model.
- Weight Lens Transfer: The Weight Lens (RL-trained adapter) showed impressive generalization. Adapters trained on certain interests successfully transferred their "persona-following" behavior to entirely unseen interests (like fashion and travel) held out from training.
Limitations and Future Work
- Ground Truth Gap: There is currently no public dataset that labels human gaze with detailed personas. Consequently, "human-likeness" can only be checked against generic, unlabeled viewers.
- Encoder Dependency: The Pixel Lens is only as good as the SigLIP2 encoder's ability to find regions. It struggled with "technology" sections compared to "sports" or "finance."
- Caricature Risk: Strong reinforcement learning (the Weight Lens) can sometimes cause "scanpath collapse," where the model becomes a caricature of a persona, staring obsessively at one element and losing the natural flow of human browsing. Future work could focus on better "naturalness" rewards to prevent this.
Summary updated