Introduction: The Challenge of GUI Hallucinations
Large Vision-Language Models (LVLMs) such as LLaVA and GPT-4V have demonstrated remarkable capabilities in general image understanding. However, when applied to Graphical User Interfaces (GUIs), these models frequently suffer from "text-bias." Because they are pre-trained on vast corpora of text, they often prioritize internal "common sense" or prior knowledge over the actual pixels on the screen. For instance, if asked how to "like" a video, a model might suggest clicking a heart icon at the bottom right simply because that is a common convention, even if the specific app being viewed places the icon elsewhere. This phenomenon, known as hallucination, renders current models unreliable for autonomous GUI agents.
Methodology: Grounding Vision in Language
To address these shortcomings, the VGA (Vision GUI Assistant) framework introduces two primary innovations designed to force the model to prioritize visual evidence over linguistic priors.
The Referent Method
The Referent Method is a data construction technique that ensures model responses explicitly include visual "referents." These include coordinates, shapes, colors, and relative spatial positions (e.g., "the blue button below the text field"). By anchoring linguistic descriptions to specific visual regions, the model learns a direct mapping between text and pixels, reducing the likelihood of generic, hallucinated responses.
Foundation and Advanced Comprehension (FAC) Training
VGA employs a two-stage training process to decouple visual grounding from complex reasoning:
- Foundation Stage: The model is trained on simple, fixed-format tasks, such as identifying bounding boxes or basic element functions. This stage "freezes" the conversational style, forcing the model to focus purely on feature extraction from the image.
- Advanced Comprehension Stage: Building on the grounded knowledge from the first stage, the model is then trained on multi-turn dialogues and complex reasoning tasks. This ensures that the model's conversational abilities are built upon a solid foundation of visual accuracy.
Datasets and Evaluation
The researchers utilized the Rico Dataset, containing 66,000 unique Android UI screens, to build the VGA Dataset. This new dataset consists of 63,800 high-quality Vision Question Answering (VQA) examples. To ensure accuracy, the team used text-only GPT-4 to extract logic from the UI code (view hierarchy) and vision-based GPT-4o to describe visual elements like color and shape. Performance was evaluated using GUI Bench, a custom benchmark of 22 images and 44 complex user questions designed to test real-world GUI understanding and reasoning.
Results and Key Insights
VGA-7b-v1 achieved state-of-the-art performance, scoring 90.83 on the GUI Bench. This significantly outperformed the base LLaVA model (64.09) and surpassed proprietary models like GPT-4o (80.75) and GPT-4V (81.82).
Key findings include:
- Overcoming Text Bias: Hallucinations in GUI tasks are often "logical" guesses based on text pre-training. VGA successfully overrides these priors with visual evidence.
- Efficiency: A "low-res" version of the model (VGA-7b-v1*) maintained strong macro-level understanding while using only 25% of the image tokens, offering a path toward faster real-time GUI agents.
- Attention Alignment: Heatmap analysis confirmed that VGA’s attention is more tightly focused on relevant UI elements than baseline models, which often exhibit scattered attention.
Conclusion and Future Work
VGA demonstrates that image-centric fine-tuning and a staged training approach can effectively mitigate hallucinations in GUI comprehension. While the current model is constrained by the age of the Rico dataset and the limitations of its base architecture (LLaVA-v1.6-Mistral-7B), it sets a new standard for open-source GUI assistants. Future work will focus on integrating modern UI styles and leveraging more powerful base models like Llama-3 to further enhance reasoning capabilities.
Summary updated