INREACT: An Inspire-Then-Reinforce Training Framework For Multimodal GUI Agent

Yuanlei Wang, Liuzhou Zhang, Haohao Luo, Ying ShenEMNLP 2025

TL;DR

This paper introduces INREACT, a novel framework for training multimodal GUI agents that can understand user instructions and interact with graphical interfaces. INREACT uses a two-stage training approach: curriculum learning to build foundational skills and reinforcement learning with a custom reward function to refine precise actions, significantly improving performance on complex GUI tasks.

What Problem Does This Paper Address?

Interacting with graphical user interfaces (GUIs) is a fundamental aspect of using digital devices. Developing intelligent GUI agents that can execute user instructions to perform tasks on these interfaces is a challenging but valuable goal. While recent advancements in multimodal large language models (MLLMs) have shown promise, existing GUI agents often struggle with two key issues: precisely locating small or visually similar elements within a dense interface, and performing complex, multi-step reasoning to achieve a goal. This paper aims to address these limitations by proposing a structured training framework that enhances both the precision of element localization and the agent's reasoning capabilities.

Methods

INREACT employs a two-stage training framework:

  • Curriculum Learning: This stage systematically builds the agent's core capabilities in a progressive manner. It's divided into three sub-stages:
  • Perception: Focuses on understanding GUI layouts, spatial awareness, and element recognition through tasks like GUI-based Question Answering and Interface Description Generation.
  • Grounding: Teaches the agent to connect user instructions with specific interface elements by predicting coordinates from instructions and generating descriptions from coordinates.
  • Reasoning: Introduces an explicit "observe-think-act" process, moving beyond single-step predictions to support multi-step decision-making.
  • Reinforcement Learning (RL): This stage refines the agent's action accuracy and grounding robustness. INREACT uses a GRPO-based optimization framework with a novel, rule-based reward function. This reward function is designed to provide dense, informative feedback by combining:
  • Action-Type Reward: A binary reward for correctly predicting the action type (e.g., click, type).
  • Distance-Aware Coordinate Reward: This component includes an accuracy reward (checking if the predicted point is within the target bounding box) and a precision reward that penalizes the normalized Euclidean distance to the center of the target box. This encourages fine-grained localization.

The overall agent workflow follows a ReAct-like pattern, integrating observation, thinking, and acting.

Datasets

The paper evaluates INREACT on several datasets for both GUI grounding and navigation tasks:

  • GUI Grounding:
    • ScreenSpot (Cheng et al., 2024): A dataset with common interface elements across mobile, desktop, and web platforms.
  • ScreenSpot-Pro (Li et al., 2025b): A more challenging dataset featuring 23 professional applications with high-resolution and complex layouts.
  • GUI Navigation:
    • Mind2Web (Deng et al., 2023): A web-based dataset for evaluating generalist agents performing complex tasks via language instructions.
  • AndroidControl (Li et al., 2024): A mobile dataset for testing multi-step task execution in realistic Android environments.

The paper also mentions various sources for training data, including GUIChat, ScreenQA, MultiUI, AutoGUI, SeeClick, GUIEnv, Aguvis-stage2, and OmniAct.

Key Insights and Ideas

  • Curriculum Learning is Crucial: The staged approach, moving from basic perception to grounding and then reasoning, is essential. Removing any stage significantly degrades performance, highlighting the importance of building skills progressively.
  • Reinforcement Learning with a Smart Reward: The proposed piecewise reward function, which combines action-type correctness with distance-aware spatial precision, is more effective than simple binary or linear rewards for improving fine-grained localization in GUIs.
  • Structured Reasoning Matters: The "think" component in the observe-think-act cycle is critical for formulating effective plans and making robust decisions, even more so than the observation phase itself in complex scenarios.
  • Real-World GUI Logic: The piecewise reward function's design, with discrete steps and a focus on central clicks, better reflects the practical nature of GUI interactions than simpler reward schemes.

Key Results

  • Superior Performance: INREACT consistently outperforms existing methods across four datasets for both GUI grounding and navigation tasks.
  • Grounding: On ScreenSpot, INREACT achieves an average accuracy of 77.8%, surpassing strong baselines. On the more challenging ScreenSpot-Pro, it achieves an overall accuracy of 20.7%.
  • Navigation: On Mind2Web, INREACT shows strong performance across cross-task, cross-website, and cross-domain generalization splits. On AndroidControl, it achieves 83.1 on Low-Level and 68.1 on High-Level tasks, outperforming prior baselines.
  • Ablation Studies: Removing curriculum learning stages led to significant performance drops. Reinforcement learning with the proposed reward function improved performance over binary or linear rewards. Removing the "thought" component had a more pronounced negative impact than removing the "observation" component.

Limitations and Future Work

  • Limited Action Space Coverage: The current action space (click, type, scroll) is insufficient for the full diversity of real-world GUI operations (e.g., drag-and-drop, multi-touch gestures). Future work could extend this action space.
  • Inference Speed and Efficiency: The structured reasoning process can introduce computational overhead, potentially affecting real-time interactions. Optimizing inference speed through model compression or caching is suggested.
  • Handling Dynamic Interfaces: The framework primarily works with static interfaces. Incorporating temporal modeling or environment change detection would be valuable for handling dynamic GUI updates.
  • Reward Function Tuning: The thresholds for the piecewise reward function were chosen empirically and may not be optimal; further tuning is left for future work.

Summary updated

BibTeX
@inproceedings{wang2025inreact,
  author = {Yuanlei Wang and Liuzhou Zhang and Haohao Luo and Ying Shen},
  title = {{INREACT: An Inspire-Then-Reinforce Training Framework For Multimodal GUI Agent}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure