TL;DR
FocusGUI is a lightweight, zero-initialized residual adapter designed to solve "Sequential Distraction" in omni-modal GUI agents. By dynamically prioritizing relevant input channels—such as audio, video, or device state—at each interaction step, it prevents agents from drifting off-course due to irrelevant context, achieving a 3.0× improvement in Exact Match accuracy over frozen backbones.
What Problem Does This Paper Address?
Modern Graphical User Interface (GUI) agents are no longer limited to static screenshots; they now operate in "omni-modal" environments, processing streaming video, audio, interaction history, and device states. While more data sounds better, it introduces a significant challenge: Sequential Distraction.
At any specific moment in a multi-step task, the "decisive evidence" needed for the next action is often transient and sparse. For example, a notification chime (audio) might be the only reason to switch apps, while the rest of the visual screen is mere clutter. If an agent fails to filter this noise, it makes a sub-optimal choice. Because GUI agents operate in a closed loop—where their actions change the environment and feed back into the next step's context—these small "misreads" compound, leading to irreversible trajectory drift.
Methods
The authors propose FocusGUI, a lightweight adapter that implements "Focused Perception." Instead of changing what the model sees (like cropping the image), it changes how the model reads the information.
Key components of the architecture include:
- Step-Adaptive Prioritization: The observation is decomposed into separable "perceptual channels" (e.g., screen, audio, video, history). FocusGUI re-computes a "decision reference" at every step to score and weight these channels based on their current relevance.
- Compact Focus Latent: The prioritized evidence is compressed into a low-dimensional latent representation ($z_t$). This ensures the adapter only carries a summary of what to emphasize, rather than trying to reconstruct the entire observation.
- Residual Alignment: This latent is projected into the frozen decoder's embedding space and added as a zero-initialized residual to the positions immediately preceding the action boundary.
Crucially, the underlying Large Multimodal Model (LMM) backbone remains completely frozen. FocusGUI is trained using only next-action supervision; it requires no manual labels for which modality is "important" at any given time.
Datasets
The researchers evaluated their method on OmniGUI, a benchmark for omni-modal smartphone interactions.
- Scope: 29 different applications.
- Scale: 708 episodes comprising 2,572 action steps.
- Modalities: Each step includes a screenshot, synchronous audio, video, and action history.
Key Insights and Ideas
- Relevance is a Step-Level Property: The "dominant" modality (the one most critical for the correct action) switches in 45.3% of adjacent steps. A static weighting of inputs is fundamentally insufficient for complex GUI tasks.
- Screen Refinement can Backfire: Existing methods that "zoom in" or crop the screen often lose the surrounding context needed to ground coordinates or read text parameters. FocusGUI avoids this by keeping the full observation available to the backbone while using the adapter to guide attention.
- Priorities Shift Over Time: As shown in Figure 4, "Interaction State" is highly prioritized at the start of an episode, while streaming modalities like "Audio" and "Video" gain priority in later phases as the task evolves.
- Solving the "Wait" Problem: One of the hardest tasks for GUI agents is knowing when not to act. FocusGUI excels at "temporal reasoning"—waiting for a specific audio cue before clicking—whereas standard backbones often act prematurely based on visual saliency alone.
Key Results
FocusGUI was tested across four major omni-modal backbones: Qwen2.5-Omni, VITA-1.5, MiniCPM-o 4.5, and Baichuan-Omni-1.5.
- Massive Accuracy Boost: On average, it raised Type Match (correct action primitive) by +33.2 points and Exact Match (correct action + correct arguments/coordinates) by 3.0× over the frozen backbones.
- Resistance to Drift: While frozen models see their accuracy plummet as trajectories lengthen (losing half their accuracy by step 4), FocusGUI maintains high performance, retaining 86.8% of its initial accuracy.
- Efficiency: The adapter is extremely lightweight. For Qwen2.5-Omni, FocusGUI adds only 1.445M trainable parameters—just 0.0162% of the total model size.
Limitations and Future Work
- Reactive vs. Proactive: The current mechanism prioritizes channels based on the present observation. The authors suggest that future work could incorporate "anticipatory signals" to prepare the agent for evidence that might become important in upcoming steps.
- Generalization: While tested across four backbone families, the authors note that extending this "focused perception" to even broader computer-use settings (like desktop environments with even more complex multi-window contexts) is a natural next step.
Summary updated