Focused Omni-Modal GUI Agents

Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, Wei Yang Bryan LimarXiv 2026 · arXiv

TL;DR

FocusGUI is a lightweight, zero-initialized residual adapter designed to solve "Sequential Distraction" in omni-modal GUI agents. By dynamically prioritizing relevant input channels—such as audio, video, or device state—at each interaction step, it prevents agents from drifting off-course due to irrelevant context, achieving a 3.0× improvement in Exact Match accuracy over frozen backbones.

What Problem Does This Paper Address?

Modern Graphical User Interface (GUI) agents are no longer limited to static screenshots; they now operate in "omni-modal" environments, processing streaming video, audio, interaction history, and device states. While more data sounds better, it introduces a significant challenge: Sequential Distraction.

At any specific moment in a multi-step task, the "decisive evidence" needed for the next action is often transient and sparse. For example, a notification chime (audio) might be the only reason to switch apps, while the rest of the visual screen is mere clutter. If an agent fails to filter this noise, it makes a sub-optimal choice. Because GUI agents operate in a closed loop—where their actions change the environment and feed back into the next step's context—these small "misreads" compound, leading to irreversible trajectory drift.

Relevance is transient across steps and sparse within them. (a) The dominant modality switches frequently. (b) Most steps contain interfering modalities. (c) Decisive evidence occupies a tiny fraction of the input.

Methods

The authors propose FocusGUI, a lightweight adapter that implements "Focused Perception." Instead of changing what the model sees (like cropping the image), it changes how the model reads the information.

Key components of the architecture include:

  1. Step-Adaptive Prioritization: The observation is decomposed into separable "perceptual channels" (e.g., screen, audio, video, history). FocusGUI re-computes a "decision reference" at every step to score and weight these channels based on their current relevance.
  2. Compact Focus Latent: The prioritized evidence is compressed into a low-dimensional latent representation ($z_t$). This ensures the adapter only carries a summary of what to emphasize, rather than trying to reconstruct the entire observation.
  3. Residual Alignment: This latent is projected into the frozen decoder's embedding space and added as a zero-initialized residual to the positions immediately preceding the action boundary.

Crucially, the underlying Large Multimodal Model (LMM) backbone remains completely frozen. FocusGUI is trained using only next-action supervision; it requires no manual labels for which modality is "important" at any given time.

FocusGUI Architecture: The frozen backbone's original pathway is left intact, while the adapter pathway dynamically prioritizes channels and injects a focus signal via residual alignment.

Datasets

The researchers evaluated their method on OmniGUI, a benchmark for omni-modal smartphone interactions.

  • Scope: 29 different applications.
  • Scale: 708 episodes comprising 2,572 action steps.
  • Modalities: Each step includes a screenshot, synchronous audio, video, and action history.

Key Insights and Ideas

  • Relevance is a Step-Level Property: The "dominant" modality (the one most critical for the correct action) switches in 45.3% of adjacent steps. A static weighting of inputs is fundamentally insufficient for complex GUI tasks.
  • Screen Refinement can Backfire: Existing methods that "zoom in" or crop the screen often lose the surrounding context needed to ground coordinates or read text parameters. FocusGUI avoids this by keeping the full observation available to the backbone while using the adapter to guide attention.
  • Priorities Shift Over Time: As shown in Figure 4, "Interaction State" is highly prioritized at the start of an episode, while streaming modalities like "Audio" and "Video" gain priority in later phases as the task evolves.
  • Solving the "Wait" Problem: One of the hardest tasks for GUI agents is knowing when not to act. FocusGUI excels at "temporal reasoning"—waiting for a specific audio cue before clicking—whereas standard backbones often act prematurely based on visual saliency alone.
Case study: FocusGUI correctly waits for an audio cue (engine sounds) before tapping, while the frozen backbone acts prematurely on visual salience and fails to recover.

Key Results

FocusGUI was tested across four major omni-modal backbones: Qwen2.5-Omni, VITA-1.5, MiniCPM-o 4.5, and Baichuan-Omni-1.5.

  • Massive Accuracy Boost: On average, it raised Type Match (correct action primitive) by +33.2 points and Exact Match (correct action + correct arguments/coordinates) by 3.0× over the frozen backbones.
  • Resistance to Drift: While frozen models see their accuracy plummet as trajectories lengthen (losing half their accuracy by step 4), FocusGUI maintains high performance, retaining 86.8% of its initial accuracy.
  • Efficiency: The adapter is extremely lightweight. For Qwen2.5-Omni, FocusGUI adds only 1.445M trainable parameters—just 0.0162% of the total model size.
Performance across interaction depth: FocusGUI (dark blue) maintains accuracy as steps increase, while the frozen backbone (light blue) suffers from compounding errors.

Limitations and Future Work

  • Reactive vs. Proactive: The current mechanism prioritizes channels based on the present observation. The authors suggest that future work could incorporate "anticipatory signals" to prepare the agent for evidence that might become important in upcoming steps.
  • Generalization: While tested across four backbone families, the authors note that extending this "focused perception" to even broader computer-use settings (like desktop environments with even more complex multi-window contexts) is a natural next step.

Summary updated

Abstract

Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.
BibTeX
@misc{zhang2026focused,
  author = {Fuyao Zhang and Wenwen He and Jiaming Zhang and Yurong Hao and Xuan Wang and Guowei Guan and Che Wang and Longtao Huang and Wei Yang Bryan Lim},
  title = {{Focused Omni-Modal GUI Agents}},
  year = {2026},
  eprint = {2610.26203},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26203},
}
Figure