GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents

Zheng Wu, Pengzhou Cheng, Zongru Wu, Lingzhong Dong, Zhuosheng Zhang

Introduction

Graphical User Interface (GUI) agents are increasingly capable of navigating complex digital environments. However, they remain brittle when faced with instructions outside their training distribution. This paper introduces GEM (Gaussian Embedding Modeling), a lightweight, plug-and-play method designed to detect out-of-distribution (OOD) tasks, preventing catastrophic failures like accidental device resets.

Overview of OOD detection challenges and the proposed GEM-enabled agent workflow.

The Challenge of OOD in GUI Agents

GUI agents typically encounter two types of OOD scenarios:

  1. Internalization-OOD: Instructions that are impossible within the current environment (e.g., launching a tool that doesn't exist).
  2. Extrapolation-OOD: Tasks involving entirely new applications or domains (e.g., finding a friend list in an unsupported app).

Traditional uncertainty-based metrics, such as output entropy or confidence scores, often fail in these contexts. Because GUI screens are visually dense and the reasoning paths for valid and invalid tasks are similar, models often "hallucinate" actions rather than signaling uncertainty.

The GEM Methodology

GEM shifts the detection focus from the model's output to its internal representations. The method follows a five-step pipeline:

  1. Embedding Extraction: Extracting input embeddings from the MLLM encoder.
  2. Centroid Mapping: Calculating a global centroid for In-Distribution (ID) training data.
  3. Distance Modeling: Computing the L2-norm distance of embeddings from the centroid.
  4. Gaussian Mixture Model (GMM): Fitting a GMM to the distance distribution, using the Bayesian Information Criterion (BIC) to determine the optimal number of clusters.
  5. Boundary Definition: Flagging instructions as OOD if their distance falls outside a set number of standard deviations from the GMM centers.

Key Insights

  • Distance-Clustering: ID samples form distinct clusters based on their distance from the centroid. Interestingly, samples further from the centroid often have higher success rates, as they represent more distinct, less ambiguous tasks.
  • Optimal Layer Selection: Analysis across datasets revealed that the 9th layer of the encoder typically provides the most effective features for OOD detection, balancing general visual-textual data with task-specific semantics.
  • Architecture Sensitivity: Standard Vision Transformers (ViTs) with global attention (like Qwen2-VL-2B) were found to be more effective for OOD detection than newer window-based attention models, likely due to a better capture of the model's capability boundaries.

Results and Performance

GEM demonstrates significant improvements over traditional baselines. On average, it boosts OOD detection accuracy by 23.70% across eight diverse datasets spanning smartphone, computer, and web environments.

Performance comparison of GEM against baseline OOD detection methods across smartphone and computer platforms.

Beyond detection accuracy, GEM is highly efficient, adding only 4.9% to training time and 6.5% to testing time. When integrated into an end-to-end system where OOD detection triggers a "call for help" to a more powerful model (e.g., GPT-4o), the agent's step-wise success rate improved by 9.40%.

Conclusion

GEM provides a robust, efficient solution for the "hallucination" problem in GUI agents. By modeling the internal embedding space rather than relying on output uncertainty, it allows agents to recognize their own limitations. Future work will explore GEM's performance in dynamic, real-world environments and its potential integration into reinforcement learning loops.

Summary updated

BibTeX
@inproceedings{wu2026gem,
  author = {Zheng Wu and Pengzhou Cheng and Zongru Wu and Lingzhong Dong and Zhuosheng Zhang},
  title = {{GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i40.40692},
}
Figure