Abstract
Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms. While these agents hold great promise for advancing digital automation, their potential for unsafe operations, such as system compromise and privacy leakage, is raising significant concerns. Detecting these safety concerns across the vast and complex operational space of mobile environments presents a formidable challenge that remains critically underexplored. To establish a foundation for mobile agent safety research, we introduce MobileRisk-Live, a dynamic sandbox environment accompanied by a safety detection benchmark comprising realistic trajectories with fine-grained annotations. Built upon this, we propose OS-Sentinel, a novel hybrid safety detection framework that synergistically combines a Formal Verifier for detecting explicit system-level violations with a VLM-based Contextual Judge for assessing contextual risks and agent actions. Experiments show that OS-Sentinel achieves 10%-30% improvements over existing approaches across multiple metrics. Further analysis provides critical insights that foster the development of safer and more reliable autonomous mobile agents.
Introduction
As Vision-Language Models (VLMs) increasingly power autonomous mobile agents, these systems are gaining the ability to navigate complex application interfaces, handle sensitive user data, and modify critical system settings. While this autonomy enables sophisticated digital assistance, it introduces significant safety risks. Agents may inadvertently leak private information, perform destructive system operations, or generate harmful content.
Current safety detection methods face a dual challenge. Rule-based systems are often too rigid to understand the semantic context of an action (e.g., distinguishing between a legitimate email and a phishing attempt), while VLM-based "judges" are typically limited to visual GUI analysis, missing "invisible" background system changes like unauthorized file modifications or background API calls. To address these gaps, we propose OS-Sentinel, a hybrid framework that bridges low-level system monitoring with high-level semantic reasoning.
OS-Sentinel: Hybrid Safety Detection
The OS-Sentinel framework employs a dual-layer defense strategy to ensure comprehensive safety coverage. It evaluates agent behavior at both the individual step level and the overall trajectory level.
Dual-Layer Validation
- Formal Verifier (Low-Level): This component monitors the System State Trace. By utilizing SHA256 cryptographic hashes, it detects unauthorized file system changes. It also employs keyword matching and regular expressions (regex) to identify the leakage of Personally Identifiable Information (PII), such as credit card numbers or passwords, which may not be visible on the GUI.
- Contextual Judge (High-Level): A VLM-based evaluator analyzes the agent's trajectory, including screenshots and actions. It is designed to understand user intent and social norms, catching nuanced risks like social engineering or inappropriate content that rule-based systems would overlook.
The system issues an "unsafe" verdict if either the Formal Verifier or the Contextual Judge flags a step, ensuring a robust safety guardrail.
MobileRisk-Live and Benchmark
To facilitate research in this area, we introduce two major resources:
- MobileRisk-Live: A dynamic Android emulator-based sandbox that allows agents to interact with 48 different applications (e.g., Google Maps, Gmail, WeChat) while recording both GUI frames and background system metadata.
- MobileRisk Benchmark: A curated dataset of 204 trajectories (102 safe, 102 unsafe) with fine-grained annotations. It covers 10 safety categories, including Privacy Violations, Destructive Operations, and Over-Automation. Notably, the benchmark includes "counterpart safe cases"—modified versions of unsafe trajectories where the violation was corrected—to rigorously test for false positives.
Experimental Results
Our experiments demonstrate that OS-Sentinel consistently outperforms existing baselines, such as standalone GPT-4o-as-a-Judge or rule checkers.
- Performance Gains: OS-Sentinel achieved a 10% to 30% improvement across multiple metrics using various backbones (GPT-4o, Claude 3.7, Qwen2.5-VL).
- Balanced Detection: The framework maintained high detection rates across all 10 safety categories, whereas rule-based evaluators struggled with "Over-Automation" and "Harmful Content."
- Efficiency: With an average latency of approximately 66ms per detection, the framework is suitable for real-world deployment as a real-time safety monitor.
Conclusion and Future Work
OS-Sentinel provides a scalable and effective solution for enhancing the safety of mobile GUI agents. By combining formal verification of system states with contextual VLM reasoning, it addresses both explicit and implicit risks. Future work will focus on adapting the Formal Verifier to closed ecosystems like iOS and reducing environment noise in live sandboxes. Ultimately, we aim to transition from external detection toward "Safety-by-Design," where these insights are used to train inherently safer autonomous agents.
Summary updated