Introduction
Most AI agents designed to operate smartphones are "digital ghosts." They live inside the operating system, relying on the Android Debug Bridge (ADB) to "see" the screen via XML metadata and "act" via system commands. While efficient in a lab, this approach is brittle: it is locked to Android, requires "developer mode" (a major security risk), and ignores the physical realities of the world—such as screen glare or the need to physically move a phone to scan a QR code.
Researchers from UCL, Huawei Noah’s Ark Lab, and other institutions have introduced See-Control, a framework that moves smartphone interaction from software bridges to physical manipulation. By using a low-cost robotic arm and a vision-only Multimodal Large Language Model (MLLM), they have created a system that can operate any smartphone—iPhone or Android—through direct physical interaction, just like a human.
The Problem: The Limitations of Digital-Only Agents
The "last mile" of home assistance often involves a smartphone, whether it is approving a two-factor authentication prompt, navigating an IoT app, or coordinating a delivery. Existing agents are limited by their reliance on ADB, which creates three major hurdles:
- Platform Lock-in: They do not work on iOS or other closed systems.
- Privacy & Security: ADB requires high-level privileges and tunnels sensitive data through debugging channels.
- Physical Blindness: Digital-only agents cannot handle physical constraints like screen glare or touch friction.
Methodology: Embodied Smartphone Operation (ESO)
The researchers framed the problem as Embodied Smartphone Operation (ESO). The See-Control framework consists of three main pillars:
The MLLM Agent and Pipeline
Using GPT-4o as a backbone, the agent treats the phone as a visual-tactile object. It receives a raw screenshot via screen mirroring and generates a reasoning chain before deciding on an action. The system uses Grounding DINO for icon detection and OCR for text recognition. It employs a "Visual Prompting" (Set-of-Mark) approach, overlaying index numbers on detected icons to help the MLLM choose the right target.
Physical Action Space
Instead of system commands, the agent outputs continuous control for a 3-axis robotic arm. It performs five basic actions: Tap, Swipe, Text (typing), Back, and Exit. Notably, "Back" and "Exit" are performed using standard physical gestures (like swiping from the edge) rather than software buttons. To type without ADB, the system uses an "anchor point" method to map the keyboard layout based on the first two keys in a row.
Datasets and Benchmarks
The authors introduced two major resources to the community:
- ESO Benchmark: A suite of 155 manually annotated tasks across 27 popular apps (Facebook, Gmail, Amazon, etc.), split into "Standard" and "Challenging" categories.
- Operation Episode Dataset: A labeled dataset of successful and unsuccessful operation sequences, including screenshots, reasoning chains, and hardware commands to help train future Vision-Language-Action (VLA) models.
Key Results and Insights
The agent was evaluated on Success Rate (SR), Completion Rate (CR), and Step Efficiency (SE):
- Standard Tasks: The agent performed well on simple navigation, achieving an SR of approximately 85% for system apps.
- Challenging & Cross-App Tasks: Performance dropped significantly on complex tasks (e.g., 3/36 SR for single-app "Hard" tasks) and cross-app workflows (2/15 SR), highlighting the difficulty of maintaining long reasoning chains across different UI environments.
- Platform Agnosticism: Because the system only "sees" pixels and "touches" the glass, it is the first truly cross-platform smartphone agent, working equally well on iOS and Android.
- Privacy Advantage: By treating the phone as a black box, See-Control ensures sensitive data stays on the device rather than being tunneled through a debugging bridge.
Limitations and Future Work
The current setup is limited by hardware (single-point contact only) and latency due to MLLM API calls. Future directions include exploring dexterous robotic hands for multi-touch gestures and moving toward on-device inference for real-time interaction.
Summary updated