Abstract
Smartphones have significantly enhanced our daily learning, communication, and entertainment, becoming an essential component of modern life. However, certain populations, including the elderly and individuals with disabilities, encounter challenges in utilizing smartphones, thus necessitating mobile app operation assistants, a.k.a. mobile app agent. With considerations for privacy, permissions, and cross-platform compatibility issues, we endeavor to devise and develop PeriGuru in this work, a peripheral robotic mobile app operation assistant based on GUI image understanding and prompting with Large Language Model (LLM). PeriGuru leverages a suite of computer vision techniques to analyze GUI screenshot images and employs LLM to inform action decisions, which are then executed by robotic arms. PeriGuru achieves a success rate of 81.94% on the test task set, which surpasses by more than double the method without PeriGuru's GUI image interpreting and prompting design.
Introduction
While mobile applications are central to modern life, they remain inaccessible to many, particularly the elderly and those with visual impairments. Existing mobile agents typically rely on software-level tools, such as Android’s UIAutomator, to scrape screen data. However, this approach faces significant hurdles regarding privacy, as users are hesitant to grant full screen access; permissions, as security restrictions often break accessibility tools; and compatibility, as software agents are often platform-specific.
The authors propose PeriGuru, a "peripheral" approach: an embodied agent that sits outside the phone, using a camera to see and a robotic arm to touch. By treating the phone as a "black box," PeriGuru ensures universal compatibility across iOS, Android, and HarmonyOS while maintaining high user privacy.
System Architecture
The PeriGuru system operates in a closed-loop cycle consisting of three distinct stages: Perception, Decision-Making, and Action.
- Perception: A high-speed camera captures the screen. The system uses YOLOv5 for widget detection, OCR for text recognition, and LabelDroid to assign semantic meanings to icons (e.g., recognizing a magnifying glass as "search").
- Decision-Making: Instead of sending raw pixels to an LLM, PeriGuru converts the UI into a structured, HTML-like summary. This format leverages the fact that LLMs are pre-trained on massive amounts of web data. The LLM (GPT-4V or GPT-4o) receives this summary, the user's task, and action history to decide the next step via a Chain-of-Thought process.
- Action: The decision is translated into a function call (e.g.,
tap(id),scroll(direction)) executed by a Yahboom DOFBOT SE robotic arm.
GUI Understanding and Prompting
A core innovation of PeriGuru is its ability to reconstruct the UI's hierarchical structure from raw images. Using DBSCAN clustering and XY-cut algorithms, the system identifies lists and buttons to generate a formal GUI summary.
Key insights from this approach include:
- HTML as a Universal Language: LLMs perform significantly better when the UI is presented as structured text rather than a coordinate grid. This leverages existing knowledge of web structures to understand mobile layouts.
- Icon Semantics: By providing text descriptions for icons, the agent gains the common sense needed to navigate complex layouts (e.g., knowing a "house" icon likely leads home).
- Bridging the Camera-to-Screenshot Gap: To handle glare and noise in camera-captured images, the authors used data augmentation (light masks and rotations), which improved object detection F1-scores by nearly 12%.
Evaluation and Results
The system was evaluated using a custom benchmark of 72 tasks across 10 popular apps and 64 tasks generated from the RICO dataset. PeriGuru achieved a plan success rate of 89.71% and a physical execution success rate of 81.94%. This performance is more than double the success rate of "grid-based" agents that lack structured UI interpretation. The system averaged 4 to 5 steps per multi-step task, demonstrating high efficiency.
Conclusion
PeriGuru demonstrates the viability of peripheral robotic assistants for mobile operation. While highly successful, future work will focus on improving task completion assessment—helping the agent realize when a task is finished—and enabling more complex robotic gestures and self-correction mechanisms to handle app latency or missed taps.
Summary updated