PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM

Kelin Fu, Yang Tian, Kaigui BianarXiv 2024 · arXiv · OpenReview

Abstract

Smartphones have significantly enhanced our daily learning, communication, and entertainment, becoming an essential component of modern life. However, certain populations, including the elderly and individuals with disabilities, encounter challenges in utilizing smartphones, thus necessitating mobile app operation assistants, a.k.a. mobile app agent. With considerations for privacy, permissions, and cross-platform compatibility issues, we endeavor to devise and develop PeriGuru in this work, a peripheral robotic mobile app operation assistant based on GUI image understanding and prompting with Large Language Model (LLM). PeriGuru leverages a suite of computer vision techniques to analyze GUI screenshot images and employs LLM to inform action decisions, which are then executed by robotic arms. PeriGuru achieves a success rate of 81.94% on the test task set, which surpasses by more than double the method without PeriGuru's GUI image interpreting and prompting design.

Introduction

While mobile applications are central to modern life, they remain inaccessible to many, particularly the elderly and those with visual impairments. Existing mobile agents typically rely on software-level tools, such as Android’s UIAutomator, to scrape screen data. However, this approach faces significant hurdles regarding privacy, as users are hesitant to grant full screen access; permissions, as security restrictions often break accessibility tools; and compatibility, as software agents are often platform-specific.

The authors propose PeriGuru, a "peripheral" approach: an embodied agent that sits outside the phone, using a camera to see and a robotic arm to touch. By treating the phone as a "black box," PeriGuru ensures universal compatibility across iOS, Android, and HarmonyOS while maintaining high user privacy.

System Architecture

The PeriGuru system operates in a closed-loop cycle consisting of three distinct stages: Perception, Decision-Making, and Action.

Overview of the PeriGuru system architecture, showing the loop between screen observation, LLM-based decision making, and robotic action.
  1. Perception: A high-speed camera captures the screen. The system uses YOLOv5 for widget detection, OCR for text recognition, and LabelDroid to assign semantic meanings to icons (e.g., recognizing a magnifying glass as "search").
  2. Decision-Making: Instead of sending raw pixels to an LLM, PeriGuru converts the UI into a structured, HTML-like summary. This format leverages the fact that LLMs are pre-trained on massive amounts of web data. The LLM (GPT-4V or GPT-4o) receives this summary, the user's task, and action history to decide the next step via a Chain-of-Thought process.
  3. Action: The decision is translated into a function call (e.g., tap(id), scroll(direction)) executed by a Yahboom DOFBOT SE robotic arm.

GUI Understanding and Prompting

A core innovation of PeriGuru is its ability to reconstruct the UI's hierarchical structure from raw images. Using DBSCAN clustering and XY-cut algorithms, the system identifies lists and buttons to generate a formal GUI summary.

The detailed pipeline for GUI image understanding, from element detection to the generation of a formal hierarchical GUI summary.

Key insights from this approach include:

  • HTML as a Universal Language: LLMs perform significantly better when the UI is presented as structured text rather than a coordinate grid. This leverages existing knowledge of web structures to understand mobile layouts.
  • Icon Semantics: By providing text descriptions for icons, the agent gains the common sense needed to navigate complex layouts (e.g., knowing a "house" icon likely leads home).
  • Bridging the Camera-to-Screenshot Gap: To handle glare and noise in camera-captured images, the authors used data augmentation (light masks and rotations), which improved object detection F1-scores by nearly 12%.

Evaluation and Results

The system was evaluated using a custom benchmark of 72 tasks across 10 popular apps and 64 tasks generated from the RICO dataset. PeriGuru achieved a plan success rate of 89.71% and a physical execution success rate of 81.94%. This performance is more than double the success rate of "grid-based" agents that lack structured UI interpretation. The system averaged 4 to 5 steps per multi-step task, demonstrating high efficiency.

Conclusion

PeriGuru demonstrates the viability of peripheral robotic assistants for mobile operation. While highly successful, future work will focus on improving task completion assessment—helping the agent realize when a task is finished—and enabling more complex robotic gestures and self-correction mechanisms to handle app latency or missed taps.

Summary updated

Abstract

Smartphones have significantly enhanced our daily learning, communication, and entertainment, becoming an essential component of modern life. However, certain populations, including the elderly and individuals with disabilities, encounter challenges in utilizing smartphones, thus necessitating mobile app operation assistants, a.k.a. mobile app agent. With considerations for privacy, permissions, and cross-platform compatibility issues, we endeavor to devise and develop PeriGuru in this work, a peripheral robotic mobile app operation assistant based on GUI image understanding and prompting with Large Language Model (LLM). PeriGuru leverages a suite of computer vision techniques to analyze GUI screenshot images and employs LLM to inform action decisions, which are then executed by robotic arms. PeriGuru achieves a success rate of 81.94% on the test task set, which surpasses by more than double the method without PeriGuru's GUI image interpreting and prompting design. Our code is available on https://github.com/Z2sJ4t/PeriGuru.
BibTeX
@misc{fu2024periguru,
  author = {Kelin Fu and Yang Tian and Kaigui Bian},
  title = {{PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM}},
  year = {2024},
  eprint = {2409.09354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.09354},
}
Figure