Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible

Lepeng Zhao, Zhenhua Zou, Shuo Li, Zhuotao Liu

TL;DR

This paper introduces a privacy framework for mobile GUI agents that makes sensitive data "available but invisible." By using a local, layered architecture to swap real-world PII (Personally Identifiable Information) with deterministic placeholders, the system allows cloud-based AI agents to complete complex tasks without ever seeing the user's actual private data.

What Problem Does This Paper Address?

Mobile GUI agents (like those powered by GPT-4o or Gemini) are incredibly useful for automating tasks like "add this contact" or "pay my utility bill." To function, these agents need full access to your screen—including screenshots and UI hierarchies (XML).

This creates a massive privacy bottleneck:

  1. Over-collection: Agents see everything, including bank balances, private messages, and addresses that might be irrelevant to the task.
  2. The Utility-Privacy Trade-off: If you blur or redact sensitive info, the agent often breaks because it can't "ground" its actions (e.g., it can't click a button if it doesn't know which "John" the user is talking about).
  3. Cloud Exposure: Most powerful agents live in the cloud, meaning your raw, unredacted screen data is transmitted to third-party servers.

The authors aim to break this trade-off by ensuring the agent can use the data for reasoning without seeing its raw value.

Methods

The authors propose a four-layer local mediation system that sits between the mobile device and the cloud AI:

  1. PII Detector Layer: Uses a hybrid approach combining GLiNER (a zero-shot Named Entity Recognition model) and Regex to identify sensitive entities (names, phone numbers, etc.) across user instructions, XML files, and screenshots.
  2. UI Transformer Layer: This is the "anonymizer." It replaces PII with type-preserving placeholders (e.g., FIRST_NAME#0t3ia). Crucially, it uses a Local Mapping Table to ensure consistency. If "Alice" appears in the instructions and on the screen, she is assigned the exact same hash-based placeholder in both places.
  3. Secure Interaction Proxy: When the cloud agent decides to act (e.g., "Type FIRST_NAME#0t3ia into the box"), this proxy intercepts the command. It looks up the real value ("Alice") in the local table and executes the command on the phone using the real data. The cloud never sees the "Alice" string.
  4. Privacy Gatekeeper: For tasks requiring logic over raw data (e.g., "Is this credit card expired?"), the system uses a small Local LLM (Qwen3-8B) to perform the check locally and return only a boolean (True/False) to the cloud agent.

Datasets

The framework was evaluated on two primary benchmarks:

  • AndroidLab: A benchmark of 138 real-world smartphone tasks (navigation, form filling, etc.) across 9 apps to test utility (success rate).
  • PrivScreen: A dataset of 500+ screenshots augmented with 1,000+ synthetic PII entities to test privacy leakage (how much sensitive info the agent can "guess" or see).

Key Insights and Ideas

  • Consistency is King: The biggest "aha" moment is that agents don't fail because data is hidden; they fail because it's hidden inconsistently. If the agent sees a name in the instructions but a blurred box on the screen, it can't connect the two. By using deterministic hashes, the agent's "worldview" remains logically sound even if it's anonymized.
  • Instruction-Driven Whitelisting: The authors realized that if a user explicitly puts a phone number in their voice command, they’ve already shared it with the agent. The system "whitelists" these tokens to avoid over-redaction that might confuse the model.
  • Type-Preserving Placeholders: Instead of just saying [REDACTED], using PHONE_NUMBER#hash provides the agent with the semantic context it needs to know what to do with that field.

Key Results

The framework achieves the best privacy-utility trade-off observed to date:

  • Privacy: On the PrivScreen benchmark using GPT-5, the Leakage Rate (LR) plummeted from 97.14% (unprotected) to just 18.84% with the proposed framework.
  • Utility: On AndroidLab, the Success Rate (SR) for GPT-4o only saw a minor dip, moving from 36.23% to 32.61%.
  • Surprising Efficiency: In some XML-based tasks, anonymization actually improved performance for Gemini models (from 48.9% to 52.1%), likely because removing specific PII reduced "noise" and helped the model focus on the structural task.
  • Latency: The local processing adds about 1.77 seconds of overhead per image, which is considered acceptable for asynchronous mobile automation.

Limitations and Future Work

  • OCR/NER Dependency: The system is only as good as its detectors. If the local OCR or PII model misses a piece of sensitive text, it will be sent to the cloud unredacted.
  • Complex Semantic Reasoning: While the "Privacy Gatekeeper" handles simple logic, extremely complex reasoning over raw private data (e.g., "Summarize the sentiment of this private medical email") still presents a challenge, as the cloud agent cannot see the content to summarize it.
  • Local Compute: Running a local LLM (Qwen3-8B) and NER models requires decent mobile/PC hardware, which might limit deployment on lower-end devices.

Summary updated

Abstract

Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks by leveraging multimodal large language models (MLLMs) and system-level control interfaces. However, this paradigm introduces significant privacy risks, as agents typically capture and process entire screen contents, thereby exposing sensitive personal data such as phone numbers, addresses, messages, and financial information. Existing defenses either reduce UI exposure, obfuscate only task-irrelevant content, or rely on user authorization, but none can protect task-critical sensitive information while preserving seamless agent usability. We propose an anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data: sensitive information remains usable for task execution but is never directly visible to the cloud-based agent. Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders (e.g., PHONE_NUMBER#a1b2c) that retain semantic categories while removing identifying details. A layered architecture comprising a PII Detector, UI Transformer, Secure Interaction Proxy, and Privacy Gatekeeper ensures consistent anonymization across user instructions, XML hierarchies, and screenshots, mediates all agent actions over anonymized interfaces, and supports narrowly scoped local computations when reasoning over raw values is necessary. Extensive experiments on the AndroidLab and PrivScreen benchmarks show that our framework substantially reduces privacy leakage across multiple models while incurring only modest utility degradation, achieving the best observed privacy-utility trade-off among existing methods. Code available at: https://github.com/one-step-beh1nd/gui_privacy_protection
BibTeX
@misc{zhao2026anonymization,
  author = {Lepeng Zhao and Zhenhua Zou and Shuo Li and Zhuotao Liu},
  title = {{Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible}},
  year = {2026},
  eprint = {2602.10139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.10139},
}
Figure