TL;DR
This paper introduces a privacy framework for mobile GUI agents that makes sensitive data "available but invisible." By using a local, layered architecture to swap real-world PII (Personally Identifiable Information) with deterministic placeholders, the system allows cloud-based AI agents to complete complex tasks without ever seeing the user's actual private data.
What Problem Does This Paper Address?
Mobile GUI agents (like those powered by GPT-4o or Gemini) are incredibly useful for automating tasks like "add this contact" or "pay my utility bill." To function, these agents need full access to your screen—including screenshots and UI hierarchies (XML).
This creates a massive privacy bottleneck:
- Over-collection: Agents see everything, including bank balances, private messages, and addresses that might be irrelevant to the task.
- The Utility-Privacy Trade-off: If you blur or redact sensitive info, the agent often breaks because it can't "ground" its actions (e.g., it can't click a button if it doesn't know which "John" the user is talking about).
- Cloud Exposure: Most powerful agents live in the cloud, meaning your raw, unredacted screen data is transmitted to third-party servers.
The authors aim to break this trade-off by ensuring the agent can use the data for reasoning without seeing its raw value.
Methods
The authors propose a four-layer local mediation system that sits between the mobile device and the cloud AI:
- PII Detector Layer: Uses a hybrid approach combining GLiNER (a zero-shot Named Entity Recognition model) and Regex to identify sensitive entities (names, phone numbers, etc.) across user instructions, XML files, and screenshots.
- UI Transformer Layer: This is the "anonymizer." It replaces PII with type-preserving placeholders (e.g.,
FIRST_NAME#0t3ia). Crucially, it uses a Local Mapping Table to ensure consistency. If "Alice" appears in the instructions and on the screen, she is assigned the exact same hash-based placeholder in both places. - Secure Interaction Proxy: When the cloud agent decides to act (e.g., "Type
FIRST_NAME#0t3iainto the box"), this proxy intercepts the command. It looks up the real value ("Alice") in the local table and executes the command on the phone using the real data. The cloud never sees the "Alice" string. - Privacy Gatekeeper: For tasks requiring logic over raw data (e.g., "Is this credit card expired?"), the system uses a small Local LLM (Qwen3-8B) to perform the check locally and return only a boolean (True/False) to the cloud agent.
Datasets
The framework was evaluated on two primary benchmarks:
- AndroidLab: A benchmark of 138 real-world smartphone tasks (navigation, form filling, etc.) across 9 apps to test utility (success rate).
- PrivScreen: A dataset of 500+ screenshots augmented with 1,000+ synthetic PII entities to test privacy leakage (how much sensitive info the agent can "guess" or see).
Key Insights and Ideas
- Consistency is King: The biggest "aha" moment is that agents don't fail because data is hidden; they fail because it's hidden inconsistently. If the agent sees a name in the instructions but a blurred box on the screen, it can't connect the two. By using deterministic hashes, the agent's "worldview" remains logically sound even if it's anonymized.
- Instruction-Driven Whitelisting: The authors realized that if a user explicitly puts a phone number in their voice command, they’ve already shared it with the agent. The system "whitelists" these tokens to avoid over-redaction that might confuse the model.
- Type-Preserving Placeholders: Instead of just saying
[REDACTED], usingPHONE_NUMBER#hashprovides the agent with the semantic context it needs to know what to do with that field.
Key Results
The framework achieves the best privacy-utility trade-off observed to date:
- Privacy: On the PrivScreen benchmark using GPT-5, the Leakage Rate (LR) plummeted from 97.14% (unprotected) to just 18.84% with the proposed framework.
- Utility: On AndroidLab, the Success Rate (SR) for GPT-4o only saw a minor dip, moving from 36.23% to 32.61%.
- Surprising Efficiency: In some XML-based tasks, anonymization actually improved performance for Gemini models (from 48.9% to 52.1%), likely because removing specific PII reduced "noise" and helped the model focus on the structural task.
- Latency: The local processing adds about 1.77 seconds of overhead per image, which is considered acceptable for asynchronous mobile automation.
Limitations and Future Work
- OCR/NER Dependency: The system is only as good as its detectors. If the local OCR or PII model misses a piece of sensitive text, it will be sent to the cloud unredacted.
- Complex Semantic Reasoning: While the "Privacy Gatekeeper" handles simple logic, extremely complex reasoning over raw private data (e.g., "Summarize the sentiment of this private medical email") still presents a challenge, as the cloud agent cannot see the content to summarize it.
- Local Compute: Running a local LLM (Qwen3-8B) and NER models requires decent mobile/PC hardware, which might limit deployment on lower-end devices.
Summary updated