Abstract
This paper introduces Computer-Using Personal Agents (CUPAs), a framework that integrates graphical automation with a secure Personal Knowledge Graph (PKG). By bridging the gap between general-purpose automation and private user context, CUPAs enable complex, multi-user task execution while maintaining user data sovereignty. We propose a conceptual architecture and a three-stage roadmap for the evolution of these agents from simple screen-interactors to collaborative, web-aware personal assistants.
Introduction
We are currently witnessing the emergence of Computer-Using Agents (CUAs)—AI models capable of navigating graphical user interfaces to perform tasks like booking flights or browsing the web. However, these agents face a significant "privacy wall." Because they lack secure, structured access to sensitive personal data (e.g., medical records, specific preferences, or contact details), they frequently require manual user intervention for critical steps. This "takeover mode" limits the utility of automation.
The authors propose a middle ground: agents that are sufficiently intelligent to utilize personal data for tedious tasks, yet secure enough to prevent the surrender of a user's entire digital identity to a single AI provider. This vision is realized through the Computer-Using Personal Agent (CUPA).
Proposed Architecture
The CUPA framework consists of two primary components designed to balance capability with privacy:
- The CUA (The "Muscle"): An LLM-based agent capable of interacting with web interfaces, APIs, and observing HTTP requests.
- The PKG (The "Brain/Vault"): A Personal Knowledge Graph that serves as a structured, private repository of user data.
By utilizing Semantic Web standards (such as ODRL for digital rights), users can define machine-readable policies that dictate exactly what the agent can access and share. This architecture facilitates a shift from vision-based interaction (interpreting pixels) to Web-aware interaction, where agents interact directly with underlying protocols for increased reliability.
Key Concepts and Philosophy
Several core ideas distinguish the CUPA vision from current automation trends:
- The "Discuss then Transact" Model: Agents are envisioned to negotiate in natural language (e.g., "My user has a shellfish allergy; can we substitute the shrimp?") before executing a structured transaction.
- Multi-dimensional Negotiation: CUPAs can optimize for complex variables—balancing cost, health, and personal taste—across multiple users simultaneously, a task humans find cognitively taxing.
- Granular Privacy: Unlike binary "all or nothing" data permissions, a CUPA manages fine-grained access, sharing only the minimum data required for a specific step (e.g., a zip code for a quote, but a full address only for final delivery).
- Empowerment through the PKG: The PKG acts as a configurable buffer, empowering the individual to control their data rather than ceding control to a central corporation.
Evolution Roadmap
The paper outlines a three-stage progression for the development of personal agents:
- Stage 1: CUAs + Personal Data. Enhancing current agents with private PKG access to minimize manual takeover events.
- Stage 2: Web-aware CUPAs. Moving beyond screen-scraping to agents that understand HTTP/APIs for robust, high-speed automation.
- Stage 3: Networks of CUPAs. A future state where autonomous agents negotiate with one another to coordinate schedules and preferences without direct human intervention.
Challenges and Future Work
Transitioning to a CUPA-centric ecosystem involves significant hurdles. Technical and ethical considerations include accountability and liability (determining responsibility for agent-led errors), explainability (tracing LLM decisions back to private data logs), and inter-agent communication (developing a standardized language for negotiation). Furthermore, agents must be capable of distinguishing between permanent preferences and temporary whims to avoid learning irrelevant or outdated patterns.
Summary updated