Towards Computer-Using Personal Agents

Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jiménez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse WrightarXiv 2025 · DOI · arXiv

Abstract

This paper introduces Computer-Using Personal Agents (CUPAs), a framework that integrates graphical automation with a secure Personal Knowledge Graph (PKG). By bridging the gap between general-purpose automation and private user context, CUPAs enable complex, multi-user task execution while maintaining user data sovereignty. We propose a conceptual architecture and a three-stage roadmap for the evolution of these agents from simple screen-interactors to collaborative, web-aware personal assistants.

Introduction

We are currently witnessing the emergence of Computer-Using Agents (CUAs)—AI models capable of navigating graphical user interfaces to perform tasks like booking flights or browsing the web. However, these agents face a significant "privacy wall." Because they lack secure, structured access to sensitive personal data (e.g., medical records, specific preferences, or contact details), they frequently require manual user intervention for critical steps. This "takeover mode" limits the utility of automation.

The authors propose a middle ground: agents that are sufficiently intelligent to utilize personal data for tedious tasks, yet secure enough to prevent the surrender of a user's entire digital identity to a single AI provider. This vision is realized through the Computer-Using Personal Agent (CUPA).

Proposed Architecture

The CUPA framework consists of two primary components designed to balance capability with privacy:

  1. The CUA (The "Muscle"): An LLM-based agent capable of interacting with web interfaces, APIs, and observing HTTP requests.
  2. The PKG (The "Brain/Vault"): A Personal Knowledge Graph that serves as a structured, private repository of user data.
Conceptual architecture of a CUPA showing the interaction between the CUA, PKG, and external entities.

By utilizing Semantic Web standards (such as ODRL for digital rights), users can define machine-readable policies that dictate exactly what the agent can access and share. This architecture facilitates a shift from vision-based interaction (interpreting pixels) to Web-aware interaction, where agents interact directly with underlying protocols for increased reliability.

Key Concepts and Philosophy

Several core ideas distinguish the CUPA vision from current automation trends:

  • The "Discuss then Transact" Model: Agents are envisioned to negotiate in natural language (e.g., "My user has a shellfish allergy; can we substitute the shrimp?") before executing a structured transaction.
  • Multi-dimensional Negotiation: CUPAs can optimize for complex variables—balancing cost, health, and personal taste—across multiple users simultaneously, a task humans find cognitively taxing.
  • Granular Privacy: Unlike binary "all or nothing" data permissions, a CUPA manages fine-grained access, sharing only the minimum data required for a specific step (e.g., a zip code for a quote, but a full address only for final delivery).
  • Empowerment through the PKG: The PKG acts as a configurable buffer, empowering the individual to control their data rather than ceding control to a central corporation.

Evolution Roadmap

The paper outlines a three-stage progression for the development of personal agents:

  • Stage 1: CUAs + Personal Data. Enhancing current agents with private PKG access to minimize manual takeover events.
  • Stage 2: Web-aware CUPAs. Moving beyond screen-scraping to agents that understand HTTP/APIs for robust, high-speed automation.
  • Stage 3: Networks of CUPAs. A future state where autonomous agents negotiate with one another to coordinate schedules and preferences without direct human intervention.

Challenges and Future Work

Transitioning to a CUPA-centric ecosystem involves significant hurdles. Technical and ethical considerations include accountability and liability (determining responsibility for agent-led errors), explainability (tracing LLM decisions back to private data logs), and inter-agent communication (developing a standardized language for negotiation). Furthermore, agents must be capable of distinguishing between permanent preferences and temporary whims to avoid learning irrelevant or outdated patterns.

Summary updated

Abstract

Computer-Using Agents (CUA) enable users to automate increasingly-complex tasks using graphical interfaces such as browsers. As many potential tasks require personal data, we propose Computer-Using Personal Agents (CUPAs) that have access to an external repository of the user's personal data. Compared with CUAs, CUPAs offer users better control of their personal data, the potential to automate more tasks involving personal data, better interoperability with external sources of data, and better capabilities to coordinate with other CUPAs in order to solve collaborative tasks involving the personal data of multiple users.
BibTeX
@misc{bonatti2025towards,
  author = {Piero A. Bonatti and John Domingue and Anna Lisa Gentile and Andreas Harth and Olaf Hartig and Aidan Hogan and Katja Hose and Ernesto Jiménez-Ruiz and Deborah L. McGuinness and Chang Sun and Ruben Verborgh and Jesse Wright},
  title = {{Towards Computer-Using Personal Agents}},
  year = {2025},
  eprint = {2503.15515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.15515},
}
Figure