DelegateEval: Evaluating Platforms for Personal External Service Agents

Jichao Li, Jinrong Lu, Manjiang Yu, Zhixuan Liu

TL;DR

DelegateEval introduces a new framework for evaluating "Personal External Service Agents"—AI proxies that represent an individual’s professional expertise to outside clients. By shifting the focus from simple task completion to platform-level metrics like expertise inheritance and service continuity, the authors provide a roadmap for scaling human professional value through AI.

What Problem Does This Paper Address?

Most AI agent research focuses on personal assistants—tools that help you get your work done. But what if an agent could work for you by serving others?

Imagine a world-class horticulturalist who can only consult for five clients a week. If they could authorize an AI agent to use their specific diagnostic methods, proprietary datasets, and service boundaries to handle 50 clients, they’ve effectively scaled their expertise.

The challenge is that current benchmarks (like SWE-bench or AppWorld) aren't designed for this. They don't measure if an agent is actually "inheriting" a specific person's professional judgment, nor do they track long-term service obligations like maintenance or follow-up. Furthermore, building such an agent is technically daunting for non-engineers. We need to evaluate the platforms that enable experts to create, manage, and scale these services reliably.

Figure 1: Service relationships for personal assistants and personal external service agents. The former serve the individual’s own needs; the latter are authorized and continuously managed by an individual to accept commissions from multiple external clients and provide delivery and maintenance. One system may perform both roles, with identities, materials, permissions, and records managed separately.

Methods

The authors propose DelegateEval, a platform-level evaluation framework. Instead of just scoring a model's "intelligence," it assesses how well a platform supports the entire lifecycle of a professional service across eight dimensions:

  1. Ease of Creation: Can a non-technical expert actually set this up?
  2. Service Discovery and Matching: Can clients find the right expert agent?
  3. Delivery Quality: Are the results verifiable and usable?
  4. Expertise Inheritance: Does the agent use the principal's unique knowledge, or just generic LLM data?
  5. Service Continuity: Does the agent remember past commitments and versions over months, not just minutes?
  6. Learning and Improvement: Does the agent get better based on the expert's feedback?
  7. Reliability and Governance: Are data boundaries and authorizations strictly enforced?
  8. Efficiency and Value: Does this actually save the expert time and provide ROI?

The framework utilizes a "contract-first" approach, where every task must have predefined, independent verification methods (automated tests or expert review) before execution.

Datasets

The paper proposes a taxonomy of ten service-task types (Q01–Q10) to serve as the basis for future benchmarking, including:

  • Code development and automation.
  • Mathematics tutoring.
  • Research synthesis.
  • Planning under multiple constraints.
  • General knowledge and health education.

Note: The authors explicitly state that the complete datasets, executable environments, and unified scoring protocols are currently under development and are not yet released as a runnable test suite.

Key Insights and Ideas

  • Expertise vs. Persona: The authors make a sharp distinction between "persona-based roleplay" (making an agent sound like a famous poet) and "expertise inheritance" (making an agent solve a problem using a specific engineer's debugging workflow). True service agents require the latter.
  • The Three-Party Relationship: Unlike the standard User-Agent loop, this involves a Principal (the expert), the Agent, and the Client. This creates complex authorization needs—for example, a client can request a change to a project, but they shouldn't be able to rewrite the expert's global service rules.
  • Longitudinal Evaluation: Service isn't a one-off prompt. DelegateEval emphasizes "longitudinal records," tracking how an agreement made on Day 3 affects a dispute on Day 45.
Figure 2: Relationships and interaction flow in agent-native services. The client’s personal assistant and the personal external service agent discover services, commission work, exchange status, and deliver results within their respective authorizations.

Key Results

As a framework and design paper, DelegateEval does not yet report quantitative performance scores for specific models. Its primary "result" is the formalization of the Personal External Service Agent (PESA) role and the multi-dimensional evaluation design.

The paper provides a "Technical Inventory" template to ensure that when researchers eventually report scores, they also document the underlying "how" (e.g., was RAG used? Fine-tuning? Human-in-the-loop?) to allow for fair comparisons across different platform architectures.

Limitations and Future Work

The authors are transparent about the current state of the project:

  • No Executable Benchmark: The paper defines the how and why, but the what (the actual data and code) is the "Next Step."
  • Human Calibration: Open-ended tasks (like research synthesis) require calibrated human reviewers, which hasn't been implemented yet.
  • Cross-Model Baselines: There is no data yet on how different LLM families (GPT-4 vs. Claude vs. Llama) perform within this specific "service proxy" context.

The immediate next steps involve assembling the initial dataset for code modification and tutoring, and building an "inspectable evaluation environment" that can replay long-term service histories.

Summary updated

Abstract

AI agents that work for an individual over time may also help that individual serve external clients. We define a personal external service agent as an agent authorized and continuously managed by a specific individual to accept client requests, deliver work, and provide ongoing support using that person's knowledge, experience, professional methods, and service boundaries. Such agents could allow individuals to serve more clients with less direct involvement and generate economic value at scale. We propose DelegateEval, a platform-level evaluation framework for agent systems that may combine large language models, reusable skills, protocols such as MCP, and executable tool environments. It covers eight dimensions: ease of creation, service discovery and matching, delivery quality, expertise inheritance, service continuity, learning and improvement, reliability and governance, and efficiency and value. These dimensions assess whether a platform enables people with different backgrounds to turn their expertise into ongoing services that deliver verifiable work, develop capabilities through experience, and remain economically feasible. We present the framework and an experimental design. Complete datasets, executable environments, a unified scoring protocol, and cross-model results remain to be developed and empirically validated.
BibTeX
@misc{li2026delegateeval,
  author = {Jichao Li and Jinrong Lu and Manjiang Yu and Zhixuan Liu},
  title = {{DelegateEval: Evaluating Platforms for Personal External Service Agents}},
  year = {2026},
  eprint = {2611.07508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07508},
}
Figure