TL;DR
DelegateEval introduces a new framework for evaluating "Personal External Service Agents"—AI proxies that represent an individual’s professional expertise to outside clients. By shifting the focus from simple task completion to platform-level metrics like expertise inheritance and service continuity, the authors provide a roadmap for scaling human professional value through AI.
What Problem Does This Paper Address?
Most AI agent research focuses on personal assistants—tools that help you get your work done. But what if an agent could work for you by serving others?
Imagine a world-class horticulturalist who can only consult for five clients a week. If they could authorize an AI agent to use their specific diagnostic methods, proprietary datasets, and service boundaries to handle 50 clients, they’ve effectively scaled their expertise.
The challenge is that current benchmarks (like SWE-bench or AppWorld) aren't designed for this. They don't measure if an agent is actually "inheriting" a specific person's professional judgment, nor do they track long-term service obligations like maintenance or follow-up. Furthermore, building such an agent is technically daunting for non-engineers. We need to evaluate the platforms that enable experts to create, manage, and scale these services reliably.
Methods
The authors propose DelegateEval, a platform-level evaluation framework. Instead of just scoring a model's "intelligence," it assesses how well a platform supports the entire lifecycle of a professional service across eight dimensions:
- Ease of Creation: Can a non-technical expert actually set this up?
- Service Discovery and Matching: Can clients find the right expert agent?
- Delivery Quality: Are the results verifiable and usable?
- Expertise Inheritance: Does the agent use the principal's unique knowledge, or just generic LLM data?
- Service Continuity: Does the agent remember past commitments and versions over months, not just minutes?
- Learning and Improvement: Does the agent get better based on the expert's feedback?
- Reliability and Governance: Are data boundaries and authorizations strictly enforced?
- Efficiency and Value: Does this actually save the expert time and provide ROI?
The framework utilizes a "contract-first" approach, where every task must have predefined, independent verification methods (automated tests or expert review) before execution.
Datasets
The paper proposes a taxonomy of ten service-task types (Q01–Q10) to serve as the basis for future benchmarking, including:
- Code development and automation.
- Mathematics tutoring.
- Research synthesis.
- Planning under multiple constraints.
- General knowledge and health education.
Note: The authors explicitly state that the complete datasets, executable environments, and unified scoring protocols are currently under development and are not yet released as a runnable test suite.
Key Insights and Ideas
- Expertise vs. Persona: The authors make a sharp distinction between "persona-based roleplay" (making an agent sound like a famous poet) and "expertise inheritance" (making an agent solve a problem using a specific engineer's debugging workflow). True service agents require the latter.
- The Three-Party Relationship: Unlike the standard User-Agent loop, this involves a Principal (the expert), the Agent, and the Client. This creates complex authorization needs—for example, a client can request a change to a project, but they shouldn't be able to rewrite the expert's global service rules.
- Longitudinal Evaluation: Service isn't a one-off prompt. DelegateEval emphasizes "longitudinal records," tracking how an agreement made on Day 3 affects a dispute on Day 45.
Key Results
As a framework and design paper, DelegateEval does not yet report quantitative performance scores for specific models. Its primary "result" is the formalization of the Personal External Service Agent (PESA) role and the multi-dimensional evaluation design.
The paper provides a "Technical Inventory" template to ensure that when researchers eventually report scores, they also document the underlying "how" (e.g., was RAG used? Fine-tuning? Human-in-the-loop?) to allow for fair comparisons across different platform architectures.
Limitations and Future Work
The authors are transparent about the current state of the project:
- No Executable Benchmark: The paper defines the how and why, but the what (the actual data and code) is the "Next Step."
- Human Calibration: Open-ended tasks (like research synthesis) require calibrated human reviewers, which hasn't been implemented yet.
- Cross-Model Baselines: There is no data yet on how different LLM families (GPT-4 vs. Claude vs. Llama) perform within this specific "service proxy" context.
The immediate next steps involve assembling the initial dataset for code modification and tutoring, and building an "inspectable evaluation environment" that can replay long-term service histories.
Summary updated