This paper from Microsoft Research introduces AutoBuddy, a framework designed to move beyond "reactive" AI agents—which wait for a specific prompt—toward "proactive," always-on assistants that can independently discover and execute tasks within a complex workplace environment.
TL;DR
AutoBuddy-Bench is a novel simulator that generates a consistent, grounded workplace environment to evaluate how well LLM agents can proactively identify "hidden" tasks in emails and chats and then execute them. The study reveals that task discovery and execution are distinct capabilities, with even the most advanced models often failing to bridge the gap between finding a task and successfully completing it in business systems.
What Problem Does This Paper Address?
Most current AI agent benchmarks are instruction-driven: you tell the agent exactly what to do (e.g., "Book a meeting with Alice"). However, in a real office, work is rarely that tidy. Tasks are buried in a mountain of "noise"—unanswered emails, Slack threads, and calendar invites. A true digital assistant needs to be proactive, scanning the user's workspace to infer what needs to be done without being asked.
The authors identify two major hurdles:
- For the Agent: Distinguishing actual obligations from background noise (like a request already handled by a colleague).
- For Evaluation: Creating a benchmark that is "mutually consistent." If an agent sends an email, the simulated world needs to reflect that change accurately across all systems to verify the outcome.
Methods
The researchers developed two core components:
- AutoBuddy-Bench: A simulator that builds a company from a short specification. It uses a temporal graph to track every employee, project, and event. When a business event occurs (e.g., a software training run finishes), the system generates a "ground-truth" obligation. This is then rendered into realistic artifacts like emails or chat messages that the agent must find.
- The AutoBuddy Agent: A reference agent that operates in two distinct phases. First, it performs Task Discovery, mining the workspace to create a structured list of owed tasks. Second, it performs Task Execution, where it uses tools (CRM, Finance systems, etc.) in a fresh session to complete each identified task.
Datasets
The primary dataset is the newly introduced AutoBuddy-Bench. For their evaluation, the authors generated an "evaluation world" simulating six months of company history, resulting in 120 ground-truth tasks across 94 actor-days and 44 different employees. These tasks span domains like HR, Finance, Sales, and Engineering research.
The benchmark was used to test 17 different LLM backbones, including models from Anthropic (Claude), OpenAI (GPT), DeepSeek, and others.
Key Insights and Ideas
- Discovery and Execution are Decoupled: A model’s ability to find a task in a messy inbox (precision/recall) is only weakly correlated with its ability to actually do the work. This suggests that "proactivity" requires a different set of cognitive skills than "instruction following."
- The "Stronger Model" Shift: There is a systematic shift in failure modes as models get more powerful. Weak models fail at discovery (they simply don't see the tasks). Mid-tier models find the tasks but stop short of execution. The strongest models find almost everything but fail during the complex logic of business-system operations.
- The "Stop and Ask" Bottleneck: Interestingly, the most advanced models (like the GPT-6 series in this study) were often too cautious. They frequently identified tasks correctly but then "stopped to ask" for clarification instead of taking action, leading to lower end-to-end scores despite high discovery recall.
Key Results
- Leaderboard: Claude Opus 5.5 emerged as the top performer with an end-to-end score of 0.698, followed closely by Claude Opus 5 (0.680) and GPT-5.6-sol (0.654).
- Performance Range: There is a massive gap between tiers; the weakest model (Claude Haiku 4.5) scored only 0.234.
- Discovery Success: GPT-6.1-sol achieved the highest discovery recall (0.989), finding nearly every task, yet it only ranked 7th overall because it declined to finish many of them, often stopping to ask for more information.
Limitations and Future Work
The authors acknowledge that while their simulator is grounded in a temporal graph, it is still a simulation. Real-world workplace data is often even noisier and more ambiguous.
Future directions include:
- Multi-Agent Interaction: Evaluating how proactive agents interact with each other or with a human-in-the-loop.
- Long-Horizon Memory: Improving how agents track tasks that span weeks or months.
- Skill Distillation: Further exploring the agent's ability to "distill" successful executions into reusable skills for future tasks.
Summary updated