TL;DR
OfficePrism is a new benchmark designed to evaluate AI agents on single-domain enterprise tasks across seven business functions like HR, Finance, and Legal. By using a state-grounded construction pipeline and a joint natural-language-and-database evaluation, it reveals that even top-tier agents struggle significantly as task complexity increases, with median pass rates dropping from 83.8% for simple tasks to 56.6% for complex workflows.
What Problem Does This Paper Address?
Evaluating AI "office agents" is currently a messy business. Existing benchmarks often conflate different skills—like navigating a UI, managing a workspace, and performing domain-specific operations—into a single score. This makes it nearly impossible to localize failures. If an agent fails to book a business trip, was it because it couldn't coordinate between the calendar and the booking tool, or because it fundamentally didn't understand the procurement rules?
OfficePrism addresses this by focusing on single-domain operations. It tests whether an agent can complete tasks that are entirely "closed" within one business function (e.g., just IT or just Finance) before we even worry about cross-functional orchestration.
Methods
The authors introduce a benchmark of 420 tasks, categorized into three levels of structural complexity (L1 to L3).
Key methodological innovations include:
- Fresh Synthetic Environments: Each task runs in a clean, in-memory environment based on a shared synthetic organization (75 employees, 12 departments). This prevents "state leakage" between trials.
- Coupled Task Synthesis: Instead of generating a prompt and then trying to find tools to solve it, their pipeline uses a "grounding packet" of business rules and records to jointly generate a Chinese-language ticket, a hidden executable "reference witness" (the gold-standard tool chain), and the expected response.
- Two Interface Packages: Agents are tested through two different interfaces:
- Full: Exposes all native tool schemas directly.
- Search: Provides a discovery-and-dispatch interface, forcing the agent to find the right tool before using it.
- Joint NL-DB Evaluation: Success isn't just about a fluent answer. The benchmark checks both the Natural Language (NL) response and the final Database (DB) state against the reference execution.
Datasets
The primary artifact is OfficePrism-420, which includes:
- 420 tasks across seven functions: Administration, Finance, HR, IT, Legal, Procurement, and Project Work.
- Complexity distribution: 105 L1 tasks (direct intent), 140 L2 tasks (dependent chains), and 175 L3 tasks (complex logic like filtering or conditional actions).
- Scale: The environment includes 42–61 top-level collections and 135–198 native functions depending on the domain.
Key Insights and Ideas
- The "Search" Advantage: Counter-intuitively, providing a search-and-dispatch interface often leads to better performance than exposing the full schema. This is likely because it reduces the "token tax" and context noise, especially for smaller or older models.
- Function-Resolved Gaps: Performance varies wildly across domains. For example, agents generally performed much better in HR (80.7% mean) than in Project Work (51.6% mean). A single aggregate score hides these massive discrepancies.
- Reasoning vs. Language: Failure analysis shows that agents rarely fail due to poor language generation. Instead, they fail at "finding the right data"—picking the wrong record from a list of similar names or using slightly incorrect filters.
- The "Last Millimeter" Failure: The authors highlight cases where an agent executes a workflow perfectly but fails at the very end—for example, passing a boolean
Falseas a string"False". The tool returns success, the agent reports success, but the database state remains wrong. This underscores why DB-state verification is non-negotiable for enterprise benchmarks.
Key Results
- Headline Performance: Across 124 model-harness configurations, the joint NL-DB pass rate ranged from 22.6% to 76.4%, with a median of 67.9%.
- Complexity Scaling: There is a clear performance cliff. Median pass rates were 83.8% (L1), 71.8% (L2), and 56.6% (L3).
- Interaction Cost: Complex L3 tasks require significantly more steps (median 16.1) compared to L1 tasks (median 5.9).
Limitations and Future Work
-
Synthetic Nature : While grounded in rules, the environment is synthetic and may not capture the messy, unstructured data found in real enterprises.
-
Language Scope: The benchmark is currently focused on Chinese-language requests.
- Single-Domain Focus: By design, it does not yet address the "extra cost" of coordinating across multiple business functions, which is the next logical step for agent research.
- Deterministic Evaluation: The reported results are based on single runs per configuration; future work could incorporate repeated runs to measure uncertainty.
Summary updated