TL;DR
ATLAS-Finance is a new, high-fidelity benchmark designed to test AI agents in the "messy" reality of investment banking. By requiring agents to navigate simulated workplace environments—searching through emails, Slack threads, and calendars to build complex financial models—the paper reveals that even frontier models like Claude Opus and GPT-6 struggle significantly, with the best agents passing fewer than 11% of tasks.
What Problem Does This Paper Address?
Most existing AI benchmarks evaluate agents in "clean" environments where the necessary information is provided in a single prompt or a well-organized document. Real professional work is far more chaotic. Requirements are often under-specified, scattered across multiple applications (email, chat, files), and subject to conflicting updates or late-stage corrections.
In investment banking, a single misunderstood assumption or an outdated forecast found in a buried email thread can render a complex financial model useless. The authors argue that to be truly "economically valuable," AI agents must move beyond simple reasoning and demonstrate the ability to reliably gather, reconcile, and integrate information across a fragmented workplace ecosystem.
Methods
The researchers developed a sophisticated evaluation framework using the Harbor framework to create 13 simulated firm environments.
- Multi-App Interaction: Agents interact with simulated versions of Outlook, Slack, Google Drive, Google Calendar, and OneNote via five Model Context Protocol (MCP) servers.
- Realistic Workflows: Agents receive a high-level task (e.g., "Build the Project Helios wind project-finance model") and must use their tools to find the "data pack," identify the VP’s specific modeling conventions, and catch any late-breaking corrections sent via Slack or email.
- Agentic Verification: To grade these complex outputs, the authors introduced Gandalf the Grader, an agentic verifier that uses Gemini 3.8 Flash to inspect submitted workbooks against expert-authored rubrics. These rubrics are incredibly detailed, with a median of 124.5 binary criteria covering everything from financial logic to spreadsheet formatting.
Datasets
The primary contribution is the ATLAS-Finance dataset:
- 100 end-to-end tasks across six workflow families: M&A, LBO underwriting, valuation, infrastructure finance, credit, and operating-model rebuilds.
- Expert-Authored: Tasks were developed by experienced bankers (up to VP level) and take humans 15–30 hours to complete.
- Comparison: Unlike previous benchmarks like BankerToolBench (which uses simplified data rooms) or MBABench (which focuses on isolated spreadsheet tasks), ATLAS-Finance requires cross-application navigation and information reconciliation.
Key Insights and Ideas
- The "Workplace Gap": When agents were given all necessary information upfront (bypassing the need to search apps), their scores improved by 7.5 to 11.8 percentage points. This confirms that navigating the "messiness" of a workplace is a distinct and significant challenge for current AI.
- Logic is the Bottleneck: The dominant failure mode (53.8% of cases) was Financial Logic—using the wrong accounting treatment or economic assumption—rather than simple spreadsheet mechanical errors.
- The Self-Correction Trap: The authors highlight a "Project Falcon" case study where an agent (Claude Opus 5) performed internal self-checks that passed. However, because the agent started with a fundamentally incorrect tax assumption, its self-validation merely reinforced the error.
- High Stakes, High Value: A survey of 43 senior bankers revealed that firms would be willing to pay over $250,000 per seat per year for an AI that could reliably pass 99% of these tasks, underscoring the massive economic potential of solving these agentic challenges.
Key Results
The benchmark proved to be a "stress test" that most current models failed:
- Low Pass Rates: At the Pass@1 metric (percentage of tasks passed on the first attempt), Claude Opus 5 led the pack at only 10.7%, followed closely by Claude Fable 5.1 (10.3%) and GPT-6 Astra (9.7%).
- Partial Success: While pass rates were low, mean rubric scores were higher (around 70% for top models), suggesting that agents are capable of completing large portions of the work but fail on critical, "deal-breaking" details.
- Model Comparison: Newer frontier models showed a significant jump over older ones (e.g., Gemini 3.1 Pro Preview scored only 0.7% Pass@1), but a clear plateau remains.
Limitations and Future Work
- Static Environments: The simulated coworkers do not respond to agent queries in real-time; agents receive predefined updates but cannot "chat" with a VP to clarify ambiguous instructions.
- No External Search: The benchmark restricts agents to the provided workplace data, excluding the real-world need to search the web for external market data or company filings.
- Domain Specificity: While deep in finance, the results may not perfectly translate to other complex fields like law or engineering, though the "scattered information" problem is universal.
- Future Direction: The authors suggest that improving agent reliability will require better long-horizon planning and more robust ways to reconcile conflicting information without human intervention.
Summary updated