ATLAS-Finance: Benchmarking AI Agents in Realistic Banking Environments

Elaine Lau, Antoine Pangas, Alexandre Finot, Collin Schweiker, Amir Bouzidi, Hui Wen Goh, Andrew Peter Martin, Jonas MuellerarXiv 2026 · arXiv

TL;DR

ATLAS-Finance is a new, high-fidelity benchmark designed to test AI agents in the "messy" reality of investment banking. By requiring agents to navigate simulated workplace environments—searching through emails, Slack threads, and calendars to build complex financial models—the paper reveals that even frontier models like Claude Opus and GPT-6 struggle significantly, with the best agents passing fewer than 11% of tasks.

What Problem Does This Paper Address?

Most existing AI benchmarks evaluate agents in "clean" environments where the necessary information is provided in a single prompt or a well-organized document. Real professional work is far more chaotic. Requirements are often under-specified, scattered across multiple applications (email, chat, files), and subject to conflicting updates or late-stage corrections.

In investment banking, a single misunderstood assumption or an outdated forecast found in a buried email thread can render a complex financial model useless. The authors argue that to be truly "economically valuable," AI agents must move beyond simple reasoning and demonstrate the ability to reliably gather, reconcile, and integrate information across a fragmented workplace ecosystem.

Methods

The researchers developed a sophisticated evaluation framework using the Harbor framework to create 13 simulated firm environments.

  • Multi-App Interaction: Agents interact with simulated versions of Outlook, Slack, Google Drive, Google Calendar, and OneNote via five Model Context Protocol (MCP) servers.
  • Realistic Workflows: Agents receive a high-level task (e.g., "Build the Project Helios wind project-finance model") and must use their tools to find the "data pack," identify the VP’s specific modeling conventions, and catch any late-breaking corrections sent via Slack or email.
  • Agentic Verification: To grade these complex outputs, the authors introduced Gandalf the Grader, an agentic verifier that uses Gemini 3.8 Flash to inspect submitted workbooks against expert-authored rubrics. These rubrics are incredibly detailed, with a median of 124.5 binary criteria covering everything from financial logic to spreadsheet formatting.
A case study showing how an agent's internal self-validation can fail to catch a fundamental logic error.

Datasets

The primary contribution is the ATLAS-Finance dataset:

  • 100 end-to-end tasks across six workflow families: M&A, LBO underwriting, valuation, infrastructure finance, credit, and operating-model rebuilds.
  • Expert-Authored: Tasks were developed by experienced bankers (up to VP level) and take humans 15–30 hours to complete.
  • Comparison: Unlike previous benchmarks like BankerToolBench (which uses simplified data rooms) or MBABench (which focuses on isolated spreadsheet tasks), ATLAS-Finance requires cross-application navigation and information reconciliation.

Key Insights and Ideas

  • The "Workplace Gap": When agents were given all necessary information upfront (bypassing the need to search apps), their scores improved by 7.5 to 11.8 percentage points. This confirms that navigating the "messiness" of a workplace is a distinct and significant challenge for current AI.
  • Logic is the Bottleneck: The dominant failure mode (53.8% of cases) was Financial Logic—using the wrong accounting treatment or economic assumption—rather than simple spreadsheet mechanical errors.
  • The Self-Correction Trap: The authors highlight a "Project Falcon" case study where an agent (Claude Opus 5) performed internal self-checks that passed. However, because the agent started with a fundamentally incorrect tax assumption, its self-validation merely reinforced the error.
  • High Stakes, High Value: A survey of 43 senior bankers revealed that firms would be willing to pay over $250,000 per seat per year for an AI that could reliably pass 99% of these tasks, underscoring the massive economic potential of solving these agentic challenges.
A case study showing how an agent's internal self-validation can fail to catch a fundamental logic error.

Key Results

The benchmark proved to be a "stress test" that most current models failed:

  • Low Pass Rates: At the Pass@1 metric (percentage of tasks passed on the first attempt), Claude Opus 5 led the pack at only 10.7%, followed closely by Claude Fable 5.1 (10.3%) and GPT-6 Astra (9.7%).
  • Partial Success: While pass rates were low, mean rubric scores were higher (around 70% for top models), suggesting that agents are capable of completing large portions of the work but fail on critical, "deal-breaking" details.
  • Model Comparison: Newer frontier models showed a significant jump over older ones (e.g., Gemini 3.1 Pro Preview scored only 0.7% Pass@1), but a clear plateau remains.
Pass rates across various frontier models, showing that even the best models pass fewer than 11% of tasks.

Limitations and Future Work

  • Static Environments: The simulated coworkers do not respond to agent queries in real-time; agents receive predefined updates but cannot "chat" with a VP to clarify ambiguous instructions.
  • No External Search: The benchmark restricts agents to the provided workplace data, excluding the real-world need to search the web for external market data or company filings.
  • Domain Specificity: While deep in finance, the results may not perfectly translate to other complex fields like law or engineering, though the "scattered information" problem is universal.
  • Future Direction: The authors suggest that improving agent reliability will require better long-horizon planning and more robust ways to reconcile conflicting information without human intervention.

Summary updated

Abstract

Professional work involves under-specified assignments, where relevant information is spread across workplace applications and may conflict or change over time. To evaluate how well frontier agents can perform economically valuable work in such settings, we introduce ATLAS-Finance, a benchmark of 100 end-to-end investment-banking tasks across 13 simulated firms. As in real banking workflows, agents must search email, chat, calendars, notes, and files to identify requirements, gather information, reconcile conflicts/updates, and then build complex financial models. Tasks were developed by experienced bankers based on their real deal work and take them 15-30 hours to complete. Agent deliverables are evaluated with expert-authored rubrics (median: 124.5 criteria) covering financial rationale and results, spreadsheet construction, completeness, and formatting. Despite completing substantial portions of the requested work, current agents rarely produce correct deliverables. Across today's frontier models, the best ones (Claude Opus and Fable) pass fewer than 11% of tasks.
BibTeX
@misc{lau2026atlas,
  author = {Elaine Lau and Antoine Pangas and Alexandre Finot and Collin Schweiker and Amir Bouzidi and Hui Wen Goh and Andrew Peter Martin and Jonas Mueller},
  title = {{ATLAS-Finance: Benchmarking AI Agents in Realistic Banking Environments}},
  year = {2026},
  eprint = {2611.22085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22085},
}
Figure