TL;DR
This paper introduces a rigorous capability framework for evaluating "Agentic AI" platforms—systems where LLMs don't just chat, but plan and execute state-changing actions—within the strict constraints of regulated industries like banking. By analyzing an industrial "bakeoff" of ten major platforms, the authors expose a significant "halo effect" in current evaluation practices and propose a new, four-layered "gate-then-score" protocol to move from subjective judgment to objective measurement.
What Problem Does This Paper Address?
Enterprises are currently procuring agentic AI platforms faster than they can effectively measure them. While public benchmarks like AgentBench or GAIA exist, they primarily score the underlying LLM in a fixed environment. However, an enterprise doesn't just buy a model; it buys an integrated platform where the orchestration, memory management, and governance layers often dictate the final outcome more than the model itself.
Furthermore, regulated institutions (like the global bank featured in this study) face unique hurdles: they must satisfy strict auditability, accountability, and model-risk obligations. Currently, buyers often fall back on vendor demonstrations, which risks mistaking "presentation quality" for "engineering quality." The authors aim to bridge this gap by defining what makes an agentic platform "enterprise-ready."
Methods
The researchers conducted an "Agentic AI Bakeoff" involving ten platforms, ranging from open-source frameworks to hyperscaler ecosystems (e.g., Google Vertex AI, Microsoft AutoGen, H2O.ai).
The evaluation was structured around:
- A Nine-Dimension Taxonomy: Platforms were scored on Reflection, Tool Use, Memory, Planning, Multi-agent coordination, Ease of Use, Protocol Support, Quantitative Performance, and Evaluation/Explainability.
- Three Financial Workloads: Counterparty credit risk (CCR), Retrieval-Augmented Generation (RAG), and Process Automation.
- A Multi-Stage Protocol: Moving from organizational screening to deep-dive sessions and live "bakeoff" demos.
- Psychometric Analysis: The authors used Cronbach’s $\alpha$ to test the internal consistency of the scores, revealing whether raters were truly distinguishing between technical dimensions or just following a general impression.
Datasets
Rather than using standard public datasets, the study employed three bespoke financial-services scenarios designed with subject matter experts (SMEs):
- Counterparty Credit Risk (CCR): A complex task combining structured exposure data, unstructured regulatory documentation, and logic.
- RAG: A standard enterprise workload focused on retrieval, grounding, and verifiability.
- Process Automation: Testing goal attainment from minimal instruction, involving multi-step orchestration and exception handling.
Key Insights and Ideas
- The "Halo Effect": The most striking finding was a Cronbach’s $\alpha$ of 0.98 across nominally distinct dimensions. This suggests a strong "halo effect"—raters who liked a platform's demo tended to give it high scores across all technical categories, regardless of specific performance. This highlights the danger of demonstration-based scoring.
- Orchestration > Model: In an enterprise context, the "governance and control plane" (Figure 2) is the most critical component. A platform's ability to handle tool failures, maintain persistent memory, and follow safety guardrails is more important than the raw reasoning power of the LLM.
- Workload-Specific Strengths: Aggregated scores often hide specialized capabilities. For example, the platform Letta excelled in memory management (D3), while Microsoft led in protocol support (D7), even if they weren't the overall winners.
- Gate-then-Score: The authors propose that evaluation should be non-compensatory. A platform shouldn't be able to "average out" a failure in a critical security gate (like prompt-injection resistance) with a high score in "ease of use."
Key Results
The bakeoff revealed a clear three-tier structure in the market:
- Tier 1 (Top Performers): H2O.ai, Google, Microsoft, Writer, and Letta. These platforms showed robust capabilities across most dimensions.
- The Gap: There was a massive 18.7-point gap between the bottom of Tier 1 and the top of Tier 2, suggesting a significant maturity divide in the current vendor landscape.
- Stability: These tiers remained stable even when the authors applied different weightings (e.g., a "governance-first" vs. a "developer-first" profile).
Limitations and Future Work
The authors acknowledge several limitations:
- Assisted Conditions: Vendors helped prepare the demos, so the results reflect "demonstrated capability" rather than "out-of-the-box" performance.
- Subjectivity: Despite the taxonomy, ratings remain subjective. To fix this, the authors propose a "Stage 5" (Figure 3) involving buyer-run harnesses and repeated, automated runs to generate trace-level reliability metrics.
- Future Directions: The paper calls for a research agenda focused on "adversarial variants" (e.g., indirect injection probes) and the development of a "business-value identity" to measure the actual ROI of agentic automation beyond just time saved.
Summary updated