The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims

Kiana Jafari Meimandi, Gabriela Aránguiz-Dias, Grace Ra Kim, Lana Saadeddin, Mykel J. KochenderferarXiv 2025 · DOI · arXiv

Abstract

Current evaluation paradigms for agentic AI systems are significantly skewed toward technical performance metrics, often neglecting the human, temporal, and contextual factors that determine real-world utility. This paper identifies a "measurement imbalance" through a meta-analysis of 84 recent publications, revealing that while 83% of studies report technical success, only a fraction consider longitudinal stability or human-centric integration. We propose a Balanced Four-Axis Evaluation Model—encompassing Technical, Human-centered, Temporal, and Contextual dimensions—to bridge the performance gap between laboratory benchmarks and production environments.

Introduction

The rapid advancement of agentic AI—systems capable of autonomous planning and tool use—has sparked a "gold rush" in the technology sector. Industry forecasts promise transformative productivity gains; however, a persistent "performance gap" remains. Agents that excel in controlled technical benchmarks (e.g., coding or mathematical reasoning) frequently fail when deployed in complex, high-stakes environments like healthcare or finance.

This failure stems from a fundamental measurement imbalance. Current evaluation cultures treat technical metrics such as accuracy, speed, and token efficiency as sufficient proxies for value. In practice, sociotechnical dimensions—including user trust, workflow alignment, and long-term robustness—are the true determinants of whether an AI system becomes an asset or a liability. By neglecting these dimensions, the research community risks making premature and misleading claims regarding AI-driven productivity.

Methodology

The study utilizes a three-pronged research design:

  1. Meta-Analysis: A systematic review of 84 academic and industry papers (2023–2025), categorized by Technical, Human-centered, Safety/Governance, and Economic Impact metrics.
  2. Case Study Analysis: Qualitative investigations into healthcare, finance, and retail sectors to identify specific failure modes of technically high-performing models in production.
  3. Framework Development: The creation of a mathematical formalism ($U = w_T T + w_H H + w_R R + w_C C$) to allow practitioners to weight success dimensions according to domain-specific risk tolerances.

The analysis draws upon a corpus of 84 research publications and references established benchmarks such as MLAgentBench, PlanBench, and VisualWebArena, alongside human-centric instruments like TrAAIT.

The Balanced Four-Axis Evaluation Model

To address the measurement imbalance, we propose a multidimensional framework that recognizes the interdependence of technical and non-technical factors.

The Balanced Four-Axis Evaluation Model illustrating the interdependencies between technical, human-centered, temporal, and contextual dimensions.

As shown in Figure 1, the framework consists of four primary axes:

  • Technical: Focuses on core model performance and domain-specific constraints.
  • Human-Centered: Evaluates the level of trust in users and the calibration of the system based on user feedback.
  • Temporal: Measures robustness over time and the system's ability to handle evolution in user dynamics.
  • Contextual: Assesses alignment with domain workflows and the ability to adapt to changes in regulation and business environments.

Key Findings

The meta-analysis reveals a stark disconnect in current research priorities:

  • The 83/15 Disconnect: While 83% of papers report technical metrics, only 15% incorporate both technical and human dimensions. Longitudinal stability (Temporal axis) is addressed in only 5% of the literature.
  • Standardization Gap: 72% of technical metrics rely on standardized benchmarks, whereas human-centered and economic metrics are largely ad hoc, with only 18% and 12% respectively using validated instruments.
  • The ROI Drop: Evidence suggests that projected ROI for AI agents often collapses by 70–80% post-deployment due to unforeseen human oversight costs and adaptation lags.
  • Verification Burden: High technical scores can mask significant risks. For instance, top-tier LLMs were found to produce hallucinations in nearly all AI-generated medical summaries, imposing a heavy "verification burden" on clinicians that technical benchmarks failed to capture.

Discussion and Future Directions

The current "herd behavior" in AI evaluation—prioritizing static technical benchmarks—creates systemic risks, particularly in volatile sectors like finance. The interdependence of the four axes means that optimizing for technical performance in a vacuum can negatively impact human trust or contextual ROI.

Future research must focus on:

  • Standardizing "Soft" Metrics: Developing toolkits for measuring trust and workflow integration.
  • Longitudinal Benchmarking: Creating tests to predict performance degradation over months of operation.
  • Policy Integration: Encouraging regulators to mandate multidimensional reporting for high-stakes AI deployments.

While this study is limited by the potential exclusion of proprietary industry data, it serves as a critical call to action for a more holistic approach to AI evaluation.

Summary updated

BibTeX
@misc{meimandi2025measurement,
  author = {Kiana Jafari Meimandi and Gabriela Aránguiz-Dias and Grace Ra Kim and Lana Saadeddin and Mykel J. Kochenderfer},
  title = {{The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims}},
  year = {2025},
  eprint = {2506.02064},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02064},
}
Figure