ML Compass: Navigating Capability, Cost, and Compliance Trade-offs in AI Model Deployment

Vassilis Digalakis Jr, Ramayya Krishnan, Gonzalo Martin Fernandez, Agni OrfanoudakiarXiv 2025 · DOI · arXiv

Introduction

The rapid expansion of the AI landscape—with over 160 foundation models released between 2023 and 2024 alone—has created a paradox of choice for developers. While static capability leaderboards like MMLU or HumanEval offer a snapshot of raw performance, they are often disconnected from the practical realities of deployment. A model that excels in mathematical reasoning may be prohibitively expensive, too slow for real-time applications, or unable to meet the stringent safety regulations of sectors like healthcare. This "capability-deployment gap" suggests that technical excellence does not automatically translate into organizational value.

ML Compass (MLC) is a new framework designed to bridge this gap. By treating model selection as a constrained optimization problem, MLC helps organizations navigate the persistent trade-offs between user utility, deployment costs, and regulatory compliance.

The ML Compass Framework

MLC moves away from "chasing benchmarks" and instead focuses on the technological frontier—the boundary of what is currently possible in AI development. The framework is built on two primary pillars: a theoretical optimization model and a practical implementation pipeline.

Theoretical Foundations

The authors formalize model selection by proving that optimal configurations generally fall into a three-regime structure:

  1. Compliance Minima: Dimensions of the model (such as safety or bias mitigation) are held at the bare minimum required by law or internal policy.
  2. Feasible Maxima: Specific capabilities are pushed to the absolute limit of current technology to maximize a specific competitive advantage.
  3. Interior Values: Dimensions where the model is finely tuned to balance marginal utility against marginal cost, seeking the most efficient "middle ground."

The MLC Pipeline

To make this theory actionable, the MLC pipeline follows four distinct stages:

  • Measure Extraction: Using Factor Analysis, the framework reduces hundreds of noisy, overlapping benchmarks into a few stable "internal measures," such as "conversational experience" or "risk-aware competence."
  • Frontier Estimation: Through a process called "Pareto peeling," MLC identifies the most efficient models currently available and uses a Constant Elasticity of Substitution (CES) aggregator to map their capabilities to their associated costs.
  • Utility Estimation: The framework learns task-specific utility functions by analyzing actual interaction data, determining how users or experts value different model outputs.
  • Optimization: Finally, MLC recommends the specific model that maximizes "deployment value" under the user's unique budget and compliance constraints.

Empirical Validation

The authors validated MLC using two diverse environments to test its flexibility across general and specialized domains:

  1. PRISM Alignment Dataset: This study utilized 68,000 human-AI interactions across 75 countries to model heterogeneous human preferences in general conversation. It demonstrated how MLC can adapt to varying cultural and individual definitions of "utility."
  2. HealthBench: In a healthcare-specific case study, the authors developed a custom evaluation dataset. They processed 5,000 physician-designed healthcare prompts through various open-source models (including DeepSeek, Qwen, Gemma, and Llama) and scored them using expert rubrics.

Key Insights and Results

The Cost of Compliance

One of the most significant findings is the quantification of "spillover effects." The research shows that on the technological frontier, regulatory tightening (e.g., demanding higher safety scores) is rarely free. It typically forces a reallocation of resources that can degrade other capabilities or increase costs. MLC allows deployers to see exactly what they are sacrificing when they adjust their compliance thresholds.

Divergence from Standard Rankings

In both the PRISM and HealthBench studies, the models recommended by MLC differed materially from those at the top of "capability-only" leaderboards. For example, in healthcare, the optimal model for a "health professional" was driven by deep knowledge and reasoning, whereas the optimal model for providing "simple answers" to patients was driven by efficiency and speed. MLC successfully identified different optimal models for these distinct sub-tasks within the same organization.

Efficiency Gains

By moving away from a single "God-rank" of models, MLC consistently achieved higher deployment value than standard single-metric heuristics. It proves that model selection should be based on the "Value-to-Difficulty" ratio—identifying which model offers the best return on investment for a specific, constrained task rather than which model is the most powerful in a vacuum.

Limitations and Future Work

While ML Compass provides a robust roadmap for deployment, it currently relies on "static snapshots" of the market. As model prices fluctuate and new architectures emerge, the frontier must be constantly re-estimated. Future iterations of the framework may explore "dynamic deployment," where systems swap models in real-time based on the complexity of individual prompts. Additionally, the authors aim to create tighter links between expert-scored proxy outcomes and downstream organizational KPIs, such as patient health outcomes or financial ROI.

Summary updated

Abstract

We study how organizations should select among competing AI models when user utility, deployment costs, and compliance requirements jointly matter. Widely used capability leaderboards do not translate directly into deployment decisions, creating a capability -- deployment gap; to bridge it, we take a systems-level view in which model choice is tied to application outcomes, operating constraints, and a capability-cost frontier. We develop ML Compass, a framework that treats model selection as constrained optimization over this frontier. On the theory side, we characterize optimal model configurations under a parametric frontier and show a three-regime structure in optimal internal measures: some dimensions are pinned at compliance minima, some saturate at maximum levels, and the remainder take interior values governed by frontier curvature. We derive comparative statics that quantify how budget changes, regulatory tightening, and technological progress propagate across capability dimensions and costs. On the implementation side, we propose a pipeline that (i) extracts low-dimensional internal measures from heterogeneous model descriptors, (ii) estimates an empirical frontier from capability and cost data, (iii) learns a user- or task-specific utility function from interaction outcome data, and (iv) uses these components to target capability-cost profiles and recommend models. We validate ML Compass with two case studies: a general-purpose conversational setting using the PRISM Alignment dataset and a healthcare setting using a custom dataset we build using HealthBench. In both environments, our framework produces recommendations -- and deployment-aware leaderboards based on predicted deployment value under constraints -- that can differ materially from capability-only rankings, and clarifies how trade-offs between capability, cost, and safety shape optimal model choice.
BibTeX
@misc{jr2025ml,
  author = {Vassilis Digalakis Jr and Ramayya Krishnan and Gonzalo Martin Fernandez and Agni Orfanoudaki},
  title = {{ML Compass: Navigating Capability, Cost, and Compliance Trade-offs in AI Model Deployment}},
  year = {2025},
  eprint = {2512.23487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.23487},
}
Figure