When evaluating a machine learning model, the standard practice is to compress its performance into a single, aggregate scalar—such as average accuracy, F1-score, or Area Under the ROC Curve (AUC-ROC). This utilitarian approach treats every data point with equal weight, calculating a global average to determine whether a model is ready for deployment.
However, a growing body of research demonstrates that this reliance on aggregate metrics creates a dangerous statistical illusion. By flattening multidimensional performance into a single number, aggregate metrics systematically mask catastrophic failures, severe demographic biases, and acute vulnerabilities within specific subgroups or long-tail edge cases. A model can appear exceptionally performant on average while failing completely for millions of users or high-stakes scenarios.
The Mathematics of the Mask
To understand why aggregate metrics fail, consider a dataset where 95% of the samples represent a dominant "majority" category (e.g., clean data, frequent entities, or a majority demographic group) and 5% represent a "minority" category (e.g., noisy data, rare entities, or a marginalized demographic).
If a model achieves 100% accuracy on the majority group but 0% accuracy on the minority group, its aggregate accuracy remains a stellar 95%. As Ryan Burnell et al. argue in Rethink reporting of evaluation results in AI, this masking effect routinely greenlights AI systems that are fundamentally unsafe for deployment in high-stakes domains like medicine or autonomous driving.
This phenomenon manifests across diverse domains:
- Audio Deepfake Detection: In Bona fide Cross Testing Reveals Weak Spot in Audio Deepfake Detection Systems, Jia Qi Yip et al. demonstrate that deepfake detectors evaluated on datasets dominated by clean, high-quality samples yield deceptively low error rates. When exposed to real-world environments containing background noise or emotional speech, the models fail over 30% of the time—a vulnerability completely hidden by the global average.
- Image Super-Resolution: Evaluating models using average scores across random samples obscures localized failures. Wenlong Zhang et al. show in SEAL: A Framework for Systematic Evaluation of Real-World Super-Resolution that models frequently over-perform on easy cases while completely failing to handle specific types of complex data corruption.
- Entity Linking: In knowledge bases, a small number of highly frequent "head" entities dominate the evaluation set. Filip Ilievski et al. establish in Systematic Study of Long Tail Phenomena in Entity Linking that standard Micro F1-scores obscure near-total system failures on rare, long-tail entities.
How Aggregation Conceals Disparities
Masking does not just happen because of simple data scarcity; it is often actively induced by the way data is structured and categorized during audits.
Simpson's Paradox in Biometric Galleries
In 1:N face identification, a model searches a "gallery" to find a match. Valeriia Cherepanova et al. reveal a striking manifestation of Simpson's paradox in A Deep Dive into Dataset Imbalance and Bias in Face Identification. When a test gallery contains the same demographic imbalances as the training data, the performance metrics falsely suggest a fair system. The algorithmic accuracy gained from over-training on the majority group is mathematically offset by the increased difficulty of searching through a larger number of majority distractors in the gallery. This statistical interplay completely hides underlying model bias during standard audits.
The Washout Effect of Coarse Categorization
When fairness audits rely on broad demographic or geographic categories, they inadvertently wash out severe localized harms. Rajiv Movva et al. demonstrate this in Coarse race data conceals disparities in clinical risk score performance, showing that evaluating clinical risk scores using broad racial categories (e.g., "Asian") conceals massive performance disparities among granular subgroups. The model's high performance on a privileged subgroup offsets poor performance on a vulnerable one, yielding an acceptable but highly misleading average.
Similarly, in geographic language processing, Yizhi Li et al. show in HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models that a language model can appear entirely neutral when evaluated at a continental level, while simultaneously harboring intense, conflicting stereotypes about individual cities and sub-regions within that continent.
The Perverse Incentives of Mean Optimization
The reliance on aggregate metrics does more than misinform; it distorts the optimization process itself. Because machine learning algorithms are designed to minimize global loss, they naturally exploit the easiest statistical paths to achieve that goal.
In Efficient Bias Mitigation Without Privileged Information, Mateo Espinosa Zarlenga et al. point out that peak aggregate accuracy frequently coincides with a model's maximum reliance on spurious features. If a stereotypical correlation holds true for the majority of the data, the model will aggressively leverage it to maximize the mean score, destroying its ability to generalize to minority slices.
Consequently, optimizing for aggregate performance can actively worsen demographic bias. Ninareh Mehrabi et al. document this counterintuitive reality in Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition, finding that newer iterations of language models that achieve higher overall accuracy scores actually exhibit significantly increased demographic bias, frequently failing to recognize female names as people compared to their less accurate predecessors.
[Training Pipeline] ──> Optimizes Global Mean ──> Exploits Spurious Majority Shortcuts
│
▼
[Standard Evaluation] <── High Aggregate Score <── Conceals Catastrophic Minority Failures
Designing Beyond the Average
To build reliable and equitable AI systems, the machine learning community is developing alternative evaluation frameworks that move away from simple utilitarian averages.
Social Choice Theory
Instead of treating benchmark aggregation as a simple arithmetic mean, Mark Rofin et al. introduce Vote'n'Rank: Revision of Benchmarking with Social Choice Theory. By modeling datasets and tasks as "voters" and models as "candidates," this framework applies voting rules to aggregate performance. This prevents a model from securing a top leaderboard ranking by hyper-performing on easy tasks while completely failing on difficult ones, forcing a truer assessment of generalizability.
Unsupervised Subgroup Discovery
In clinical settings, predefined demographic categories may not capture the true axes of model failure. To protect vulnerable populations in intensive care units, Harini Suresh et al. develop a framework in Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU that uses unsupervised learning to discover latent patient subgroups. By treating these discovered subgroups as distinct tasks within a multi-task learning framework, they prevent minority patient clusters from being mathematically erased by global population averages.
Intersectional and Granular Auditing
To make multi-feature failures visible to developers, interactive tools and documentation standards are becoming essential. David Munechika et al. developed Visual Auditor: Interactive Visualization for Detection and Summarization of Model Biases to automatically surface data slices where a model underperforms across combinations of demographic and behavioral traits. This computational approach pairs with organizational frameworks like Model Cards, proposed by Ben Hutchinson et al. in Model Cards for Model Reporting, which mandate the explicit breakdown of performance metrics across distinct demographic, phenotypic, and intersectional groups before a model is released.