In machine learning, we typically judge a model's success by its performance on a test set, condensed into a single number like accuracy, F-measure, or Area Under the Curve (AUC). However, a growing body of research suggests that these aggregate metrics are "blunt instruments" that mask a deeper, more unsettling phenomenon: predictive multiplicity.
Even when two models achieve identical aggregate scores, they often disagree on the specific predictions for a significant portion of individual cases. This instability means that a model’s decision for a particular person—whether they receive a loan, a medical diagnosis, or a high-risk classification—can be arbitrary, depending more on training noise than on the data itself.
The Illusion of Global Stability
The core of the problem lies in the "Rashomon Effect"—the existence of many different models that perform almost equally well on a dataset but represent different functional mappings. Marx et al. formalize this as predictive multiplicity, demonstrating that in high-stakes fields like recidivism prediction, "equally good" models can disagree on the fate of nearly half the population.
Standard metrics fail to capture this because they prioritize the average over the individual. As Rahmani et al. note, metrics like AUC can remain stable even when a model’s underlying predictive strategy is fluctuating. This creates a false sense of reliability. For instance, Dutta et al. found that when Large Language Models (LLMs) are compressed, their aggregate accuracy often stays the same, yet they "flip" their answers on 10-15% of individual questions. The global score is preserved, but the model’s local behavior has fundamentally changed.
Sources of Prediction Churn
What causes this instability? Research identifies several "jittery" components in the machine learning pipeline:
- Training Stochasticity: Minor variations, such as changing a random seed, can lead to wildly different internal representations. Du et al. and Nguyen argue that surface-level performance consistency does not guarantee a stable functional state, as models with identical accuracy often exhibit significant disagreement on individual test cases.
- Prompt Sensitivity: In the context of LLMs, Zhuo et al. show that models are highly volatile at the individual question level, failing on different subsets of data when prompt wording is slightly modified, even if average accuracy remains constant.
- Privacy Constraints: The randomness required for Differential Privacy (DP) introduces a hidden "arbitrariness cost." Kulynych et al. demonstrate that different DP models, trained on the same data with the same privacy guarantees, can reach conflicting conclusions for the same individual.
The Human Cost of Arbitrariness
In high-stakes domains, this lack of per-instance consistency is not just a technical curiosity; it is a reliability crisis. In clinical settings, Lopez Martinez et al. found that models can maintain stable ROC-AUC scores while producing wildly different risk rankings for specific patients across training runs. Similarly, Tran highlights that models with identical dataset-level scores may provide conflicting risk assessments for individual high-stakes cases, leading to brittle clinical decisions.
This instability also complicates fairness. Long et al. argue that traditional fairness interventions often ignore predictive multiplicity. A model might appear "fair" on a group level while remaining highly inconsistent for individuals, making the specific model choice appear arbitrary to the end-user.
Toward Granular Reliability
To address these failures, researchers are moving toward metrics that evaluate predictability and local stability.
Pacchiardi et al. propose shifting focus from average accuracy to predictability, identifying "safe zones" where a model can be trusted. To detect instability without the cost of retraining multiple models, Hamman et al. introduce a "local stability" measure that probes a model's behavior in the embedding space around specific data points.
When stability cannot be achieved by a single model, ensembling offers a provable solution. Wang et al. demonstrate that ensembles are inherently more stable than single models, reducing the likelihood of a system "flipping" its decision on a specific sample during a version update. By prioritizing instance-level consistency over mere aggregate accuracy, these approaches aim to move machine learning from global utility toward individual reliability.
Key Foundational Work
Quantifying the reliability of fault classifiers (2014) — contributed 1 concept elements to this direction.
Predicting the functions of proteins in Protein-Protein Interaction networks from global information (2010) — contributed 1 concept elements to this direction.