Abstract: As artificial intelligence (AI) becomes central to critical decision-making, the reliability of individual predictions becomes paramount. This paper proposes a novel data-centric approach to quantify "distrust" in predictions by evaluating the training data's fitness for specific queries. By measuring local representation and uncertainty, the proposed algorithms provide scalable, model-agnostic reliability scores that correlate strongly with actual model performance.
Introduction
Standard machine learning models typically offer "average" performance guarantees across a dataset. However, in high-stakes fields like medicine or finance, a decision-maker needs to know if the model is reliable for a specific instance. Current reliability measures are often model-centric, relying on internal states or output probabilities. This paper argues that such approaches have a blind spot: if a model is trained on biased or sparse data, it may produce a "confident" but incorrect prediction simply because it has never seen a similar case.
Methodology: Representation and Uncertainty (RU)
The authors introduce two data-centric reliability measures based on the $k$-vicinity (the neighborhood of the $k$-nearest neighbors) of a query point:
- Representativeness: Does the training set contain enough samples similar to the query point? If the query point is an outlier, the model's learned patterns are unlikely to apply.
- Uncertainty: Even if similar samples exist, do their labels fluctuate? High variance in the target values of a query point's neighbors suggests the region is inherently noisy.
From these, two distrust scores are derived: weakRU, which is raised if a point is either an outlier or in an uncertain region, and strongRU, a more conservative warning raised only if both conditions are met.
To ensure practical utility, the authors developed sub-linear algorithms for efficient computation. They also designed a "no-data access" version that uses proxy regression models to estimate RU scores at inference time without needing to store the full training dataset.
Experimental Evaluation
The framework was tested on a mix of synthetic and real-world benchmarks, including financial risk (Default of Credit Card Clients) and high-dimensional image classification (Gisette).
Key Results
- High Correlation with Error: RU scores showed a consistent Pearson correlation (up to 0.91) with actual model failure. As RU scores increased, model accuracy plummeted and error rates (MSE/FPR) spiked.
- Efficiency: The algorithms are highly scalable; preprocessing 100,000 points takes less than a second, and query times remain sub-second even as dataset sizes grow.
- Model-Agnosticism: Because these measures look at the data rather than the model, they work effectively across various architectures, from simple Logistic Regression to complex Deep Neural Networks.
Conclusion
This research highlights the necessity of dismissing prediction outcomes for cases with high distrust values. By focusing on the "scope of use" defined by the training data, the RU measures provide a critical safety layer for responsible AI. Future work will explore integrating these data-centric warnings directly into the training loop to help models better understand their own limitations.
Summary updated