Data-Centric Distrust Quantification for Responsible AI: When Data-driven Outcomes Are Not Reliable

Nima Shahbazi, Abolfazl AsudeharXiv 2022 · arXiv · OpenReview

Abstract: As artificial intelligence (AI) becomes central to critical decision-making, the reliability of individual predictions becomes paramount. This paper proposes a novel data-centric approach to quantify "distrust" in predictions by evaluating the training data's fitness for specific queries. By measuring local representation and uncertainty, the proposed algorithms provide scalable, model-agnostic reliability scores that correlate strongly with actual model performance.

Introduction

Standard machine learning models typically offer "average" performance guarantees across a dataset. However, in high-stakes fields like medicine or finance, a decision-maker needs to know if the model is reliable for a specific instance. Current reliability measures are often model-centric, relying on internal states or output probabilities. This paper argues that such approaches have a blind spot: if a model is trained on biased or sparse data, it may produce a "confident" but incorrect prediction simply because it has never seen a similar case.

Example of a reliability dashboard for a loan application query, showing how the system flags a prediction as 'Not Reliable' due to low representation.

Methodology: Representation and Uncertainty (RU)

The authors introduce two data-centric reliability measures based on the $k$-vicinity (the neighborhood of the $k$-nearest neighbors) of a query point:

  1. Representativeness: Does the training set contain enough samples similar to the query point? If the query point is an outlier, the model's learned patterns are unlikely to apply.
  2. Uncertainty: Even if similar samples exist, do their labels fluctuate? High variance in the target values of a query point's neighbors suggests the region is inherently noisy.

From these, two distrust scores are derived: weakRU, which is raised if a point is either an outlier or in an uncertain region, and strongRU, a more conservative warning raised only if both conditions are met.

Conceptual visualization of query points ($q^1$ to $q^4$) relative to training data distribution. Points like $q^3$ represent regions with low representation, leading to higher distrust scores.

To ensure practical utility, the authors developed sub-linear algorithms for efficient computation. They also designed a "no-data access" version that uses proxy regression models to estimate RU scores at inference time without needing to store the full training dataset.

Experimental Evaluation

The framework was tested on a mix of synthetic and real-world benchmarks, including financial risk (Default of Credit Card Clients) and high-dimensional image classification (Gisette).

Key Results

  • High Correlation with Error: RU scores showed a consistent Pearson correlation (up to 0.91) with actual model failure. As RU scores increased, model accuracy plummeted and error rates (MSE/FPR) spiked.
  • Efficiency: The algorithms are highly scalable; preprocessing 100,000 points takes less than a second, and query times remain sub-second even as dataset sizes grow.
  • Model-Agnosticism: Because these measures look at the data rather than the model, they work effectively across various architectures, from simple Logistic Regression to complex Deep Neural Networks.

Conclusion

This research highlights the necessity of dismissing prediction outcomes for cases with high distrust values. By focusing on the "scope of use" defined by the training data, the RU measures provide a critical safety layer for responsible AI. Future work will explore integrating these data-centric warnings directly into the training loop to help models better understand their own limitations.

Summary updated

Abstract

At the same time that artificial intelligence (AI) and machine learning are becoming central to human life, their potential harms become more vivid. In the presence of such drawbacks, a critical question to address before using individual predictions for critical decision-making is whether those are reliable. Aligned with recent efforts on data-centric AI, this paper proposes a novel approach, complementary to the existing work on trustworthy AI, to address the reliability question through the lens of data. Specifically, it associates data sets with distrust quantification that specifies their scope of use for individual predictions. It develops novel algorithms for efficient and effective computation of distrust values. The proposed algorithms learn the necessary components of the measures from the data itself and are sublinear, which makes them scalable to very large and multi-dimensional settings. Furthermore, an estimator is designed to enable no-data access during the query time. Besides theoretical analyses, the algorithms are evaluated experimentally, using multiple real and synthetic data sets and different tasks. The experiment results reflect a consistent correlation between distrust values and model performance. This highlights the necessity of dismissing prediction outcomes for cases with high distrust values, at least for critical decisions.
BibTeX
@misc{shahbazi2022data,
  author = {Nima Shahbazi and Abolfazl Asudeh},
  title = {{Data-Centric Distrust Quantification for Responsible AI: When Data-driven Outcomes Are Not Reliable}},
  year = {2022},
  eprint = {2204.07682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07682},
}
Figure