A Model-Agnostic Heuristics for Selective Classification

Andrea Pugnana, Salvatore Ruggieri

Abstract

Selective classification enables machine learning models to abstain from making predictions on uncertain inputs, a critical feature for high-stakes applications. This paper introduces SCROSS (Selective CROSS-classification), a model-agnostic framework that allows any probabilistic classifier to meet target coverage goals without requiring a separate validation set. By leveraging cross-fitting and a second-order quantile estimation strategy, SCROSS achieves competitive selective accuracy and superior computational efficiency compared to specialized deep learning methods.

Introduction

In fields like healthcare and credit scoring, a wrong prediction can have severe consequences. Selective classification, or "classification with a reject option," addresses this by allowing models to output "I don't know" when confidence is low. However, state-of-the-art methods like SelectiveNet or Self-Adaptive Training (SAT) are often tied to specific Deep Neural Network (DNN) architectures. This creates a barrier for tabular data applications where gradient-boosted trees typically outperform DNNs.

SCROSS is designed to be model-agnostic, data-efficient, and scalable. It acts as a wrapper for any classifier that outputs probabilities, making it particularly useful for the tabular datasets prevalent in industry.

Methodology: The SCROSS Framework

SCROSS relies on two primary statistical techniques to calibrate rejection thresholds without sacrificing training data:

  1. Cross-Fitting: SCROSS employs a $K$-fold strategy. It trains $K$ models on different subsets and uses out-of-fold predictions to build a distribution of confidence scores. This allows the final model to be trained on the entire dataset while maintaining a robust threshold estimation.
  2. Advanced Quantile Estimation: To determine the rejection threshold $\theta$, SCROSS uses a weighted subsample estimator. This approach reduces the variance of the threshold estimate, ensuring the model more accurately hits its target coverage (the percentage of samples it chooses to classify).

The framework's versatility is demonstrated by its performance across diverse base learners, including Logistic Regression, Random Forests, and XGBoost.

Performance of SCROSS across different base learners, demonstrating its model-agnostic nature.

Experimental Results

The authors evaluated SCROSS against eight datasets, including financial (Lending Club, UCI Credit) and image (CatsVsDogs) data.

Performance and Efficiency

SCROSS consistently hits target coverage more accurately than baselines. In tabular domains, SCROSS paired with LightGBM often outperforms specialized DNN methods like SelectiveNet in selective accuracy.

Crucially, SCROSS offers a massive reduction in computational cost. As shown in the experimental results, SCROSS can be orders of magnitude faster than DNN-based alternatives. For instance, on the Adult dataset, SCROSS trains in under 2 seconds, while SelectiveNet requires over 2,000 seconds.

Comparison of SCROSS against PLUGIN, SELNET, and SAT baselines across multiple datasets, highlighting empirical coverage, selective accuracy, and training time.

Key Insights

  • Tabular Dominance: While DNN methods excel in computer vision, tree-based models (like XGBoost) remain superior for tabular data. SCROSS brings the benefits of selective classification to these high-performing models.
  • Threshold Stability: By minimizing the variance of the rejection threshold, SCROSS provides more predictable behavior for practitioners who need to guarantee specific coverage levels.

Discussion and Future Work

While SCROSS is highly effective, challenges remain. The method's performance can dip on extremely imbalanced datasets where quantile estimation becomes difficult at the tails. Furthermore, the authors highlight the need to integrate fairness constraints, as selective classification can inadvertently lead to higher rejection rates for specific demographic groups. Future research will also explore extending the framework to non-additive loss functions and integrating explainable AI (XAI) to provide reasons for model abstention.

Conclusion

SCROSS provides a practical, efficient, and model-agnostic solution for selective classification. By decoupling the rejection logic from the model architecture, it empowers practitioners to use the best model for their data while maintaining the safety net of a reject option.

Summary updated

Abstract

Selective classification (also known as classification with reject option) conservatively extends a classifier with a selection function to determine whether or not a prediction should be accepted (i.e., trusted, used, deployed). This is a highly relevant issue in socially sensitive tasks, such as credit scoring. State-of-the-art approaches rely on Deep Neural Networks (DNNs) that train at the same time both the classifier and the selection function. These approaches are model-specific and computationally expensive. We propose a model-agnostic approach, as it can work with any base probabilistic binary classification algorithm, and it can be scalable to large tabular datasets if the base classifier is so. The proposed algorithm, called SCROSS, exploits a cross-fitting strategy and theoretical results for quantile estimation to build the selection function. Experiments on real-world data show that SCROSS improves over existing methods.
BibTeX
@inproceedings{pugnana2023model,
  author = {Andrea Pugnana and Salvatore Ruggieri},
  title = {{A Model-Agnostic Heuristics for Selective Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26133},
}
Figure