Abstract
Selective classification enables machine learning models to abstain from making predictions on uncertain inputs, a critical feature for high-stakes applications. This paper introduces SCROSS (Selective CROSS-classification), a model-agnostic framework that allows any probabilistic classifier to meet target coverage goals without requiring a separate validation set. By leveraging cross-fitting and a second-order quantile estimation strategy, SCROSS achieves competitive selective accuracy and superior computational efficiency compared to specialized deep learning methods.
Introduction
In fields like healthcare and credit scoring, a wrong prediction can have severe consequences. Selective classification, or "classification with a reject option," addresses this by allowing models to output "I don't know" when confidence is low. However, state-of-the-art methods like SelectiveNet or Self-Adaptive Training (SAT) are often tied to specific Deep Neural Network (DNN) architectures. This creates a barrier for tabular data applications where gradient-boosted trees typically outperform DNNs.
SCROSS is designed to be model-agnostic, data-efficient, and scalable. It acts as a wrapper for any classifier that outputs probabilities, making it particularly useful for the tabular datasets prevalent in industry.
Methodology: The SCROSS Framework
SCROSS relies on two primary statistical techniques to calibrate rejection thresholds without sacrificing training data:
- Cross-Fitting: SCROSS employs a $K$-fold strategy. It trains $K$ models on different subsets and uses out-of-fold predictions to build a distribution of confidence scores. This allows the final model to be trained on the entire dataset while maintaining a robust threshold estimation.
- Advanced Quantile Estimation: To determine the rejection threshold $\theta$, SCROSS uses a weighted subsample estimator. This approach reduces the variance of the threshold estimate, ensuring the model more accurately hits its target coverage (the percentage of samples it chooses to classify).
The framework's versatility is demonstrated by its performance across diverse base learners, including Logistic Regression, Random Forests, and XGBoost.
Experimental Results
The authors evaluated SCROSS against eight datasets, including financial (Lending Club, UCI Credit) and image (CatsVsDogs) data.
Performance and Efficiency
SCROSS consistently hits target coverage more accurately than baselines. In tabular domains, SCROSS paired with LightGBM often outperforms specialized DNN methods like SelectiveNet in selective accuracy.
Crucially, SCROSS offers a massive reduction in computational cost. As shown in the experimental results, SCROSS can be orders of magnitude faster than DNN-based alternatives. For instance, on the Adult dataset, SCROSS trains in under 2 seconds, while SelectiveNet requires over 2,000 seconds.
Key Insights
- Tabular Dominance: While DNN methods excel in computer vision, tree-based models (like XGBoost) remain superior for tabular data. SCROSS brings the benefits of selective classification to these high-performing models.
- Threshold Stability: By minimizing the variance of the rejection threshold, SCROSS provides more predictable behavior for practitioners who need to guarantee specific coverage levels.
Discussion and Future Work
While SCROSS is highly effective, challenges remain. The method's performance can dip on extremely imbalanced datasets where quantile estimation becomes difficult at the tails. Furthermore, the authors highlight the need to integrate fairness constraints, as selective classification can inadvertently lead to higher rejection rates for specific demographic groups. Future research will also explore extending the framework to non-additive loss functions and integrating explainable AI (XAI) to provide reasons for model abstention.
Conclusion
SCROSS provides a practical, efficient, and model-agnostic solution for selective classification. By decoupling the rejection logic from the model architecture, it empowers practitioners to use the best model for their data while maintaining the safety net of a reject option.
Summary updated