Abstract
Credit risk models often suffer from sampling bias because they are trained only on approved applicants, a challenge known as the "reject inference" problem. This paper proposes a self-training framework that uses calibrated probabilities to selectively label and incorporate rejected applications into the training set. Experimental results on real-world data from Intuit show that this approach increases loan approval rates by up to 8.8% without increasing default risk, demonstrating a significant improvement over traditional methods.
Introduction: The Challenge of Sampling Bias
In credit lending, machine learning models face a fundamental "catch-22" known as sampling bias. To train a model to predict default risk, researchers require labels indicating whether a borrower repaid ("Good") or defaulted ("Bad"). However, these labels are only available for applicants who were previously approved.
This creates a skewed view of the applicant pool. If a previous policy rejected everyone with specific characteristics, the model never learns that some of those individuals might have been reliable borrowers. This phenomenon, known as the Reject Inference problem, often leads to overly conservative models. Traditional solutions like "Fuzzy Augmentation" (assigning partial labels to all rejected cases) frequently introduce noise that can degrade model performance compared to simply ignoring the rejected pool.
Methodology: Selective Self-Training
The authors propose a Self-Training Method with a Confidence Model to address these biases. Rather than attempting to label every rejected applicant, the algorithm iteratively selects only the "most certain" cases to augment the training set.
The process follows a specific loop:
- Base Training: A model (e.g., XGBoost) is trained on known "accepted" data.
- Prediction: The model predicts outcomes for the unlabeled "rejected" pool.
- Filtering: Only predictions meeting a high confidence threshold are assigned "pseudo-labels."
- Augmentation: These pseudo-labeled cases are added to the training set for the next iteration.
The paper evaluates two primary confidence measures: Trust Score (a density-based approach) and Probability Calibration (ST-CP). The latter uses Isotonic Regression to ensure that predicted probabilities accurately reflect real-world default rates.
Data and Validation
The study utilizes a real-world dataset from Intuit (QuickBooks Online) comprising approximately 50,000 anonymized loan applications. To validate the model's performance on rejected applicants—who typically lack labels—the researchers cross-referenced external credit bureau data. This provided a "ground truth" for how rejected individuals performed on loans obtained elsewhere, creating a high-quality benchmark for testing.
Key Results and Business Impact
The proposed ST-CP method (Self-Training with Calibrated Probability) demonstrated superior performance, particularly for XGBoost models. The researchers prioritized a business-centric metric: Approval Rate at a fixed Default Rate, which measures the ability to approve more customers without increasing overall risk.
Key findings include:
- Experimental Gains: ST-CP increased the loan approval rate by 8.8% over the baseline while maintaining a constant default rate.
- Production Success: Following deployment at Intuit in March 2021, the model achieved a 6.8% improvement in the K-S statistic, indicating better separation between good and bad borrowers.
- Expanded Reach: In production, the method allowed the institution to serve 39% more applicants while maintaining stable predicted risk levels.
Conclusion
The study highlights that selective learning is more effective than bulk labeling. By focusing on the most confident predictions through calibrated probabilities, models can generalize better to the entire applicant population. While there is a "sweet spot" for data augmentation—typically around a 15-26% increase in training size—this self-training approach offers a robust path toward more inclusive and accurate credit scoring. Future work will explore more sophisticated uncertainty estimates to further refine the pseudo-labeling process.
Summary updated