Combating Sampling Bias: A Self-Training Method in Credit Risk Models

Jingxian Liao, Wei Wang, Jason Xue, Anthony Lei, Xue Han, Kun LuAAAI 2022 · DOI

Abstract

Credit risk models often suffer from sampling bias because they are trained only on approved applicants, a challenge known as the "reject inference" problem. This paper proposes a self-training framework that uses calibrated probabilities to selectively label and incorporate rejected applications into the training set. Experimental results on real-world data from Intuit show that this approach increases loan approval rates by up to 8.8% without increasing default risk, demonstrating a significant improvement over traditional methods.

Introduction: The Challenge of Sampling Bias

In credit lending, machine learning models face a fundamental "catch-22" known as sampling bias. To train a model to predict default risk, researchers require labels indicating whether a borrower repaid ("Good") or defaulted ("Bad"). However, these labels are only available for applicants who were previously approved.

This creates a skewed view of the applicant pool. If a previous policy rejected everyone with specific characteristics, the model never learns that some of those individuals might have been reliable borrowers. This phenomenon, known as the Reject Inference problem, often leads to overly conservative models. Traditional solutions like "Fuzzy Augmentation" (assigning partial labels to all rejected cases) frequently introduce noise that can degrade model performance compared to simply ignoring the rejected pool.

Methodology: Selective Self-Training

The authors propose a Self-Training Method with a Confidence Model to address these biases. Rather than attempting to label every rejected applicant, the algorithm iteratively selects only the "most certain" cases to augment the training set.

The process follows a specific loop:

  1. Base Training: A model (e.g., XGBoost) is trained on known "accepted" data.
  2. Prediction: The model predicts outcomes for the unlabeled "rejected" pool.
  3. Filtering: Only predictions meeting a high confidence threshold are assigned "pseudo-labels."
  4. Augmentation: These pseudo-labeled cases are added to the training set for the next iteration.
The iterative self-training framework for reject inference.

The paper evaluates two primary confidence measures: Trust Score (a density-based approach) and Probability Calibration (ST-CP). The latter uses Isotonic Regression to ensure that predicted probabilities accurately reflect real-world default rates.

Data and Validation

The study utilizes a real-world dataset from Intuit (QuickBooks Online) comprising approximately 50,000 anonymized loan applications. To validate the model's performance on rejected applicants—who typically lack labels—the researchers cross-referenced external credit bureau data. This provided a "ground truth" for how rejected individuals performed on loans obtained elsewhere, creating a high-quality benchmark for testing.

Key Results and Business Impact

The proposed ST-CP method (Self-Training with Calibrated Probability) demonstrated superior performance, particularly for XGBoost models. The researchers prioritized a business-centric metric: Approval Rate at a fixed Default Rate, which measures the ability to approve more customers without increasing overall risk.

Performance comparison of different credit risk models and reject inference methods.

Key findings include:

  • Experimental Gains: ST-CP increased the loan approval rate by 8.8% over the baseline while maintaining a constant default rate.
  • Production Success: Following deployment at Intuit in March 2021, the model achieved a 6.8% improvement in the K-S statistic, indicating better separation between good and bad borrowers.
  • Expanded Reach: In production, the method allowed the institution to serve 39% more applicants while maintaining stable predicted risk levels.

Conclusion

The study highlights that selective learning is more effective than bulk labeling. By focusing on the most confident predictions through calibrated probabilities, models can generalize better to the entire applicant population. While there is a "sweet spot" for data augmentation—typically around a 15-26% increase in training size—this self-training approach offers a robust path toward more inclusive and accurate credit scoring. Future work will explore more sophisticated uncertainty estimates to further refine the pseudo-labeling process.

Summary updated

BibTeX
@inproceedings{liao2022combating,
  author = {Jingxian Liao and Wei Wang and Jason Xue and Anthony Lei and Xue Han and Kun Lu},
  title = {{Combating Sampling Bias: A Self-Training Method in Credit Risk Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21528},
}
Figure