Abstract
Credit scoring is a critical component of financial risk management, yet it faces significant challenges regarding data scalability, extreme class imbalance, and privacy constraints. This paper introduces a novel framework that integrates dataset distillation with TabPFN, a pretrained transformer for tabular data. By employing "imbalance-aware" loss functions such as Focal Loss (FL) and Asymmetric Sigmoid Adjusted (ASIG) functions, the proposed method compresses massive, imbalanced financial datasets into small, representative synthetic sets. Experimental results demonstrate that this approach enables high-performance Bayesian inference on large-scale data while maintaining privacy and improving predictive accuracy for minority classes.
Introduction
Traditional credit scoring has long been dominated by gradient-boosted decision trees. While pretrained models like TabPFN offer a promising alternative, their application in finance is hindered by three primary issues:
- Scalability: TabPFN is optimized for small datasets and loses efficiency as sample sizes grow.
- Class Imbalance: Defaulters typically represent a tiny fraction of the population, often leading models to overlook critical risk signals.
- Privacy: Legal and commercial barriers prevent the sharing of raw financial records.
This research bridges these gaps by distilling the essence of large datasets into a fraction of their original size, making them compatible with advanced pretrained architectures.
Methodology
The proposed pipeline utilizes Dataset Distillation (DD) based on Kernel Inducing Points (KIP) to create a "coreset" of synthetic data.
Imbalance-Aware Distillation
Standard distillation techniques often fail to capture the nuances of minority classes. To counter this, the authors integrated specialized loss functions into the distillation process:
- Focal Loss (FL): Focuses the distillation on "hard" examples by down-weighting easy-to-classify samples.
- Asymmetric Sigmoid Adjusted Function (ASIG): A novel approach designed to shift decision boundaries, ensuring better representation of the minority "default" class.
Algorithmic Flow
The process involves distilling original datasets (ODs) into distilled datasets (DDs) using various objective functions and sample sizes, followed by model training and cross-evaluation.
Experimental Setup and Datasets
The framework was evaluated on four diverse financial datasets:
- GMC (Give Me Some Credit): 150,000 samples with a high imbalance ratio (IR) of 14.9.
- Credit Card: 30,000 samples (IR 4.5).
- CHN Banks: 60,780 samples (IR 5.7).
- Lending Club: 14,785 samples (IR 1.7).
Key Insights and Results
Performance and Efficiency
The inclusion of Focal Loss during distillation resulted in a 2.5% improvement in AUC for TabPFN compared to standard methods. Remarkably, models trained on distilled sets containing only 1% of the original data achieved performance levels nearly identical to those trained on the full datasets.
Privacy and Distribution
Since the distilled samples are synthetic inducing points rather than actual customer records, the framework provides a robust layer of privacy. PCA visualizations confirmed that as the distilled size increases to approximately 500-900 samples, the synthetic data distribution converges to replicate the complex patterns of the original full dataset.
Model Synergy
While TabPFN is effective for small-scale inference, the study found that tree-based models like LightGBM also benefit significantly from distilled data, suggesting that the distillation process successfully captures the underlying structural patterns of the data.
Conclusion
This research demonstrates that dataset distillation is a powerful tool for scaling pretrained tabular models to enterprise-level financial tasks. By making distillation "imbalance-aware," the authors provide a pathway for more accurate and private credit risk assessment. Future work will focus on refining the theoretical underpinnings of asymmetric distillation to further enhance performance on highly biased datasets.
Summary updated