Abstract
This paper aims to explore models based on the extreme gradient boosting (XGBoost) approach for business risk classification. Feature selection (FS) algorithms and hyper-parameter optimizations are simultaneously considered during model training. The five most commonly used FS methods including weight by Gini, weight by Chi-square, hierarchical variable clustering, weight by correlation, and weight by information are applied to alleviate the effect of redundant features. Two hyper-parameter optimization approaches, random search (RS) and Bayesian tree-structured Parzen Estimator (TPE), are applied in XGBoost. The performance of XGBoost is compared to the traditionally utilized logistic regression (LR) model in terms of classification accuracy, area under the curve (AUC), recall, and F1 score obtained from the 10-fold cross validation. Results show that hierarchical clustering is the optimal FS method for LR while weight by Chi-square achieves the best performance in XGBoost. Both TPE and RS optimization in XGBoost outperform LR significantly. TPE optimization shows a superiority over RS since it results in a significantly higher accuracy and a marginally higher AUC, recall and F1 score. Furthermore, XGBoost with TPE tuning shows a lower variability than the RS method. Finally, the ranking of feature importance based on XGBoost enhances the model interpretation. Therefore, XGBoost with Bayesian TPE hyper-parameter optimization serves as an operative while powerful approach for business risk modeling.
Introduction
In the financial sector, risk modeling serves as the primary gatekeeper for credit. Traditionally, banks have relied on Logistic Regression (LR) due to its ease of implementation and transparency. However, as datasets grow in complexity, simple linear models often struggle to capture non-linear relationships, leading to sub-optimal loan decisions. While ensemble methods like XGBoost offer a powerful alternative, they introduce challenges regarding feature redundancy and the difficulty of hyper-parameter tuning. This paper explores how to simultaneously optimize feature selection and hyper-parameter tuning to create a business risk model that is both highly accurate and stable.
Methodology
The researchers compared a traditional LR baseline against two versions of XGBoost using a multi-stage pipeline. The study utilized a massive real-world dataset from a US national credit bureau containing over 10 million de-identified commercial records. After initial pre-processing, a stratified sample of 8,000 observations was used for the core experiments. The overall workflow is illustrated in Figure 1.
- Data Pre-processing: Includes variable normalization and missing value imputation.
- Feature Selection (FS): Five methods were tested to reduce noise: Weight by Gini, Chi-square, Hierarchical Variable Clustering, Correlation, and Information Gain.
- Modeling: Comparison between LR and XGBoost variants.
- Scoring: Applying models to the validation set.
- Model Evaluation: Performance was measured using Accuracy, AUC, Recall, and F1 score, validated via the Wilcoxon Signed Rank Test.
Hyper-parameter Optimization
A critical component of the XGBoost implementation was the comparison of two optimization approaches: Random Search (RS) and Bayesian Tree-structured Parzen Estimator (TPE). While RS picks parameters randomly, TPE uses a probabilistic approach to learn from previous iterations. The specific parameters tuned and their respective search domains are detailed in Figure 2.
Key Results and Discussion
The experimental results provided several significant insights into modern risk modeling:
- Feature Selection is Model-Dependent: The "best" features depend on the algorithm. For LR, Hierarchical Clustering was optimal, whereas Chi-square was the winner for XGBoost. This suggests that practitioners should tailor feature engineering to the specific model architecture.
- TPE Superiority: XGBoost with TPE tuning achieved significantly higher accuracy and lower variability than Random Search. In finance, this consistency is vital, as a model that fluctuates wildly across data folds is a liability.
- XGBoost vs. LR: Both XGBoost variants (RS and TPE) significantly outperformed Logistic Regression across all metrics.
- Interpretability: By using feature importance scores, the authors identified that the percentage of non-financial charge-off accounts was the most critical predictor, providing a clear "why" behind the model's risk predictions.
Conclusion
The study demonstrates that combining XGBoost with Bayesian TPE optimization and Chi-square feature selection significantly outperforms traditional methods. This framework provides a robust approach for data scientists to modernize risk modeling, achieving high predictive power without sacrificing the stability or interpretability required in financial applications.
Summary updated