Abstract
Predicting corporate bankruptcy is a critical task in financial risk management, characterized by extreme class imbalance and high stakes. This study evaluates whether modern foundation models—including Large Language Models (LLMs) and tabular-specific transformers like TabPFN—can outperform classical gradient boosting machines (GBMs). Through a comprehensive evaluation on a dataset of over one million financial records, we demonstrate that specialized models like XGBoost and CatBoost maintain a significant lead in predictive accuracy, probability calibration, and computational efficiency. Our findings suggest that for structured, imbalanced financial data, the "bigger is better" paradigm of foundation models has yet to surpass domain-optimized classical machine learning.
Introduction
Corporate bankruptcy prediction is a cornerstone of financial stability, yet it remains a "needle in a haystack" problem where bankruptcies typically represent less than 1% of observations. While Large Language Models (LLMs) have achieved remarkable success in natural language tasks, their utility in structured numerical forecasting is less clear. This paper investigates whether general-purpose foundation models can replace the classical machine learning methods that have dominated financial forecasting for decades.
Methodology
We conducted a head-to-head comparison across three model categories:
- Classical Baselines: The industry standard, including XGBoost, CatBoost, LightGBM, Multi-Layer Perceptrons (MLP), and Logistic Regression.
- Tabular Foundation Models: TabPFN, a transformer-based model designed for tabular data, scaled to large datasets using a "partition-then-predict" strategy.
- Large Language Models: Llama-3.3-70B-Instruct, evaluated in both zero-shot and in-context learning (ICL) settings by serializing financial features into natural-language prompts.
Models were evaluated across five prediction horizons ($h=0$ to $h=4$ years ahead) using a massive real-world dataset from the Visegrád Group (Poland, Hungary, Slovakia, and the Czech Republic), comprising over 1 million financial records and 131 indicators.
Key Results: The Dominance of Gradient Boosting
Across all prediction horizons, XGBoost and CatBoost were the clear winners. For immediate risk prediction ($h=0$), XGBoost achieved a near-perfect ROC-AUC of 0.996 and an F1-score of 0.465. In contrast, even with in-context learning, Llama-3.3 struggled significantly, achieving an F1-score of only 0.114.
As the forecasting horizon extended to four years, performance naturally decayed for all models. However, the gap persisted: XGBoost maintained a ROC-AUC of 0.891, while TabPFN dropped to 0.771. While TabPFN showed some competitiveness in F1-scores compared to simpler models like MLPs, its high computational overhead makes it difficult to justify over GBMs.
Critical Insights
The Calibration Crisis
A primary concern for financial applications is the reliability of predicted probabilities. Our analysis revealed that LLMs are "confident but wrong." Their self-reported probability estimates are poorly calibrated and highly discretized, clustering around fixed values (e.g., 0.1, 0.2, 0.7) rather than providing a continuous, smooth distribution. This behavior makes them unsuitable for risk-sensitive settings where precise probability thresholds are required for decision-making.
Efficiency and Specialization
The computational gap is equally stark. XGBoost can process nearly 3 million samples per second on a standard CPU. In contrast, Llama-3.3 (via API) manages fewer than 4 samples per second. The broad "world-knowledge" of LLMs appears to offer no advantage over the inherent ability of gradient boosting trees to capture non-linear patterns in structured numerical ratios.
Conclusion and Future Work
Currently, classical gradient boosting remains the gold standard for bankruptcy prediction. Foundation models face significant hurdles in calibration and efficiency before they can be considered viable for large-scale financial risk assessment. Future research may explore "reasoning" models or hybrid approaches that combine structured numerical data with unstructured text from annual reports to bridge this gap. For now, practitioners should continue to rely on domain-optimized classical ML.
Summary updated