Are Foundation Models Useful for Bankruptcy Prediction?

Marcin Kostrzewa, Oleksii Furman, Roman Furman, Sebastian K. Tomczak, Maciej Zieba

Abstract

Predicting corporate bankruptcy is a critical task in financial risk management, characterized by extreme class imbalance and high stakes. This study evaluates whether modern foundation models—including Large Language Models (LLMs) and tabular-specific transformers like TabPFN—can outperform classical gradient boosting machines (GBMs). Through a comprehensive evaluation on a dataset of over one million financial records, we demonstrate that specialized models like XGBoost and CatBoost maintain a significant lead in predictive accuracy, probability calibration, and computational efficiency. Our findings suggest that for structured, imbalanced financial data, the "bigger is better" paradigm of foundation models has yet to surpass domain-optimized classical machine learning.

Introduction

Corporate bankruptcy prediction is a cornerstone of financial stability, yet it remains a "needle in a haystack" problem where bankruptcies typically represent less than 1% of observations. While Large Language Models (LLMs) have achieved remarkable success in natural language tasks, their utility in structured numerical forecasting is less clear. This paper investigates whether general-purpose foundation models can replace the classical machine learning methods that have dominated financial forecasting for decades.

Methodology

We conducted a head-to-head comparison across three model categories:

  1. Classical Baselines: The industry standard, including XGBoost, CatBoost, LightGBM, Multi-Layer Perceptrons (MLP), and Logistic Regression.
  2. Tabular Foundation Models: TabPFN, a transformer-based model designed for tabular data, scaled to large datasets using a "partition-then-predict" strategy.
  3. Large Language Models: Llama-3.3-70B-Instruct, evaluated in both zero-shot and in-context learning (ICL) settings by serializing financial features into natural-language prompts.

Models were evaluated across five prediction horizons ($h=0$ to $h=4$ years ahead) using a massive real-world dataset from the Visegrád Group (Poland, Hungary, Slovakia, and the Czech Republic), comprising over 1 million financial records and 131 indicators.

Key Results: The Dominance of Gradient Boosting

Across all prediction horizons, XGBoost and CatBoost were the clear winners. For immediate risk prediction ($h=0$), XGBoost achieved a near-perfect ROC-AUC of 0.996 and an F1-score of 0.465. In contrast, even with in-context learning, Llama-3.3 struggled significantly, achieving an F1-score of only 0.114.

Performance comparison of models across different prediction horizons (h). Values represent ROC-AUC / F1-score.

As the forecasting horizon extended to four years, performance naturally decayed for all models. However, the gap persisted: XGBoost maintained a ROC-AUC of 0.891, while TabPFN dropped to 0.771. While TabPFN showed some competitiveness in F1-scores compared to simpler models like MLPs, its high computational overhead makes it difficult to justify over GBMs.

Critical Insights

The Calibration Crisis

A primary concern for financial applications is the reliability of predicted probabilities. Our analysis revealed that LLMs are "confident but wrong." Their self-reported probability estimates are poorly calibrated and highly discretized, clustering around fixed values (e.g., 0.1, 0.2, 0.7) rather than providing a continuous, smooth distribution. This behavior makes them unsuitable for risk-sensitive settings where precise probability thresholds are required for decision-making.

Distribution of Llama-3.3 self-reported probabilities, showing high discretization and poor calibration.

Efficiency and Specialization

The computational gap is equally stark. XGBoost can process nearly 3 million samples per second on a standard CPU. In contrast, Llama-3.3 (via API) manages fewer than 4 samples per second. The broad "world-knowledge" of LLMs appears to offer no advantage over the inherent ability of gradient boosting trees to capture non-linear patterns in structured numerical ratios.

Conclusion and Future Work

Currently, classical gradient boosting remains the gold standard for bankruptcy prediction. Foundation models face significant hurdles in calibration and efficiency before they can be considered viable for large-scale financial risk assessment. Future research may explore "reasoning" models or hybrid approaches that combine structured numerical data with unstructured text from annual reports to bridge this gap. For now, practitioners should continue to rely on domain-optimized classical ML.

Summary updated

BibTeX
@inproceedings{kostrzewa2025are,
  author = {Marcin Kostrzewa and Oleksii Furman and Roman Furman and Sebastian K. Tomczak and Maciej Zieba},
  title = {{Are Foundation Models Useful for Bankruptcy Prediction?}},
  booktitle = {GenAI in Finance @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=x1noi32iHp},
}
Figure