This study re-evaluates the bold claim that Tabular Foundation Models (TFMs) have finally dethroned Gradient Boosted Decision Trees (GBDTs) for tabular tasks like customer churn. By shifting the focus from standard ranking metrics to actual business profit and model calibration, the authors provide a more nuanced—and grounded—view of the current state of tabular AI.
TL;DR
While Tabular Foundation Models like TabPFN-3.5 can outperform hyperparameter-tuned GBDTs in telecom churn prediction, the margin is narrower than previously reported. The study highlights that business profit and probability calibration are just as important as ranking accuracy, and currently, restrictive licensing remains the biggest hurdle for deploying these models in production.
What Problem Does This Paper Address?
Recent benchmarks suggested that TFMs "decisively outperform" tree ensembles for customer churn. However, the authors of this paper noticed two major red flags in those studies:
- The Tuning Gap: TFMs (which require no tuning) were compared against GBDTs using library-default hyperparameters.
- The Metric Gap: Models were scored on ranking metrics like AUC, whereas real-world retention campaigns are judged by money—specifically, how much profit a model generates by correctly identifying which customers to target with incentives.
This paper addresses whether TFMs still hold their lead when GBDTs are properly tuned and when the "yardstick" is changed to Expected Maximum Profit (EMP).
Methods
The researchers conducted a rigorous head-to-head battle:
- The Contenders: Four TFMs (TabPFN-2.5, 3.5, 3.5-Fast, and TabICL v2) against Optuna-tuned versions of LightGBM, XGBoost, and CatBoost.
- The Protocol: 10 seeded stratified splits (60/20/20) for training, calibration, and testing.
- The Metrics: Beyond standard AUC, they used EMP (which models the financial trade-off of a retention campaign) and e-Profits (which uses per-customer values).
- Calibration Analysis: They tested how well the models' predicted probabilities matched reality using Expected Calibration Error (ECE) and Brier scores, which is critical for setting decision thresholds.
Datasets
The study used five main datasets:
- Real-world: Cell2Cell (US wireless), Iranian (telecom operator), and Orange (KDD Cup 2009).
- Fictional/Teaching: IBM Telco and Maven Analytics.
- Replication: An E-Commerce dataset was used specifically to check the results of previous benchmarks.
Key Insights and Ideas
- Tuning Matters: A significant portion of the "TFM advantage" reported in earlier work was actually just a "tuning gap." Tuning the GBDT baselines removed about a third of the reported lead for TFMs on the E-Commerce dataset.
- Ranking $\neq$ Profit: Models with similar AUC scores can result in different profit outcomes. Profit metrics are better at breaking ties and identifying which model actually helps a business's bottom line.
- The Calibration Trap: For a model to be useful in a "threshold policy" (e.g., "offer a discount to everyone with a >20% churn risk"), its probabilities must be accurate. Some models, like TabICL v2, significantly underpredicted churn on certain datasets, which would lead to lost profit in a real campaign unless recalibrated.
- Licensing is the Real Barrier: The most performant model, TabPFN-3.5, is released under a non-commercial license. This means that even if it's technically superior, it cannot currently be used for "internal commercial decision-making" without a separate agreement.
Key Results
- TabPFN-3.5 is the strongest TFM: It earned significantly more profit (EMP) than the best tuned GBDT on two of the three real-operator datasets: Cell2Cell (+1.9%) and Iranian (+7.6%).
- Orange Dataset: TabPFN-3.5 was ahead in all 10 splits, showing a 14.3% EMP gain, though this result was not statistically significant after strict multiple-comparison correction.
- Calibration Consistency: TabPFN-3.5 and its "Fast" variant were consistently well-calibrated out of the box. In contrast, TabICL v2 required isotonic recalibration to avoid losing significant campaign profit on the Orange dataset.
Limitations and Future Work
- Generalization: The evidence is based on a small number of telecom datasets. It’s unclear if these gains hold in other industries or for massive datasets that exceed the TFM's context window.
- Feature Engineering: The study used a basic preprocessing pipeline. Advanced feature engineering, which is standard in Kaggle-style GBDT pipelines, might further narrow the gap.
- Tuning Objective: The GBDTs were tuned for AUC, not profit. Future work could explore if tuning GBDTs directly for EMP would close the remaining gap.
- Production Constraints: While scoring cost is manageable with caching, the legal restrictions on TFM weights remain the primary obstacle for practitioners.
Summary updated