Tabular Foundation Models vs Tuned Gradient Boosting for Telecom Churn: A Profit- and Calibration-Aware Evaluation

Ismail Hossain PolasarXiv 2026 · arXiv

This study re-evaluates the bold claim that Tabular Foundation Models (TFMs) have finally dethroned Gradient Boosted Decision Trees (GBDTs) for tabular tasks like customer churn. By shifting the focus from standard ranking metrics to actual business profit and model calibration, the authors provide a more nuanced—and grounded—view of the current state of tabular AI.

TL;DR

While Tabular Foundation Models like TabPFN-3.5 can outperform hyperparameter-tuned GBDTs in telecom churn prediction, the margin is narrower than previously reported. The study highlights that business profit and probability calibration are just as important as ranking accuracy, and currently, restrictive licensing remains the biggest hurdle for deploying these models in production.

What Problem Does This Paper Address?

Recent benchmarks suggested that TFMs "decisively outperform" tree ensembles for customer churn. However, the authors of this paper noticed two major red flags in those studies:

  1. The Tuning Gap: TFMs (which require no tuning) were compared against GBDTs using library-default hyperparameters.
  2. The Metric Gap: Models were scored on ranking metrics like AUC, whereas real-world retention campaigns are judged by money—specifically, how much profit a model generates by correctly identifying which customers to target with incentives.

This paper addresses whether TFMs still hold their lead when GBDTs are properly tuned and when the "yardstick" is changed to Expected Maximum Profit (EMP).

Methods

The researchers conducted a rigorous head-to-head battle:

  • The Contenders: Four TFMs (TabPFN-2.5, 3.5, 3.5-Fast, and TabICL v2) against Optuna-tuned versions of LightGBM, XGBoost, and CatBoost.
  • The Protocol: 10 seeded stratified splits (60/20/20) for training, calibration, and testing.
  • The Metrics: Beyond standard AUC, they used EMP (which models the financial trade-off of a retention campaign) and e-Profits (which uses per-customer values).
  • Calibration Analysis: They tested how well the models' predicted probabilities matched reality using Expected Calibration Error (ECE) and Brier scores, which is critical for setting decision thresholds.

Datasets

The study used five main datasets:

  • Real-world: Cell2Cell (US wireless), Iranian (telecom operator), and Orange (KDD Cup 2009).
  • Fictional/Teaching: IBM Telco and Maven Analytics.
  • Replication: An E-Commerce dataset was used specifically to check the results of previous benchmarks.

Key Insights and Ideas

  • Tuning Matters: A significant portion of the "TFM advantage" reported in earlier work was actually just a "tuning gap." Tuning the GBDT baselines removed about a third of the reported lead for TFMs on the E-Commerce dataset.
  • Ranking $\neq$ Profit: Models with similar AUC scores can result in different profit outcomes. Profit metrics are better at breaking ties and identifying which model actually helps a business's bottom line.
  • The Calibration Trap: For a model to be useful in a "threshold policy" (e.g., "offer a discount to everyone with a >20% churn risk"), its probabilities must be accurate. Some models, like TabICL v2, significantly underpredicted churn on certain datasets, which would lead to lost profit in a real campaign unless recalibrated.
  • Licensing is the Real Barrier: The most performant model, TabPFN-3.5, is released under a non-commercial license. This means that even if it's technically superior, it cannot currently be used for "internal commercial decision-making" without a separate agreement.

Key Results

  • TabPFN-3.5 is the strongest TFM: It earned significantly more profit (EMP) than the best tuned GBDT on two of the three real-operator datasets: Cell2Cell (+1.9%) and Iranian (+7.6%).
  • Orange Dataset: TabPFN-3.5 was ahead in all 10 splits, showing a 14.3% EMP gain, though this result was not statistically significant after strict multiple-comparison correction.
  • Calibration Consistency: TabPFN-3.5 and its "Fast" variant were consistently well-calibrated out of the box. In contrast, TabICL v2 required isotonic recalibration to avoid losing significant campaign profit on the Orange dataset.
Figure 1: EMP difference between each TFM and the reference GBDT, per customer, with corrected-test 95% intervals (unadjusted) over 10 splits. Filled markers: significant after Holm correction across the 20 comparisons; an open marker whose interval excludes zero is not significant (Orange, TabPFN-3.5).
Figure 2: Reliability on Orange, test customers pooled over 10 splits, ten quantile bins. Raw TabICL v2 sits above the diagonal (churn underpredicted); isotonic recalibration on the cali- bration split removes the gap.

Limitations and Future Work

  • Generalization: The evidence is based on a small number of telecom datasets. It’s unclear if these gains hold in other industries or for massive datasets that exceed the TFM's context window.
  • Feature Engineering: The study used a basic preprocessing pipeline. Advanced feature engineering, which is standard in Kaggle-style GBDT pipelines, might further narrow the gap.
  • Tuning Objective: The GBDTs were tuned for AUC, not profit. Future work could explore if tuning GBDTs directly for EMP would close the remaining gap.
  • Production Constraints: While scoring cost is manageable with caching, the legal restrictions on TFM weights remain the primary obstacle for practitioners.

Summary updated

Abstract

Tabular foundation models (TFMs) predict from a labelled context in a single forward pass. A recent benchmark reported that they decisively outperform tree ensembles for customer churn, but it compared them with untuned gradient-boosted trees (GBDTs) and scored only ranking metrics, while a retention campaign is judged in money. We re-evaluate TabPFN-2.5, TabPFN-3.5, TabPFN-3.5-Fast and TabICL v2 against Optuna-tuned GBDTs on five public telecom churn datasets over 10 seeded splits. We separate three questions: whether a model ranks churners well, whether that ranking earns more expected maximum profit (EMP), and whether its probabilities can drive a threshold policy. With a split-aware test and Holm correction over 20 pre-declared comparisons, TabPFN-3.5 earns significantly more EMP than the best tuned GBDT on Cell2Cell (1.9%) and Iranian (7.6%), and these two gains also hold against each of the three tuned GBDTs separately. On Orange it is ahead in all 10 splits (14.3%), but the gain is not significant after correction (Holm p = 0.155). The splits resample the same datasets, so these results do not stand for independent operators. TabICL v2, the earlier benchmark's leader, beats the best tuned GBDT only on Iranian, not all three of them, and underpredicts Orange churn by 2.1 points on a 7.3% base rate. On the datasets and splits where we tested them, a substantially larger tuning budget and a duplicate-aware split leave the main conclusions unchanged. Retuning the baseline cuts the earlier headline gap from 6.1 to 4.0 PR-AUC points without closing it. The licence of the TabPFN-3.5 weights we used (License v1.0), not scoring cost, is what stands between its gains and production use.
BibTeX
@misc{polas2026tabular,
  author = {Ismail Hossain Polas},
  title = {{Tabular Foundation Models vs Tuned Gradient Boosting for Telecom Churn: A Profit- and Calibration-Aware Evaluation}},
  year = {2026},
  eprint = {2611.12942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12942},
}
Figure