ElTab: Elastically Looped Tabular Transformers

Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian KerstingarXiv 2026 · arXiv

TL;DR

ElTab (Elastically Looped Tabular Transformer) is a new tabular foundation model that allows you to trade off accuracy for speed at inference time without retraining. By combining weight-sharing loops with a novel "elastic-depth" training objective, a single ElTab model matches the performance of multiple specialized models while cutting training energy costs by 58%.

What Problem Does This Paper Address?

Tabular foundation models like TabPFN have revolutionized how we handle small-to-medium datasets, providing high-accuracy predictions in a single forward pass without needing dataset-specific tuning. However, they come with a steep "inference tax." Because they use in-context learning, their computational cost grows linearly with the number of layers and quadratically with the number of rows in the dataset.

In resource-constrained or real-time environments, you might need a faster, shallower model. Currently, the only solution is to train and store multiple models of different depths—a process that is both computationally expensive and storage-intensive. While "looped" transformers (which reuse the same weights across multiple iterations) save on parameters, they typically lose massive amounts of accuracy if you stop them before their pre-defined final loop because their intermediate layers were never trained to produce a final answer.

Methods

The authors introduce ElTab, which uses a shared stack of Transformer blocks applied repeatedly. The "secret sauce" is the elastic-depth training objective.

Instead of just training the model to be accurate at the final loop, ElTab is supervised at multiple points:

  1. Dual Forward Passes: During each training step, the model performs a full-depth pass (e.g., 6 loops) and a "shortcut" pass at a randomly sampled shallower depth.
  2. Self-Distillation: The model is trained to minimize standard cross-entropy loss at both the full and shortcut exits. Crucially, it also uses self-distillation, where the full-depth prediction acts as a "teacher," guiding the shallower exit to produce similar results.
  3. Shared Components: The same embedder and decoder are used regardless of how many loops the model performs, ensuring the latent representations remain consistent across depths.
Figure 2: ElTab architecture. ElTab predicts the labels of query rows from a labeled context using three components. An embedder (Eψ) turns the table into token representations. A shared stack of Transformer blocks (Fθ) then refines these representations and can be applied repeatedly, up to L times. Finally, a decoder (gϕ) turns the refined query representations into class probabilities. Because ElTab is trained to predict after every loop, it can stop after any loop M ≤L, letting the deployment choose how much compute to spend. All loops share the same weights and decoder.

Datasets

The models were pretrained on millions of synthetic datasets sampled from a prior, following the standard TabPFN protocol. For evaluation, the authors used 30 classification datasets from the OpenML-CC18 benchmark, filtered for tasks with at most 2,000 observations, 100 features, and 10 classes.

Key Insights and Ideas

  • Graceful Degradation: Unlike standard looped models that "crash" in performance if stopped early, ElTab's accuracy degrades gracefully. This makes it truly "elastic"—you can decide the compute budget at the moment of inference.
  • Latent Alignment: Through Centered Kernel Alignment (CKA) studies, the authors found that elastic training forces the model's internal representations to become similar across different depths. This alignment is what allows the decoder to work effectively whether it's looking at the output of loop 2 or loop 6.
  • Training Efficiency: Training one elastic model is significantly cheaper than training a suite of fixed-depth models. ElTab provides a "buy one, get six free" deal for inference budgets.
ElTab (blue) stays close to the performance of specialized "oracle" models across all budgets, whereas fixed-depth models (gray) fail when run at budgets they weren't trained for.

Key Results

  • Energy Savings: A single ElTab model matches the performance of six different depth-specific models while consuming 58% less GPU energy (25.7 kWh vs 60.6 kWh) during training.
  • Inference Efficiency: ElTab matches the predictive performance of a standard 12-layer TabPFN-style baseline while requiring 25% fewer inference FLOPs.
  • Accuracy: On average across 30 datasets, ElTab matches the "oracle" performance (the best possible performance of a model trained specifically for a given depth) within 0.12 AUC-ROC points.
Figure 1: ElTab matches six budget-specific models, cutting training energy consumption by more than half. (a) Per-dataset AUC-ROC on 30 OpenML datasets at each inference budget. Fixed-depth looped models are budget-specific, because their depth is set at training time, before the inference budget is known. This depth can be chosen at random (gray) or, as an idealized upper bound, by an oracle that knows the budget in advance (orange). A single ElTab model (blue), trained without knowing the budget, matches the oracle at every budget. (b) Estimated GPU energy for full training on one NVIDIA A100. Covering all six budgets with budget-specific models requires training six models (gray segments), which consume 60.6 kWh in total. A single ElTab model covers the same budgets with 25.7 kWh.

Limitations and Future Work

  • Static Stopping: Currently, the user manually selects the number of loops. The authors suggest that future work could involve adaptive computation, where the model learns a "stopping rule" to spend more time on difficult queries and exit early on easy ones.
  • Scaling: The current study focuses on models with a maximum of 6 loops (18 effective blocks). Scaling this to much deeper architectures or larger datasets remains to be explored.
  • Path Sampling: Beyond just looping, the authors propose exploring "computational paths" where different blocks are skipped or repeated dynamically during training to further increase elasticity.

Summary updated

Abstract

Tabular transformer models such as TabPFN achieve strong predictive performance but incur substantial inference costs, making them difficult to deploy in resource-constrained settings. While looped transformers reduce parameter count through weight sharing, they can lose significant predictive power when stopped early. To address this challenge, we introduce the Elastically Looped Tabular Transformer (ElTab), a single model trained to operate across different computational budgets through supervision at randomly sampled early exits. Across 30 OpenML classification datasets, a single ElTab model matches six depth-specific looped models while cutting training cost by more than half, and matches a 12-layer TabPFN-style baseline with 25% fewer inference FLOPs. These results show that elastic-depth training yields a single model that adapts to the deployment budget at hand, offering a favorable trade-off among predictive performance, storage, and computational cost.
BibTeX
@misc{komisarczyk2026eltab,
  author = {Mieszko Komisarczyk and Saurabh Mathur and Maurice Kraus and Sriraam Natarajan and Kristian Kersting},
  title = {{ElTab: Elastically Looped Tabular Transformers}},
  year = {2026},
  eprint = {2610.41819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41819},
}
Figure