TL;DR
ElTab (Elastically Looped Tabular Transformer) is a new tabular foundation model that allows you to trade off accuracy for speed at inference time without retraining. By combining weight-sharing loops with a novel "elastic-depth" training objective, a single ElTab model matches the performance of multiple specialized models while cutting training energy costs by 58%.
What Problem Does This Paper Address?
Tabular foundation models like TabPFN have revolutionized how we handle small-to-medium datasets, providing high-accuracy predictions in a single forward pass without needing dataset-specific tuning. However, they come with a steep "inference tax." Because they use in-context learning, their computational cost grows linearly with the number of layers and quadratically with the number of rows in the dataset.
In resource-constrained or real-time environments, you might need a faster, shallower model. Currently, the only solution is to train and store multiple models of different depths—a process that is both computationally expensive and storage-intensive. While "looped" transformers (which reuse the same weights across multiple iterations) save on parameters, they typically lose massive amounts of accuracy if you stop them before their pre-defined final loop because their intermediate layers were never trained to produce a final answer.
Methods
The authors introduce ElTab, which uses a shared stack of Transformer blocks applied repeatedly. The "secret sauce" is the elastic-depth training objective.
Instead of just training the model to be accurate at the final loop, ElTab is supervised at multiple points:
- Dual Forward Passes: During each training step, the model performs a full-depth pass (e.g., 6 loops) and a "shortcut" pass at a randomly sampled shallower depth.
- Self-Distillation: The model is trained to minimize standard cross-entropy loss at both the full and shortcut exits. Crucially, it also uses self-distillation, where the full-depth prediction acts as a "teacher," guiding the shallower exit to produce similar results.
- Shared Components: The same embedder and decoder are used regardless of how many loops the model performs, ensuring the latent representations remain consistent across depths.
Datasets
The models were pretrained on millions of synthetic datasets sampled from a prior, following the standard TabPFN protocol. For evaluation, the authors used 30 classification datasets from the OpenML-CC18 benchmark, filtered for tasks with at most 2,000 observations, 100 features, and 10 classes.
Key Insights and Ideas
- Graceful Degradation: Unlike standard looped models that "crash" in performance if stopped early, ElTab's accuracy degrades gracefully. This makes it truly "elastic"—you can decide the compute budget at the moment of inference.
- Latent Alignment: Through Centered Kernel Alignment (CKA) studies, the authors found that elastic training forces the model's internal representations to become similar across different depths. This alignment is what allows the decoder to work effectively whether it's looking at the output of loop 2 or loop 6.
- Training Efficiency: Training one elastic model is significantly cheaper than training a suite of fixed-depth models. ElTab provides a "buy one, get six free" deal for inference budgets.
Key Results
- Energy Savings: A single ElTab model matches the performance of six different depth-specific models while consuming 58% less GPU energy (25.7 kWh vs 60.6 kWh) during training.
- Inference Efficiency: ElTab matches the predictive performance of a standard 12-layer TabPFN-style baseline while requiring 25% fewer inference FLOPs.
- Accuracy: On average across 30 datasets, ElTab matches the "oracle" performance (the best possible performance of a model trained specifically for a given depth) within 0.12 AUC-ROC points.
Limitations and Future Work
- Static Stopping: Currently, the user manually selects the number of loops. The authors suggest that future work could involve adaptive computation, where the model learns a "stopping rule" to spend more time on difficult queries and exit early on easy ones.
- Scaling: The current study focuses on models with a maximum of 6 loops (18 effective blocks). Scaling this to much deeper architectures or larger datasets remains to be explored.
- Path Sampling: Beyond just looping, the authors propose exploring "computational paths" where different blocks are skipped or repeated dynamically during training to further increase elasticity.
Summary updated