Attention Quantization for Tabular Foundation Models

Jonas M. Kübler, Benjamin Jäger, Klemens Flöge, Noah Hollmann, Frank HutterarXiv 2026 · arXiv

TL;DR

This paper introduces an FP8 attention quantization strategy specifically designed for tabular foundation models, delivering up to a 1.7x end-to-end inference speedup. The key breakthrough is the realization that, unlike large language models (LLMs), tabular models require coordinated quantization scales between training and test data to prevent severe accuracy degradation.

What Problem Does This Paper Address?

Tabular foundation models (such as TabPFN and TabICL) are transforming machine learning on structured data by enabling in-context learning directly from data rows. However, their computational cost scales quadratically $O(N^2)$ with the number of training rows ($N$).

While the AI community has heavily optimized LLMs using weight and KV-cache quantization, these techniques offer limited benefits for tabular foundation models because the models themselves are small in parameter size. Instead, the primary bottleneck is the attention computation across large datasets. Standard quantization techniques fail here: if you quantize the attention matrix naively, the model's predictive accuracy collapses. This paper tackles how to successfully apply low-precision FP8 arithmetic to tabular attention without sacrificing performance.

Methods

The authors developed a custom Triton kernel that quantizes queries ($Q$), keys ($K$), and values ($V$) to the 8-bit floating-point format (e4m3fn).

[Q, K, V in BF16/FP16] -> [Per-head Absmax Quantization] -> [FP8 MMA Matrix Mult] -> [Shifted Softmax] -> [Output]

Its core components include:

  • Per-Head Dynamic Scaling: Quantization scales are computed dynamically for each attention head using an absolute maximum (absmax) operation.
  • Subnormal Avoidance in Softmax: Because softmax outputs lie in $(0, 1]$, converting them to FP8 normally wastes exponent range and forces small probabilities into subnormal numbers. The authors circumvent this by adding a global constant to the exponent before exponentiation, lifting the row maximum to 256 (a technique adapted from FlashAttention-3).
  • Coordinated Scale Sharing: In tabular in-context learning, the model processes two distinct attention passes: train-to-train (self-attention among training rows) and test-to-train (cross-attention from test rows to training rows). The authors' crucial innovation is reusing the quantization scale computed on the training queries for the test queries. This ensures that the quantization error remains symmetric across both passes and keeps test predictions batch-invariant.
Figure 2: Quantization approach illustrated for a single head. We quantize Q, K, V with per-head scaled absmax quantization, the Q and K scales are then applied in the kernel together with the softmax scale, whereas the V scale is applied in the epilogue. The P matrix is never materialized in global memory and is quantized with a fixed scale inside the kernel.

Datasets

The approach was rigorously evaluated using three random preprocessing seeds across two major tabular benchmarks:

  • TabArena: A benchmark suite consisting of 51 diverse tabular datasets.
  • BeyondArena: A broader benchmark containing 142 datasets, including massive, large-scale datasets with up to 1 million training rows.

Key Insights and Ideas

  • The "Aha!" Moment—Error Alignment: Quantizing only the train-to-train attention or only the test-to-train attention causes a massive drop in model performance (losing up to 31 Elo points). However, when both are quantized using a shared query scale, the accuracy drop practically vanishes.
  • Symmetric Noise Tolerance: Through controlled Gaussian noise experiments, the authors proved that tabular foundation models are remarkably resilient to precision errors, but only if the keys ($K$) and values ($V$) are perturbed identically across both self- and cross-attention blocks.
  • Smart Gating: FP8 quantization introduces a small fixed overhead for tiny datasets. To ensure a pure win-win scenario, the authors implement a gate that only activates the FP8 kernel when a dataset contains more than 8,192 training rows.
Figure 3: TabArena performance across three random seeds when Gaussian noise is added to Q, K, V (left) and E2E runtime scaling.

Key Results

  • Substantial Speedups: Tested on an NVIDIA RTX Pro 6000 (Blackwell Server Edition), the FP8 Triton kernel achieved up to a 1.72x speedup for the attention operation alone and up to a 1.67x end-to-end speedup (fit + predict) on large datasets (512k rows).
  • Negligible Accuracy Loss: On TabArena, the gated FP8 model maintained its performance within seed noise, showing a negligible drop of just 0.1 to 0.5 Elo points (where typical model variations span more than 10 points).
  • Macro Efficiency: For the entire BeyondArena evaluation suite, the total wall-clock execution time plummeted from 253.8 minutes down to 183.7 minutes—a massive practical win for benchmarking and deployment.

Limitations and Future Work

  • Hardware Feature Utilization: The current kernel relies on a FlashAttention-2 design and does not yet exploit advanced hardware-specific features like the Tensor Memory Accelerator (TMA) found in NVIDIA Hopper or Blackwell architectures.
  • The Softmax Bottleneck: On ultra-powerful GPUs like the NVIDIA B200, raw matrix multiplication is scaled so drastically that the non-matrix operations (like the softmax exponentiation) become the new primary bottleneck. Simply speeding up matrix multiplication with FP8 will yield diminishing returns on these chips unless the softmax calculation itself is further optimized.

Summary updated

Abstract

With the recent rise and adoption of tabular foundation models, optimizing their inference performance becomes an emerging field for efficiency research. While the models are architecturally similar to transformer-based large language models (LLMs), the size and serving patterns differ significantly. We show that the focus should be on the attention calculation and less on weight or KV cache quantization, which are more popular in LLMs. We develop a quantization strategy for queries, keys, and values to FP8 and use explicit FP8 matrix multiplication instructions to speed up the attention calculation. We find that it is crucial to align the quantization error in the test rows with the quantization error in the training rows, as otherwise the accuracy drops drastically. Our Triton kernel achieves a speedup up to 1.7x over regular 16-bit kernels, and we show that on TabPFN-v3 and TabICLv2 there is no relevant accuracy loss across TabArena and BeyondArena.
BibTeX
@misc{kubler2026attention,
  author = {Jonas M. Kübler and Benjamin Jäger and Klemens Flöge and Noah Hollmann and Frank Hutter},
  title = {{Attention Quantization for Tabular Foundation Models}},
  year = {2026},
  eprint = {2609.13031},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.13031},
}
Figure