Keeping the Different Columns: Redundancy-Aware Progressive Token Pruning for Tabular Transformers

Tabular transformers turn every column of a table into its own vector, or "token," so that attention (the mechanism that lets each token weigh every other token) can model interactions between features. That cost grows with the square of the number of columns. On wide tables with thousands of columns, such as gene panels or engineered click logs, the model becomes too slow, and it is also prone to overfitting because many columns are near-copies or noisy transformations of one another. Vision and language models already handle a similar problem by progressively dropping unimportant tokens between layers. The usual rule is to keep the tokens that the summary token attends to most. On redundant tables, that rule keeps the wrong ones. Five near-duplicate columns all receive high attention and all survive, while a unique, weakly attended column that carries complementary signal gets discarded.

We propose a pruning rule that fixes this, and we treat one question as central: is it worth deciding per row which columns to keep, rather than choosing a fixed column subset once for the whole dataset? At each of a few layers, the model keeps tokens that are both relevant to the prediction and different from the tokens already kept. It measures "different" in the model's own learned token space rather than on raw columns. Instead of deleting the discarded tokens, it folds each one into its most similar survivor. The idea is judged against three things at an equal token budget: dropping by attention alone, choosing a static feature subset up front, and running the dense model.

Core Claim

Inside a feature-tokenized transformer, redundancy between columns is best removed during the forward pass by a relevance-times-novelty selection over the current layer's token vectors. This should beat attention-only pruning on tables with collinear or derived columns. It should beat a static, globally selected column subset of equal size specifically when a column's informativeness varies from row to row, for example under row-specific corruption, missingness, or features that matter only for a subgroup. When redundancy is purely a column-level property of the dataset, we expect a good static filter to match our method. We treat that as a prediction to confirm, not an outcome to argue away.

The selection score at layer $\ell$ is a greedy maximum of a determinant-based objective. With $r_i$ the attention that the classification token pays to feature token $i$ and $S_{ij}$ the cosine similarity between tokens $i$ and $j$ at that layer, we select the kept set $K$ of fixed size $k_\ell$ to approximately maximize

$$\log\det\big(\mathrm{diag}(r)_K\, S_{KK}\, \mathrm{diag}(r)_K\big),$$

The determinant collapses toward zero when two kept tokens point in the same direction, so near-duplicates penalize each other. Relevance enters as a per-token scale. Greedy incremental selection costs roughly $O(N k_\ell^2)$ per layer rather than the cubic cost of exact inference. The fixed $k_\ell$ per layer keeps batch tensors rectangular, which avoids the padding overhead that content-adaptive budgets incur.

Relation to Prior Work

Three kinds of prior work motivate the proposal.

Revisiting Deep Learning Models for Tabular Data introduced the FT-Transformer tokenizer, which gives each column a linear projection or a lookup table and adds a classification token. It showed that the classification token's attention tracks feature importance, and it flagged quadratic column scaling as the main limitation. It never used that attention to drop tokens.

A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning showed that FT-Transformer degrades specifically under corrupted and second-order (product) redundant features. It proposed Deep Lasso, a group-lasso penalty on input gradients that selects columns before training. That makes it a global, input-level decision, and it is our main static competitor.

Deep Tabular Representation Corrector found that higher singular value entropy (a measure of how spread out the learned representations are) goes with worse accuracy in tabular models. It corrects this after training, on a frozen backbone, and saves no computation. We borrow its entropy measurement as a diagnostic.

Three lines of work supply the method.

From vision, DynamicViT and Adaptive Token Sampling give the multi-stage schedule and the classification-token relevance score. Token Fusion gives the idea of merging dropped tokens into survivors rather than deleting them. We use merging here to soften the irreversibility of early drops, which Adaptive Token Sampling itself identifies as a weakness.

CDPruner supplies the determinant-based selection of relevant but diverse tokens, and it diagnoses the attention-clustering failure we target. It works once, on image patches, with a text prompt as the condition. We apply the selection progressively, over heterogeneous column tokens, with the classification token as the condition. The underlying repulsion prior comes from Priors for Diversity in Generative Latent Variable Models.

Two papers constrain the design.

Which Tokens to Use? found plain attention Top-K to be a very strong baseline in vision. Our claim must therefore rest on redundant data, where CDPruner's diagnosis applies, and not on generic tables. BiSHop already zeroes out attention to uninformative features through sparse Hopfield layers. It keeps all tokens in the tensor, so it gains robustness but not speed. That separates two questions: does the robustness come from sparsifying attention, or from removing tokens?

Two further papers target wide tables. DynaTab reorders and masks features but keeps them all, and it reports running out of memory at scale. FTT+ uses a static attention mask that connects only feature tokens and the classification token. Neither changes the sequence length.

Research Program

We build the pruner on top of FT-Transformer. Pruning happens after roughly one third and two thirds of the depth, with keep ratios chosen per dataset from a small grid. The selection itself is training-free given the current attention and token vectors. Because the kept-set size is fixed, the model is trained end to end with pruning active, using a straight-through mask. Each dropped token is added, with a weight proportional to its similarity, to its nearest kept token.

The analysis has three parts. First, we test whether novelty measured in token space catches nonlinear redundancy, such as product features and noisy copies, that correlation on raw columns misses. Second, we track how much per-row variation there is in the kept sets. If nearly every row keeps the same columns, the per-row mechanism is pointless and a static schedule should replace it. Third, we measure singular value entropy of the classification-token representation with and without pruning, to see whether removing tokens during the forward pass reproduces the gain that the post-hoc corrector gets.

Baselines

The naive reference is the dense FT-Transformer with all tokens at every layer. Its reported standing is an average rank of 1.8 across 11 datasets, against 3.3 for ResNet (Revisiting Deep Learning Models for Tabular Data). We also include random pruning at the same budget, to show that the selection rule matters at all.

The strongest pruning competitor is classification-token attention Top-K with the same schedule, following DynamicViT and Adaptive Token Sampling. We run it with and without Adaptive Token Sampling's value-norm weighting. It is the strongest competitor because Which Tokens to Use? found it matches learned pruners. No reported result was found on Lacuna for Top-K pruning in any tabular transformer, so we establish the numbers ourselves.

The strongest static competitor is a Deep Lasso column subset of the same size as our final-layer budget, fed to a dense FT-Transformer. The Deep Lasso benchmark reports no aggregate table in the file ("no reported result found on Lacuna" for a headline number), so we rerun it on the same splits.

For architecture context we report two further models. BiSHop has a reported mean rank of 2.79 against FT-Transformer's 3.58 on a 19-dataset suite (BiSHop). XGBoost is tuned with the same number of hyperparameter trials.

Two ablations isolate the ingredients. One uses diversity selection without merging. The other uses merging with attention-only selection. Every method gets the same splits, the same hyperparameter-search trial count, and the same hardware. Speed is reported as measured wall-clock throughput and peak memory, not only FLOPs.

Beating the competitors has a specific meaning. On the redundancy-injected and row-corrupted settings, our method must exceed attention Top-K and the static subset at equal budget, with the gap larger than seed variance across at least five seeds. On clean tables it must stay within seed noise of the dense model.

Why not just…?

Why not just select columns once with Deep Lasso and run a smaller dense model? This is the real threat. If redundancy is a fixed property of the columns, a global subset gives the same speedup with no runtime selection. We expect a tie in that case. Our advantage should appear only when informativeness varies by row. We test this directly by corrupting a random subset of rows per column, a per-row variant of the Deep Lasso benchmark's corrupted-feature protocol. If the static subset still ties there, we drop the per-row claim and recommend a learned static per-layer schedule.

Why not just use attention Top-K, which already works in vision? CDPruner shows that attention-only selection keeps clusters of redundant tokens. The Deep Lasso benchmark shows FT-Transformer's attention is misled by duplicated and product features. The injected-redundancy experiment measures how many kept tokens are copies of one source column under each rule, which settles this directly.

Why not just sparsify attention (BiSHop) or correct representations after training (TRC)? Both may match our accuracy. Neither reduces sequence length, so neither addresses the memory limits DynaTab reports on wide tables. We compare accuracy at matched quality and report memory, and we test whether our method's entropy reduction accounts for any accuracy gain.

Proposed Experiments

The controlled experiment uses the 12-dataset suite from A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning. We inject random, corrupted-copy, and second-order features at increasing fractions. We then add a new row-dependent corruption variant. Metrics are accuracy or RMSE, the fraction of kept tokens traceable to one source column, and per-row kept-set variability.

For generality without redundancy, we use the 11-dataset suite of Revisiting Deep Learning Models for Tabular Data. There we expect only parity.

For speed, we use the wide HDLSS and HDHSS datasets of DynaTab, where DynaTab's reported average ranks are 2.62 and 2.50. We also use the gene-scale biomedical tables of PLATO, and we plot accuracy against throughput and peak memory as the budget varies. On these datasets, a good result means running where the dense model runs out of memory and matching the static subset's accuracy.

The decisive check is the row-dependent corruption result against the equal-budget Deep Lasso subset. It determines whether per-row, in-model pruning of redundant columns is a real capability or just an expensive way to do feature selection.