Deep learning has revolutionized fields with homogeneous data structures. In computer vision, every pixel shares the same numerical format and geometric meaning. In natural language processing, text is cleanly broken down into uniform, discrete tokens from a single vocabulary.
Tabular data, however, is fundamentally chaotic. A single row in a table might contain a continuous float representing income, a binary flag for employment status, a high-cardinality categorical string for a zip code, and a sequential timestamp. Historically, neural networks handled this heterogeneity by simply concatenating raw continuous scalars with one-hot encoded categorical vectors.
This naive approach introduces a severe geometric mismatch. A neural network layer or an attention mechanism struggles to compute meaningful interactions when its input vector is a disjointed mix of a single large scalar and a sparse, 100-dimensional binary array. The gradients become unstable, and the model fails to capture complex cross-feature correlations.
To bridge this gap, a rapidly growing body of research focuses on feature tokenization: transforming every heterogeneous attribute into a uniform, high-dimensional vector space before it ever reaches the core model backbone.
Homogenizing the Input Space
The core intuition behind feature tokenization is to translate all data types into a "common currency." Instead of treating a tabular row as a single heterogeneous vector, tokenization treats it as a collection or sequence of uniform tokens, much like words in a sentence.
Raw Tabular Row:
[ Age: 34 (Numerical) | Country: "US" (Categorical) | Income: 75000 (Numerical) ]
│
▼ Feature Tokenizer
Unified Token Sequence:
[ Token_Age (d-dim) | Token_Country (d-dim) | Token_Income (d-dim) ]
For categorical features, this is straightforward: each category maps to a learnable $d$-dimensional embedding vector via a standard lookup table. For numerical features, the transformation requires more deliberate design to preserve numerical relationships while matching the target dimensionality:
- Linear Projection: Multiplying the scalar value by a column-specific weight vector to scale it into $d$ dimensions.
- Discretization: Binning continuous values into discrete intervals and treating them as categorical tokens, an approach utilized by Madaan et al. in their work on structured counterfactual generation (Navigating the Structured What-If Spaces: Counterfactual Generation via Structured Diffusion).
- Hybrid Encodings: Combining piece-wise linear encodings with learnable embeddings to capture non-linear continuous patterns, as demonstrated by Xu et al. in BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model.
By establishing this uniform format, Gorishniy et al. introduced the FT-Transformer in Revisiting Deep Learning Models for Tabular Data, proving that a dedicated feature tokenization layer allows deep learning models to directly challenge the dominance of Gradient Boosted Decision Trees (GBDTs).
Unlocking Uniform Attention
Why does this uniform representation perform so much better? The primary beneficiary is the self-attention mechanism.
When features are represented as uniform $d$-dimensional tokens, the model can compute pairwise dot-product attention between any two features natively. In SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, Somepalli et al. demonstrated that projecting continuous features into this high-dimensional space allows attention layers to capture intricate, non-linear feature interactions that simple scalar concatenation completely obfuscates.
This architectural flexibility has allowed researchers to deploy transformers across a wide array of tabular tasks. For instance, Panagiotou et al. leverage feature tokenization in TABCF: Counterfactual Explanations for Tabular Data Using a Transformer-Based VAE to apply uniform attention across mixed-type inputs for generating counterfactual explanations. Similarly, Wu et al. use a specialized tokenization layer in APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuning to handle highly irregular regression targets, while Shimizu et al. rely on it to enable sparse attention pre-training in Sparse attention is all you need for pre-training on tabular data.
Enabling Continuous Generative Modeling
The benefits of tokenization extend far beyond classification and regression; it is a critical enabler for generative tabular models, such as diffusion and flow-matching architectures.
Generative diffusion models naturally operate on continuous manifolds and struggle with the discrete jumps inherent to categorical data. By mapping categorical features into continuous embedding spaces and standardizing continuous features alongside them, the entire tabular row is compressed into a single, dense continuous vector.
In TabUnite: Efficient Encoding Schemes for Flow and Diffusion Tabular Generative Models, Si et al. show that processing heterogeneous features within a single, unified continuous space allows a generative model to effectively learn complex interdependencies between all features simultaneously, rather than managing separate, disjoint generation processes.
To optimize this further, the authors introduced a simplified continuous representation called CatConverter in TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation, which maps categorical features onto a 2D unit circle. This avoids the gradient instabilities of one-hot encodings and allows diffusion backbones to operate efficiently. This continuous unification strategy is also vital for missing data imputation, as shown by Zhang et al. in DiffPuter: Empowering Diffusion Models for Missing Data Imputation.
Scaling to Real-World Heterogeneity
Beyond standard benchmarks, feature tokenization serves as a foundational bridge for multimodal and highly specialized domain applications where tabular data intersects with other sequences.
Online Advertising and Sequence Modeling
In complex dynamic environments like e-commerce bidding, models must ingest mixed categorical attributes, continuous metrics, and historical time-series. Wang et al. developed a unified embedding layer in Bid2X: Revealing Dynamics of Bidding Environment in Online Advertising from A Foundation Model Lens to transform these diverse inputs into a standardized format, allowing a single transformer foundation model to generalize across distinct advertising scenarios without requiring specialized sub-networks.
Clinical and Network Security Metadata
In healthcare, Mbaye et al. created M-BEHRT (Multimodal BEHRT: transformers for multimodal electronic health records to predict breast cancer prognosis), mapping binarized lab results and administrative codes into a unified token space to seamlessly fuse structured clinical tables with unstructured medical text. In cybersecurity, Watabe et al. adapted BERT for network intrusion detection (Flow-based network intrusion detection based on BERT masked language model) by discretizing and embedding heterogeneous flow features like ports and protocols into a unified vector sequence.
Industrial Processes and Remote Sensing
Whether standardizing diverse equipment IDs and recipes across thousands of semiconductor manufacturing steps (Miyaguchi et al., Cross-Process Defect Attribution using Potential Loss Analysis) or fusing passive microwave data with optical remote sensing inputs (Zhao et al., A Snow Depth Downscaling Algorithm Based on Deep Learning Fusion of Enhanced Passive Microwave and Cloud-Free Optical Remote Sensing Data in China), tokenization consistently acts as the primary standardizing mechanism.
By mapping every column to a fixed-size embedding vector, feature tokenization completely homogenizes tabular data. As demonstrated by Kang et al. in On Learning Representations for Tabular Data Distillation, this homogenization is so effective that it even enables advanced downstream tasks like dataset distillation, allowing standard neural architectures to process mixed tabular data with the same mathematical elegance typically reserved for text and images.
Key Foundational Work
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (2021) — contributed 2 concept elements to this direction.
Revisiting Deep Learning Models for Tabular Data (2021) — contributed 1 concept elements to this direction.
Flow-based Network Intrusion Detection Based on BERT Masked Language Model (2022) — contributed 1 concept elements to this direction.