In computer vision, deep learning models thrive on natural geometric invariants. If you rotate, crop, or adjust the brightness of an image of a cat, it remains an image of a cat. Similarly, in natural language processing, sentences possess a sequential structure where words can be masked and inferred from context.
Tabular data enjoys no such luxury. A typical spreadsheet row contains a heterogeneous mix of continuous and categorical variables—such as age, income, blood pressure, and zip code—with no inherent spatial or temporal relationships (Guansong Pang et al.). The columns can be arbitrarily reordered without changing the underlying meaning, yet the logical relationships between individual features are incredibly brittle. Modifying a single value without a structured understanding of the data can instantly destroy inter-variable correlations or invalidate the ground-truth label (Adrián Bazaga et al.).
Because tabular data lacks these intuitive geometric symmetries, standard data augmentation heuristics cannot be easily applied (Inwon Kang et al.; Andrei Margeloiu et al.). This structural dependency gap creates a major bottleneck for self-supervised learning (SSL), contrastive representation learning, and semi-supervised learning in tabular domains.
The Failure of Naive Augmentation
Self-supervised frameworks, particularly contrastive learning, rely on creating multiple "views" of the same data point. The network learns robust representations by pulling different views of the same instance together in the embedding space while pushing views of different instances apart.
When researchers attempt to port these techniques directly to tabular data using standard vision-centric perturbations (like adding Gaussian noise or masking random values), performance often degrades. As shown by Minqi Jiang et al. in anomaly detection, conventional data augmentations frequently fail to capture the structural characteristics of tabular datasets and can lead to model degradation.
Without a principled methodology for generating informative data views, models struggle to converge without memorizing noise (Witold Wydmański et al.; Ravid Shwartz-Ziv et al.). This issue is amplified in real-world settings such as class-imbalanced tasks (Ravid Shwartz-Ziv et al.), test-time adaptation under distribution shifts (Zhi Zhou et al.), or open-environment semi-supervised learning (Lan-Zhe Guo et al.).
To overcome this, recent research focuses on two primary paradigms: designing specialized, tabular-safe feature corruption mechanisms, or escaping the raw input space entirely by performing augmentations in a learned latent space.
Designing Tabular-Safe Views
Instead of treating tabular rows like images, one line of work designs corruption and subsetting strategies that respect the empirical distributions of the features.
Random Feature Corruption (SCARF)
To create a meaningful alternative view of a tabular row without breaking its statistical properties, Dara Bahri et al. introduced SCARF (Self-Supervised Contrastive Learning using Random Feature Corruption). Instead of adding noise, SCARF selects a random subset of features in a row and replaces their values with values sampled from the empirical marginal distribution of those same features across the rest of the dataset.
Original Row: [ Age: 25, Income: 50k, Zip: 90210 ]
Marginal Pool: [ Age: 45, Income: 95k, Zip: 10001 ]
│
▼ (Swap Income)
Corrupted View: [ Age: 25, Income: 95k, Zip: 90210 ]
By swapping values from the marginal distribution, SCARF ensures that the corrupted view remains within a realistic range and maintains the overall distribution of the dataset, while forcing the contrastive model to learn which feature combinations define the identity of the original row.
Feature Subsetting (SubTab)
Rather than altering values, the authors of SubTab (Talip Ucar et al.) turned the lack of spatial structure into an advantage through multi-view feature subsetting. SubTab breaks a single high-dimensional row into multiple distinct, overlapping subsets of features.
The model is then tasked with taking any of these partial subsets and reconstructing the entire original row. By framing the pretext task as an autoencoder-based reconstruction from partial feature views, the network is forced to learn the underlying correlations and dependencies that bind different columns together.
Shifting to the Latent Space
A second major paradigm bypasses the messy, heterogeneous raw input space altogether. Because tabular features can be a mix of continuous scales and high-cardinality categorical fields, creating valid synthetic data points at the input level is exceptionally difficult. Instead, researchers apply mathematical transformations to the structured representations inside the neural network.
Weight and Representation Space Perturbations
Moonjung Eo et al. developed RaTab to perform representation-space augmentation. Instead of manipulating the input data, RaTab applies a truncated Singular Value Decomposition (SVD) directly to the encoder's weights during training. This mathematical truncation acts as a structural filter, generating positive pairs for contrastive learning in the latent space by stripping away noise while preserving the dominant coordinate systems of the data representations.
Manifold Subsetting and Distillation
Operating in a learned manifold also allows models to generate high-quality synthetic data for small datasets. Adrián Bazaga et al. introduced TabMDA, which uses pre-trained Transformers to perform "In-context Subsetting" within a learned manifold space, outputting diverse embeddings that respect the data's global structure.
This strategy is mirrored in dataset distillation. When trying to condense a massive tabular dataset into a tiny synthetic summary, distilling in the raw feature space often fails due to the lack of geometric symmetries. To resolve this, Inwon Kang et al. developed TDColER, a framework that performs dataset distillation entirely within a learned latent space, yielding synthetic samples that preserve label consistency and predictive power far better than raw-space alternatives.
Bypassing Views with Geometric Pretext Tasks
Rather than trying to engineer the perfect data view or latent perturbation, some approaches eliminate the need for contrastive pairs entirely by leveraging the mathematical properties of random projections.
Guansong Pang et al. proposed an unsupervised representation learning framework that forces a neural network to predict distances in a randomly projected space.
Raw Tabular Row (Xᵢ) ───► Neural Network ───► Embedding (Eᵢ) ───┐
▼
Predicted Distance
║ ║ (Minimize Error)
▼ ▼
Raw Tabular Row (Xᵢ) ───► Random Projection ──► Target Distance
Because random projections preserve relative distances according to the Johnson-Lindenstrauss lemma, forcing a neural network to predict these random distances acts as a powerful surrogate signal. The model learns to map high-dimensional, unstructured tabular rows into a continuous embedding space that preserves proximity relationships, completely bypassing the need for heuristic data augmentations or domain-specific labels.
Key Foundational Work
Unsupervised Representation Learning by Predicting Random Distances (2020) — contributed 1 concept elements to this direction.
VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain (2020) — contributed 1 concept elements to this direction.
SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning (2021) — contributed 1 concept elements to this direction.