Self-supervised learning (SSL) has a "specialization" problem. To learn that two versions of an image represent the same object, researchers typically rely on hand-crafted, domain-specific tricks like random cropping, color jittering, or rotation. While these heuristics work for natural images, they fail when applied to medical tables, financial time series, or genomic sequences. "Cropping" a table of patient data or "rotating" a financial trend doesn't just change the view—it can destroy the underlying meaning.
A growing body of research is moving away from these manual priors toward modality-agnostic representation learning. The goal is to develop universal mechanisms that learn invariant features regardless of whether the input is a pixel, a byte, or a sensor reading.
The Latent Space Escape
The most direct way to bypass domain-specific rules is to move the "gymnastics" from the raw input to the model's internal hidden layers. In MODALS, Dit-Yan Yeung and Tsz-Him Cheung demonstrate that by learning to interpolate and perturb abstract data representations in the latent space, models can eliminate the need for hand-crafted augmentation rules.
This "feature-space" approach allows the model to discover which transformations preserve identity. Similarly, Zichang Liu and Anshumali Shrivastava introduced LeMDA (Learning Multimodal Data Augmentation in Feature Space), which uses adversarial training to generate informative new samples in the latent space. By perturbing the features rather than the raw data, these frameworks can be applied to any combination of image, text, and tabular data without modification.
Universal Noise and Interpolation
If we cannot rely on domain-specific transformations like "flipping," we need a universal language for creating "different views" of the same data. One such language is Mixup—the process of creating new training samples by linearly interpolating between existing ones.
Hieu Pham 0001, Quoc V Le, and colleagues proposed DACL (Towards Domain-Agnostic Contrastive Learning), which replaces specialized image tricks with "Mixup noise." Because interpolation is a purely mathematical operation on vectors, it works identically across tabular, image, and graph data.
Taking this further, Hongyuan Zhang and Sida Huang frame data augmentation as the estimation of Positive-incentive Noise ($\pi$-noise). Their framework, PiNDA (Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise), replaces hand-crafted augmentations with a learned noise generator that discovers beneficial perturbations for any dataset, providing a theoretical bridge between information theory and representation learning.
Masking as a Universal Primitive
Another path to modality independence is treating all data as a sequence or a collection of channels that can be partially hidden. This shifts the task from "comparing two views" to "reconstructing the whole from a part."
- Randomized Quantization (RQ): Huimin Wu and Xiao Sun introduced Randomized Quantization, which treats quantization as a form of "channel-wise masking." This single tool outperforms specialized methods across images, audio, and 3D point clouds.
- MetaMAE: Huiwon Jang and Jihoon Tack developed MetaMAE, which uses meta-learning to adapt the popular Masked Auto-Encoder (MAE) architecture to vastly different data types, from genomic sequences to industrial sensor logs.
- Byte-Level Processing: Perhaps the ultimate form of modality independence is ByteFormer. Maxwell Horton and Mohammad Rastegari showed that a Transformer can operate directly on raw file bytes (like JPEGs or MP3s) without any decoding. By treating every file as a sequence of bytes, the same architecture and training process can achieve high accuracy on both ImageNet and speech commands.
Bypassing Augmentation Entirely
Some researchers argue that the need for "views" or "masks" is itself a limitation. Yi Sui and George Stein proposed Learning from Randomness (LFR) (Self-supervised Representation Learning from Random Data Projectors). Instead of augmenting the data, they train an encoder to reconstruct the outputs of fixed, randomly initialized data projectors. Because the "target" is generated by a random projection of the original data, the model learns high-quality features for tabular and medical data without ever needing a rotation or a crop.
Similarly, Ilan Naiman developed SPYL (Sample and Predict Your Latent), which samples "positive" and "negative" views directly from the internal distributions of a Variational Autoencoder (VAE). This allows for sequential disentanglement in video, audio, and medical time series without any modality-specific engineering.
Why It Works
The shift toward modality-agnostic methods works because it forces models to learn structural invariance rather than visual invariance.
When we use image-specific augmentations, we are hard-coding our own human biases—telling the model that "color doesn't matter for object identity." Modality-agnostic methods like Modality-Agnostic Debiasing (MAD) by Sanqing Qu and Ting Yao instead identify and suppress domain-specific biases within the classifier itself. By removing the reliance on hand-crafted priors, these models become more robust to "in-the-wild" distribution shifts where human intuition about what "should" be invariant often fails (Sagawa et al., WILDS 2.0).