Regularizing Deep Networks with Latent Space Interpolation

36 concepts38 papers1 proposal

Data augmentation is a cornerstone of modern deep learning, acting as a powerful regularizer that forces models to generalize rather than memorize. In continuous domains like computer vision, simple input-space manipulations—such as rotating an image or linearly blending two pixel matrices via standard Mixup—work remarkably well.

However, when dealing with discrete tokens (like text or source code) or highly structured data (like graphs), input-space mixing breaks down completely. As surveyed by Steven Y. Feng et al. in A Survey of Data Augmentation Approaches for NLP, averaging the discrete token IDs or one-hot vectors of two words yields semantic nonsense. Similarly, linearly combining the raw syntax trees of two source code files results in uncompilable garbage, a challenge highlighted in code classification by Zeming Dong et al. in MixCode: Enhancing Code Classification by Mixup-Based Data Augmentation.

To circumvent the limitations of discrete input spaces, a powerful paradigm has emerged: Latent Space Interpolation (often called Manifold Mixup). Instead of mixing raw inputs, this technique allows data to pass through the initial layers of a neural network into a continuous, high-dimensional hidden representation space. It is within this smooth semantic manifold that linear interpolation takes place.


The Core Mechanism

The mathematical formulation of latent space interpolation is elegant and computationally inexpensive. Let $x_A$ and $x_B$ be two distinct training samples with target labels $y_A$ and $y_B$.

  1. Forward Propagation: Pass both inputs through the network up to a chosen hidden layer $l$. This yields two continuous feature representations, $h_A^l$ and $h_B^l$.
  2. Linear Interpolation: Mix these hidden states using a mixing coefficient $\lambda \in [0, 1]$ typically drawn from a Beta or Dirichlet distribution: $$h_{mix}^l = \lambda h_A^l + (1 - \lambda) h_B^l$$
  3. Label Blending: Linearly combine their corresponding target labels to form a soft target: $$y_{mix} = \lambda y_A + (1 - \lambda) y_B$$
  4. Remaining Forward Pass: Feed the synthetic hidden state $h_{mix}^l$ through the remaining layers of the network to compute the final prediction and optimize against $y_{mix}$.
Input Space (Discrete/Rigid)       Hidden Manifold (Continuous/Smooth)
   x_A  ------------[ Layer 1...l ]------------>  h_A^l \
                                                         \--->  h_mix^l  ---> [ Remaining Layers ] ---> Loss vs y_mix
   x_B  ------------[ Layer 1...l ]------------>  h_B^l /

This operation can occur at the initial embedding layer, as explored by Abele Mălan et al. in DALLMi: Domain Adaption for LLM-Based Multi-label Classifier, or at a randomly selected deep layer during each training step, as implemented by Amit Jindal et al. in SpeechMix - Augmenting Deep Sound Recognition Using Hidden Space Interpolations.


Why It Works: Geometric Intuition

To understand why latent space interpolation is so effective, we must look at how deep networks organize information geometrically.

Smoothing Decision Boundaries

Deep neural networks naturally tend to construct highly complex, jagged decision boundaries in order to separate training clusters perfectly. In the unpopulated gaps between these clusters, the model's predictions can be erratic and overconfident.

By forcing the network to predict a linear combination of labels ($y_{mix}$) for a linear combination of hidden states ($h_{mix}^l$), manifold mixup flattens the decision boundaries. Jishnu Ray Chowdhury et al. leverage this property in Cross-Lingual Disaster-related Multi-label Tweet Classification with Manifold Mixup, demonstrating that a smoother latent space prevents the model from overfitting to language-specific quirks, enabling zero-shot generalization across entirely different languages.

Preventing Representation Collapse

In semi-supervised and contrastive learning, models often suffer from representation collapse—a failure mode where the network maps structurally distinct inputs to identical or highly clustered embeddings to minimize loss trivially.

Xihong Yang et al. address this in Mixed Graph Contrastive Network for Semi-Supervised Node Classification. By interpolating node embeddings within a Graph Neural Network (GNN), they force the model to maintain a strict linear relationship between high-level representations. This expands the decision boundaries outward, ensuring that the network preserves discriminative, task-specific features instead of collapsing them.

Filling the Semantic Manifold

Real-world datasets are inherently sparse; they contain vast empty spaces between known data points. Latent space interpolation effectively "hallucinates" plausible intermediate states that lie on the continuous path between real concepts.

In machine translation, Guandan Chen et al. demonstrate in Manifold Adversarial Augmentation for Neural Machine Translation that mixing high-level semantic representations fills critical gaps in the translation manifold. This allows the decoder to learn from synthetic examples that exist between real sentences, smoothing out the generation process.


Advanced Variations

As the field has matured, researchers have moved beyond simple single-step linear combinations toward more sophisticated routing mechanisms within the latent space.

Multi-Step and Neighborhood Mixing

Rather than just drawing a straight line between two random points, some approaches construct richer local neighborhoods. In DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification, Soujanya Poria et al. propose a two-step interpolation. First, they generate multiple perturbed versions of an input and blend their hidden states using a Dirichlet distribution to form a synthetic anchor. Second, they mix the original input's representation with this synthetic anchor using a Beta distribution. This multi-layered blending creates a highly robust, continuous region around each training sample, which is particularly effective in low-resource settings.

Confidence-Driven Propagation

Instead of pairing representations entirely at random, training can be guided by the model's own certainty. In SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe, Yuxin Xiao introduces a strategy that groups data based on the model's internal confidence scores. By interpolating representations across different confidence subsets, the model learns a smooth transition from easy to hard data points, effectively propagating supervision signals across the latent space without requiring aggressive data filtering.

Generative Latent Navigation

While standard mixup relies on direct linear paths, generative models can learn the underlying structure of the manifold itself. Kang Min Yoo et al. implement this in the Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation. By modeling conversations through disentangled latent variables, navigating and interpolating within this learned generative space allows the system to synthesize entirely new, coherent dialog trajectories that blend complex user goals.


A Universal Domain-Agnostic Regularizer

Because every deep learning architecture maps inputs to continuous hidden states, latent space interpolation is fundamentally domain-agnostic.

Key Foundational Work

Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation (2020) — contributed 1 concept elements to this direction.
Cross-Lingual Disaster-related Multi-label Tweet Classification with Manifold Mixup (2020) — contributed 1 concept elements to this direction.
SpeechMix - Augmenting Deep Sound Recognition Using Hidden Space Interpolations (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_space_interpolation_manifold_mixu_8e23cc61,
  title = {latent space interpolation (manifold mixup)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-interpolation-manifold-mixup/rdr_8e23cc61df3340d38cf5a230afed651e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_space_interpolation_manifold_mixu_8e23cc61_v1,
  title = {latent space interpolation (manifold mixup)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-interpolation-manifold-mixup/rdr_8e23cc61df3340d38cf5a230afed651e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake