Data augmentation by interpolation—popularized by standard Mixup—operates on a simple, elegant premise: if you draw a straight line between two real data points, any point along that line should represent a smooth, convex combination of both the inputs and their corresponding labels.
While this assumption works well in simple geometric spaces, real-world data distributions are rarely flat hyperplanes. Instead, they exist as highly non-linear, curved, and low-dimensional manifolds embedded within high-dimensional spaces. When we blindly draw a straight line between two distant points on a curved manifold, that line inevitably leaves the true data distribution, passing through empty space or cutting directly through the territory of an entirely unrelated class.
This phenomenon is known as manifold intrusion. It creates a severe label-feature misalignment: the synthetic sample is assigned a blended label from its parent classes, yet its location in feature space places it squarely inside the domain of a third class.
The Cost of Blind Interpolation
To understand why manifold intrusion degrades performance, consider a geometric landscape where Class A wraps around Class B in a horseshoe shape. If we pick two points at opposite tips of the Class A horseshoe and linearly interpolate between them, the midpoint will land directly inside the high-density region of Class B. Standard Mixup, however, will label this midpoint as 100% Class A.
When this synthetic sample is fed into a neural network, it introduces a destructive label conflict. The model is forced to resolve two contradictory signals: the real training data asserting that this region belongs to Class B, and the synthetic data asserting it belongs to Class A.
Standard Mixup (Linear Path):
[Class A] ------------------ (Intrusion Zone: Labeled as A, but looks like B) ------------------ [Class A]
▲
Actual Class B Manifold
As Zixuan Liu et al. demonstrate in Over-Training with Mixup May Hurt Generalization, this contradiction acts as a regularizer early in training, but prolonged exposure forces the network to fit these flawed, ambiguous labels. The result is sharper loss minima, underfitting, and ultimately poorer generalization.
Muthu Chidambaram et al. mathematically formalize this failure mode in Towards Understanding the Data Dependency of Mixup-style Training. They show that when a sample from one class is geometrically positioned between samples of another class, the synthetic mixtures generated from the outer points can dominate the loss function. This overrides the true signal of the central point, causing the model to misclassify the original, clean data points it was meant to learn.
Furthermore, this label noise directly damages a model's self-awareness. As noted by Sunil Thulasidasan et al. in On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks, if the mixing intensity is not carefully calibrated, these out-of-manifold samples lead to severe model under-confidence and degraded predictive uncertainty.
To solve this, modern research focuses on three main strategies: rewriting the labels, restricting the interpolation paths, or abandoning linear paths altogether.
Strategy 1: Manifold-Aware Relabeling
If we cannot prevent a synthetic sample from landing inside an intrusion zone, the next best option is to correct its label before it confuses the model. Instead of blindly trusting the mathematical interpolation ratio, we can adjust the label to match the semantic reality of the space where the sample landed.
Manifold-Aware Relabeling:
[Class A] ------------------ (Intrusion Zone: Relabeled to Class B) ------------------ [Class A]
In GenLabel: Mixup Relabeling using Generative Models, Jy-yong Sohn et al. introduce a framework that uses generative models to estimate the true class-conditional likelihoods of mixed inputs. If an interpolated sample drifts into the feature space of a third, uninvolved class, GenLabel detects the intrusion and overrides the original label with the high-density class label of that zone. By providing accurate "soft labels" that resolve label conflicts, this approach cleans up decision boundaries and maximizes classification margins.
Strategy 2: Proximity and Similarity Constraints
A second preventative strategy is to restrict interpolation to pairs of samples that are already close to one another in feature or label space. If two points are highly similar, the straight line connecting them is much less likely to exit the local data manifold or cross into an adversarial class territory.
For regression tasks, where interpolating between distant points frequently creates highly unrealistic feature-label pairings, Huaxiu Yao et al. propose C-Mixup: Improving Generalization in Regression. C-Mixup intelligently conditions the selection of mixing pairs on their label similarity, minimizing the risk of generating noisy, out-of-manifold training signals.
For classification and model calibration, Quentin Bouniot et al. develop Similarity Kernel Mixup (SK Mixup) in Tailoring Mixup to Data for Calibration. They treat the distance between training samples as a direct proxy for the risk of manifold intrusion. By dynamically reducing the interpolation strength when forced to mix highly dissimilar points, SK Mixup ensures that the generated labels remain reliable and well-aligned with the actual data distribution.
Rather than relying on simple pair-wise heuristics, Kristjan Greenewald et al. look at the problem through the lens of global geometry in $k$-Mixup Regularization for Deep Learning via Optimal Transport. By using Optimal Transport to match batches of $k$ points, their framework constructs "locally-informed" synthetic training examples that respect the underlying cluster structures of the dataset.
Strategy 3: Non-linear and Geodesic Paths
The most fundamental departure from standard Mixup involves changing the interpolation path itself. If the data manifold is curved, our interpolation paths should bend along with it.
Geodesic Interpolation (Non-linear Path):
[Class A] \ / [Class A]
\___ (Follows the true manifold curve, avoiding Class B entirely) ___/
In Nonlinear Mixup: Out-Of-Manifold Data Augmentation for Text Classification, Hongyu Guo breaks the linear constraint entirely. By introducing a matrix-based mixing policy combined with a learnable label-assignment function, this approach allows the model to safely navigate non-linear, out-of-manifold spaces without generating destructive label noise.
This geometric precision becomes even more critical when dealing with structured data like graphs, where simple linear averaging of adjacency matrices yields structurally incoherent results. To address this, Zhichen Zeng et al. introduce GEOMIX in Graph Mixup on Approximate Gromov–Wasserstein Geodesics. Instead of drawing a straight line through Euclidean space, GEOMIX interpolates along Gromov-Wasserstein geodesics—the mathematically principled "shortest paths" within the graph space. This guarantees that the structural properties of the newly synthesized graphs remain perfectly aligned with their target labels, eliminating the structural mismatches that plague naive graph mixing.
Key Foundational Work
MixUp as Locally Linear Out-of-Manifold Regularization (2019) — contributed 3 concept elements to this direction.
Nonlinear Mixup: Out-Of-Manifold Data Augmentation for Text Classification (2020) — contributed 1 concept elements to this direction.
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (2019) — contributed 1 concept elements to this direction.