Staying on the Manifold: Geometry-Aware Noise Injection

Albert Kjøller Jacobsen, Johanna Marie Gegenfurtner, Georgios Arvanitidis

Abstract

Data augmentation through noise injection is a cornerstone of robust machine learning. However, for data residing on low-dimensional manifolds, standard Gaussian noise often results in "off-manifold" samples that are non-informative or physically impossible. This work introduces geometry-aware noise injection techniques that constrain perturbations to the data's natural geometric structure. By evaluating strategies such as projected tangent noise, geodesic noise, and intrinsic Brownian motion, we demonstrate that respecting the manifold leads to significantly better generalization and robustness to noise intensity, especially on datasets with high curvature.

Introduction

In machine learning, adding random Gaussian noise to training data is a common technique to improve model robustness. However, most high-dimensional data—such as images, climate patterns, or biological signals—actually lives on a much lower-dimensional "manifold."

The fundamental problem with standard Gaussian noise is that in high dimensions, it is almost always perpendicular to this manifold. Adding "ambient" noise creates synthetic data points that deviate from the underlying geometric structure, resulting in samples that may be physically impossible or non-informative. This paper addresses this gap by proposing methods to keep perturbations "on the manifold."

Geometry-Aware Noise Strategies

The authors propose three escalating strategies for geometry-aware noise injection, moving from linear approximations to intrinsic stochastic processes:

  1. Projected Tangent Space Noise: Standard Gaussian noise is projected onto the tangent plane of the data point. This linear approximation keeps noise close to the manifold locally.
  2. Geodesic Noise: A more sophisticated approach that maps noise samples onto the manifold using the "exponential map." This ensures the noisy point follows the actual curvature of the surface.
  3. Intrinsic Brownian Motion: Instead of a single jump, this simulates a random walk directly on the manifold. This is the most natural way to model movement on a curved surface and is computationally efficient as it avoids high-precision ODE solvers.
Comparison of noise injection strategies: ambient noise (red) deviates from the manifold, while geometry-aware noise (green/orange) respects the data's underlying structure.

Theoretically, the authors demonstrate that adding tangential noise is equivalent to applying a Tikhonov regularizer specifically to the tangential component of the model's gradient, encouraging the model to be "flat" along the data surface rather than in arbitrary directions.

Experimental Evaluation

The researchers evaluated these methods across two distinct domains:

  • Synthetic 3D Manifolds: A suite of geometries including Spheres, Squeezed Spheres, and the highly curved Swiss Roll. These provided precise control over curvature to test the limits of each noise strategy.
  • MNIST Digits: To test real-world high-dimensional data, an autoencoder was used to learn a 16-dimensional latent manifold. Brownian motion noise was then applied within this learned latent space.

Key Results and Robustness

The most striking finding is the robustness of geometry-aware noise to intensity. While standard ambient noise quickly destroys model performance as the variance increases, Brownian motion noise remains beneficial even at high intensities. On the complex Swiss Roll manifold, Brownian motion noise reduced the Mean Squared Error (MSE) by over 50% compared to the baseline (0.46 vs 1.00).

On the Swiss Roll dataset, geometry-aware strategies (Geodesic and Brownian) show significantly higher robustness to noise intensity compared to ambient noise, which causes test loss to spike.

In real-world applications, the benefits were equally clear. In low-data regimes (using only 1% of MNIST training data), Brownian motion noise improved test accuracy from 88.3% to 89.6%, outperforming both standard images and those augmented with ambient Gaussian noise.

Conclusion

Geometry-aware noise injection ensures that a model learns to be smooth in directions that actually matter for the data distribution. While there is a computational overhead in calculating Jacobians or simulating stochastic differential equations, the gains in generalization and hyperparameter robustness—especially on "wiggly" or highly curved manifolds—make it a superior alternative to standard ambient noise. Future work may involve incorporating label information to ensure that long-duration Brownian motion does not inadvertently cross class boundaries.

Summary updated

BibTeX
@inproceedings{jacobsen2026staying,
  author = {Albert Kjøller Jacobsen and Johanna Marie Gegenfurtner and Georgios Arvanitidis},
  title = {{Staying on the Manifold: Geometry-Aware Noise Injection}},
  booktitle = {NLDL},
  year = {2026},
  url = {https://openreview.net/forum?id=GRk65sJxpn},
}
Figure