When we simulate or learn the dynamics of real-world systems—whether it is a double pendulum swinging, a molecule folding, or a satellite orbiting Earth—we are rarely working in an unconstrained Euclidean space. Instead, the system's valid states live on a lower-dimensional surface embedded within a higher-dimensional space, known mathematically as a manifold. For example, a pendulum arm of fixed length $L$ restricts a particle's position to a circle or a sphere, rather than allowing it to fly freely anywhere in 3D space.
If we use a standard neural network or numerical solver to predict how such a system evolves over time, a fundamental issue arises: numerical drift. Unconstrained models do not inherently understand the underlying geometry. Over time, tiny errors accumulate at each integration step, causing the predicted trajectory to fly off the manifold. This leads to unphysical predictions, such as violating conservation laws (like energy or momentum) or generating impossible states.
To solve this, a powerful shared direction has emerged across machine learning, physics, and control theory: tangent space projection. Instead of letting a system drift and trying to patch it later, we can mathematically restrict the system's velocity vectors so that they always point along the manifold's tangent space. If the velocity is always tangent to the surface, the state trajectory is geometrically guaranteed to stay on the surface.
The Intuition: Staying on the Track
Imagine a race car driving on a banked, curved track. The track represents our manifold. At any given moment, the car has a position and a velocity vector. If the velocity vector points even slightly upward into the air or downward into the ground, the car will instantly leave the track. To keep the car on the track, its velocity vector must lie entirely within the flat plane that sits flush against the track at that exact position. This plane is the tangent space.
Off-manifold velocity (Drifts away!)
^
| .--- Tangent Space (Valid velocities)
| /
| /
_______________|/______
( State (x) o ) <--- Constraint Manifold
`---------------------'
By forcing the system's velocity vector to lie within this tangent space, we transform a complex, constrained optimization problem into an unconstrained one. We no longer need to constantly check if the state is valid; the geometry of the velocity field ensures validity automatically.
The Problem of Drift in Machine Learning
Modern machine learning models, such as Neural Ordinary Differential Equations (Neural ODEs), excel at learning continuous-time trajectories from data. However, White et al. (Projected Neural Differential Equations for Learning Constrained Dynamics) demonstrate that standard NDEs frequently suffer from drifting because they ignore physical laws like fixed joint lengths or energy conservation.
Similarly, when machine learning surrogates are combined with traditional physics simulators, distribution shifts can cause the model to wander into unphysical states. To combat this, Zhao et al. (Mitigating distribution shift in machine learning-augmented hybrid simulation) introduce a tangent-space regularization technique. By adding a penalty term to the loss function that minimizes any velocity component perpendicular to a learned data manifold, they force the simulation to maintain long-term physical consistency, even when encountering states slightly outside the training distribution.
Mechanisms of Projection
Researchers have developed two primary paradigms to enforce these geometric constraints: preventing drift before it happens (a priori restriction), or correcting it immediately after a step is taken (a posteriori projection).
A Priori Vector Field Restriction
In this approach, the model is mathematically designed so that it is physically incapable of producing an off-manifold velocity vector.
For neural networks, Elamvazhuthi et al. (Learning on Manifolds: Universal Approximations Properties using Geometric Controllability Conditions for Neural ODEs) define manifold-invariant dynamics by restricting the network's output to vector fields that are strictly tangent to a known manifold at every point. Similarly, Matsubara et al. (FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantities) project the output of a neural differential equation onto a tangent space defined by learned invariant quantities (like total energy), ensuring that specific scalar properties remain perfectly constant throughout the simulation.
This is not limited to neural networks. For instance, Huang (Learning vector fields of differential equations on manifolds with geometrically constrained operator-valued kernels) utilizes operator-valued kernels that are mathematically restricted to produce vector fields lying solely within the tangent space of a manifold. This geometric prior allows the model to maintain stable kinetic energy where unconstrained approaches fail.
A Posteriori Projection and Correction
Alternatively, a solver can take a standard, unconstrained numerical step and then immediately pull the state back onto the manifold.
Zamuner et al. (An Efficient Algorithm to Perform Local Concerted Movements of a Chain Molecule) describe this as a predictor-corrector sequence for molecular chains: a small step is taken along the tangent space to reach an intermediate state, and a root-finding algorithm then projects this state back onto the manifold by moving orthogonally to the constraints.
1. Start at State (x)
2. Take a step along the Tangent Space ---> Intermediate State
|
| 3. Project back orthogonally
v
New Valid State (x')
Kemeth et al. (Learning emergent partial differential equations in a learned emergent space) apply a similar filtering step during time-integration, using singular value decomposition to project neural network predictions back onto a learned manifold to prevent data drift. In materials science, Zhang et al. (A New Formulation of Coupling and Sliding Motions of Grain Boundaries Based on Dislocation Structure) utilize this update-then-project strategy to satisfy local geometric constraints without relying on computationally expensive long-range integrals.
Continuous Landing Forces
An elegant alternative to rigid projection steps is the integration of a continuous force that pulls trajectories back toward the manifold. Jeon et al. (Fast Non-Log-Concave Sampling under Nonconvex Equality and Inequality Constraints with Landing) introduce a deterministic "landing" drift term directly into Langevin dynamics. Instead of performing expensive, iterative algebraic projections at every step, this continuous force automatically pulls sample trajectories toward the feasible constraint set, achieving exponential convergence.
Kinematic Consistency in Reduced-Order Models
Tangent space projection becomes vital when trying to speed up massive physical simulations using dimensionality reduction. Instead of simulating millions of degrees of freedom (like individual particles in a fluid), we can map the system to a low-dimensional latent manifold learned by a deep autoencoder.
However, Lee et al. (Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders) point out that mapping dynamics to a learned latent space can lead to kinematic inconsistency if the system's velocity is not properly handled. To maintain physical validity, the time derivative of the high-dimensional state must reside precisely within the tangent space of the nonlinear manifold at that current point.
By enforcing this constraint, Chen et al. (Model Reduction for the Material Point Method on Nonlinear Manifolds Using Deep Learning) project computed velocities onto the tangent space of a learned manifold at each time step. This projection step constrains the evolution of the system to the manifold's geometry, allowing them to reduce complex material simulations down to just a handful of generalized coordinates with less than 1% error.
Applications Across Domains
The principle of constraining dynamics via tangent space projection is highly versatile and spans multiple domains:
- Classical Mechanics & Robotics: Finzi et al. (Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints) show that switching from complex generalized coordinates (like angles) to simple Cartesian coordinates ($x, y, z$) and applying a projection matrix to the state derivatives keeps trajectories strictly on the physical manifold. This makes the underlying physics much easier for models to learn, improving data efficiency up to 100x.
- Trajectory Optimization: Bonalli et al. (Trajectory Optimization on Manifolds: A Theoretically-Guaranteed Embedded Sequential Convex Programming Approach and Analysis of Theoretical and Numerical Properties of Sequential Convex Programming for Continuous-Time Optimal Control) prove that when system dynamics are naturally tangent to a state manifold, explicit manifold constraints can actually be omitted from the convex subproblems during optimization. The iterative linearization process naturally pulls the trajectory toward the manifold, drastically simplifying the implementation for complex systems like satellites or robotic arms.
- Mathematical Finance: Bayraktar et al. (Projecting the Forward Rate Flow on a Finite Dimensional Manifold and Projecting the Forward Rate Flow onto a Finite Dimensional Manifold) provide a framework for projecting infinite-dimensional interest rate models onto finite-dimensional manifolds. When an infinite-dimensional velocity vector points outside the manifold, it is orthogonally projected onto the tangent space, allowing banks to calibrate complex financial models using standard discrete market data.
- Generative Modeling: Bovik (Rectified CFG++ for Flow Based Models) applies a manifold-constrained approach to flow-based generative models. By restricting sampling trajectories to a bounded neighborhood of the learned transport path, it prevents the off-manifold drift common in deterministic ODE solvers, eliminating visual artifacts and oversaturation in generated images.
Whether applied to the motion of a molecule, the stability of a power grid, or the generation of an image, projecting dynamics onto the tangent space of a manifold provides a mathematically rigorous way to enforce hard constraints, eliminate numerical drift, and preserve the intrinsic geometry of the system.
Key Foundational Work
Projecting the Forward Rate Flow on a Finite Dimensional Manifold (2003) — contributed 1 concept elements to this direction.
On an existence and uniqueness theory for nonlinear differential-algebraic equations (1991) — contributed 1 concept elements to this direction.