Geometric Foundations of Tuning without Forgetting in Neural ODEs

Erkan Bayram, Mohamed-Ali Belabbas, Tamer BasararXiv 2025 · DOI · arXiv

Introduction

In the field of continual learning, models must adapt to new tasks without erasing previously acquired knowledge—a phenomenon known as catastrophic forgetting. Neural Ordinary Differential Equations (Neural ODEs) provide a continuous-time framework for deep learning, where the evolution of states is governed by a differential equation $\dot{x} = f(x, u)$. While the "Tuning without Forgetting" (TwF) algorithm was previously proposed as a heuristic to mitigate this issue, its theoretical underpinnings remained largely unexplored until now.

The Manifold of Knowledge

The core contribution of this work is the rigorous proof that the set of parameters (or control functions) that preserve the mapping of specific inputs to desired outputs forms a smooth Banach submanifold. In a Neural ODE, the goal is to find a control $u$ such that the flow $\phi_t(u, \cdot)$ maps initial data to target states.

Mapping input data points to target planes ($x_3 = \pm 1$) using control functions in a Neural ODE framework.

By treating the space of controls as an infinite-dimensional Banach space, the authors demonstrate that the constraints imposed by a finite training set do not collapse the space. Instead, they define a "surface" within this space where the model's performance on old data remains constant. This shift from first-order linear approximations to exact manifold preservation is the key to avoiding the accumulation of errors that typically leads to forgetting.

Sequential Learning as Path-Following

Traditional approaches to continual learning often rely on Taylor approximations to stay near "good" parameters. However, these linearizations are only valid locally. This paper shows that one can move exactly along the manifold of "good" parameters to incorporate new data by utilizing the Transversal Mapping Theorem. This theorem ensures that the requirements for different data points do not conflict in a way that destroys the smooth manifold structure.

The trajectory of control updates $u^k$ moving from a state $u_j$ (preserving $j$ points) to $u_{j+1}$ (preserving $j+1$ points) within the intersection of constraint sets.

As shown in the figure above, the learning process can be viewed as finding a path through the intersection of sets $U(x^i, y^i)$, where each set represents the controls that satisfy the $i$-th data point. The TwF algorithm effectively projects the gradient of a new task onto the tangent space of this intersection, allowing for a continuous "deformation" of the control function without leaving the submanifold of prior knowledge.

Theoretical Significance and the LCP

The validity of this geometric structure hinges on the Linearized Controllability Property (LCP). This property ensures that the system has sufficient degrees of freedom to reach any target state from a given input. In the context of over-parameterized neural networks, LCP is generally satisfied, meaning the model has enough "room" to learn new information. The authors also note that the theory holds as long as the model avoids "abnormal extremals"—singular states where controllability is lost—which are rare in typical neural architectures.

Furthermore, the authors prove that the submanifold has finite codimension. Despite the infinite-dimensional nature of the control space, each new data point only removes a finite number of dimensions. This explains why large-scale models can continue to learn indefinitely without necessarily interfering with previous tasks.

Conclusion

By moving from intuitive heuristics to geometric control theory, this research provides a solid mathematical foundation for sequential training in Neural ODEs. The transition from approximate to exact preservation marks a significant step toward solving catastrophic forgetting, suggesting that with the right geometric updates, models can truly learn without forgetting. Future work aims to extend these findings to normalized flows and continuous probability distributions.

Summary updated

BibTeX
@misc{bayram2025geometric,
  author = {Erkan Bayram and Mohamed-Ali Belabbas and Tamer Basar},
  title = {{Geometric Foundations of Tuning without Forgetting in Neural ODEs}},
  year = {2025},
  eprint = {2509.03474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03474},
}
Figure