From NeurODEs to AutoencODEs: A mean-field control framework for width-varying neural networks

Cristina Cipriani, Massimo Fornasier, Alessandro Scagliotti

TL;DR

This paper introduces AutoencODEs, a novel framework that extends Neural Ordinary Differential Equations (NeurODEs) to handle architectures with varying layer widths, such as autoencoders. By mathematically "turning off" specific dimensions over time within a high-dimensional space, the authors provide a rigorous mean-field control theory for these networks and a stabilized training algorithm that remains effective even with low regularization.

What Problem Does This Paper Address?

NeurODEs have gained popularity because they treat neural networks as continuous-time dynamical systems, allowing researchers to use tools from control theory to analyze deep learning. However, standard NeurODEs have a major limitation: they are "rectangular." Because the state $X(t)$ evolves according to an ODE, the dimensionality of the "layers" must remain constant throughout the network.

In contrast, most modern deep learning architectures—like autoencoders or U-Nets—rely on varying widths. They compress data into a low-dimensional "bottleneck" (encoding) before expanding it back (decoding). This paper bridges the gap by creating a mathematical framework that allows NeurODEs to change dimensions while maintaining the theoretical benefits of the continuous-time perspective.

Methods

The authors propose a clever "embedding" strategy to simulate variable width:

  1. Discontinuous Vector Fields: Instead of physically changing the dimension of the space, they keep the system in a high-dimensional space $\mathbb{R}^d$ but use a vector field that is discontinuous in time. During the "encoding" phase, specific components of the state vector are artificially forced to remain constant (inactive), effectively reducing the active dimensionality.
  2. Mean-Field Control: They analyze the system in the "mean-field" limit, where the training dataset is treated as an infinite population represented by a probability distribution. This allows them to derive optimality conditions using the Pontryagin Maximum Principle (PMP).
  3. Minimizing Movement Scheme: Training NeurODEs with low regularization often leads to numerical instability and non-convex cost landscapes. The authors introduce a proximal stabilization term (a "memory" term) into their shooting method algorithm. This ensures that updates to the network parameters don't swing too wildly, providing a stable "learning rate" rooted in optimization theory.

Datasets

The authors validated their framework on three types of tasks:

  • Bidimensional Classification: A synthetic task using particles sampled from a standard Gaussian distribution in $\mathbb{R}^2$, labeled based on the sign of their first component.
  • Parabola Reconstruction: A 2D reconstruction task where the network must learn to map points back to a parabolic curve after passing through a 1D bottleneck.
  • MNIST: The classic dataset of 70,000 grayscale images of handwritten digits ($28 \times 28$ pixels), used to test the model on high-dimensional data.

Key Insights and Ideas

  • The "Natural" vs. "Unnatural" Turn-off: In 2D experiments, the authors showed that even if they "turned off" the most important dimension during the bottleneck phase, the network was smart enough to rearrange the data into the active dimension before the bottleneck hit. This suggests the architecture is robust to how dimensions are selected for deactivation.
  • Automatic Intrinsic Dimension Discovery: Perhaps the most striking finding occurred during the MNIST experiment. Even when given a 32-dimensional bottleneck, the AutoencODE (initialized at zero) naturally converged to a sparse representation using only 13 non-zero components. This matches existing independent research estimating the intrinsic dimensionality of MNIST to be approximately 13.
  • Stability via Lipschitz Constraints: The authors used the inverse of the Lipschitz constant of the vector field as a "sanity check" for their architecture. They showed that if the discretization step ($dt$) is too large relative to this constant, the network becomes unstable—providing a principled way to choose the number of layers.

Key Results

  • Theoretical Soundness: The paper proves the existence and uniqueness of solutions for the mean-field optimal control problem and establishes a quantitative bound on generalization error.
  • MNIST Performance: The AutoencODE successfully reconstructed MNIST digits with high fidelity. Unlike standard autoencoders, which often fail when initialized with zeros, the AutoencODE’s stabilized training allowed it to converge to a sparse, meaningful latent space even from a zero initialization.
  • Local Convexity: While the global loss function is non-convex at low regularization, the authors demonstrated that the landscape is locally semiconvex, meaning standard optimization tools can still find high-quality local minima.

Limitations and Future Work

  • Computational Comparison: The authors note that a direct performance comparison with standard autoencoders (trained via Adam/SGD) is currently difficult because standard deep learning libraries are highly optimized for discrete backpropagation, whereas this framework uses a specialized "shooting method."
  • Stochasticity: The current algorithm is deterministic. The authors suggest that incorporating a "stochastic PMP"—subsampling data batches during control updates—could significantly speed up convergence and improve generalization.
  • Complex Topologies: While the paper mentions extensions to U-Nets (architectures with skip connections across the bottleneck), the primary focus remains on sequential encoder-decoder structures. Exploring more complex branching dynamics is a suggested next step.

Summary updated

Abstract

Abstract The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks, which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modelling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularisation, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularisation may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples.
BibTeX
@inproceedings{cipriani2024from,
  author = {Cristina Cipriani and Massimo Fornasier and Alessandro Scagliotti},
  title = {{From NeurODEs to AutoencODEs: A mean-field control framework for width-varying neural networks}},
  booktitle = {European},
  year = {2024},
  doi = {10.1017/s0956792524000032},
}
Figure