Abstract
Deep learning's reliance on backpropagation presents significant computational challenges and theoretical obscurities. This paper introduces a "forward-only" training framework that replaces traditional feedback mechanisms with geometric trajectories on function manifolds. By framing learning as a realization of differential equations, the authors establish a formal link between geometric regularization and Residual Networks (ResNets), achieving high accuracy on MNIST without backpropagation.
Introduction
Despite the empirical success of deep learning, it remains a "black box" characterized by training complexity and a lack of theoretical clarity. Standard backpropagation is computationally expensive and prone to vanishing or exploding gradients. Furthermore, providing an exact assessment of a network's expressive power remains an open challenge. This work seeks to "demystify" these processes by replacing feedback-heavy optimization with a feed-forward iterative projection method rooted in differential geometry.
Methodology: Geometric Regularization and Forward Projection
The proposed framework treats supervised learning as the realization of a differential equation rather than the global optimization of a weight matrix.
Geometric Constraints
Instead of unconstrained parameter updates, the model restricts "tangent vectors"—the directions of function evolution—to a specific subspace or "cone." This geometric regularization ensures the model remains structured and prevents wild overfitting. Using the Frobenius Integrability Theorem, the authors demonstrate that if update directions meet certain involutivity conditions, the resulting function is guaranteed to reside on a smooth, low-dimensional manifold.
Iterative Updates
Parameters are updated layer-by-layer during the forward pass. This incremental approach mimics the flow of a differential equation. To enhance practicality, the system incorporates momentum (low-pass filtering) and convolutional structures to handle image data effectively.
ResNets as Continuous Limits
A key insight of this research is that modified Residual Network (ResNet) architectures are discrete approximations of continuous differential equations. As shown in the architectural diagrams, the transition from standard layers to residual connections provides a formal geometric justification for the superior performance of ResNets. This "semi-parametric" flexibility allows the model to exploit a large parameter space while avoiding over-parametrization pitfalls through geometric constraints.
Experimental Results
The "forward-only" approach was evaluated on the MNIST handwritten digits database using 50,000 training images and 10,000 testing images. The method achieved competitive results across different configurations: plain iterations reached 97.4% accuracy, convolutional iterations reached 98.0%, and a 2-stage learning process achieved 98.7%.
Convergence and Stability
The algorithm demonstrates remarkable convergence speed. For specific step sizes, the model reached 96% accuracy in only 10 iterations. Figure 1 illustrates the accuracy trajectories for various step sizes (μ) and adaptive methods, highlighting the stability of the forward-only approach.
Conclusion and Future Work
This research provides a powerful proof of concept that backpropagation-free training is viable. While the results on MNIST are slightly below state-of-the-art, the geometric framework offers a clearer mathematical foundation for deep learning. Future work will focus on scaling this approach to larger datasets like ImageNet, formalizing the universal approximation properties of modified ResNets, and reducing sensitivity to hyperparameters like momentum and step size.
Summary updated