Demystifying Deep Learning: a Geometric Approach to Iterative Projections

Ashkan Panahi, Hamid Krim, Liyi Dai

Abstract

Deep learning's reliance on backpropagation presents significant computational challenges and theoretical obscurities. This paper introduces a "forward-only" training framework that replaces traditional feedback mechanisms with geometric trajectories on function manifolds. By framing learning as a realization of differential equations, the authors establish a formal link between geometric regularization and Residual Networks (ResNets), achieving high accuracy on MNIST without backpropagation.

Introduction

Despite the empirical success of deep learning, it remains a "black box" characterized by training complexity and a lack of theoretical clarity. Standard backpropagation is computationally expensive and prone to vanishing or exploding gradients. Furthermore, providing an exact assessment of a network's expressive power remains an open challenge. This work seeks to "demystify" these processes by replacing feedback-heavy optimization with a feed-forward iterative projection method rooted in differential geometry.

Methodology: Geometric Regularization and Forward Projection

The proposed framework treats supervised learning as the realization of a differential equation rather than the global optimization of a weight matrix.

Geometric Constraints

Instead of unconstrained parameter updates, the model restricts "tangent vectors"—the directions of function evolution—to a specific subspace or "cone." This geometric regularization ensures the model remains structured and prevents wild overfitting. Using the Frobenius Integrability Theorem, the authors demonstrate that if update directions meet certain involutivity conditions, the resulting function is guaranteed to reside on a smooth, low-dimensional manifold.

Iterative Updates

Parameters are updated layer-by-layer during the forward pass. This incremental approach mimics the flow of a differential equation. To enhance practicality, the system incorporates momentum (low-pass filtering) and convolutional structures to handle image data effectively.

Figure 2: Evolution of network architectures from plain iterations (a) to standard ResNets (b) and the proposed modified geometric structure (c).

ResNets as Continuous Limits

A key insight of this research is that modified Residual Network (ResNet) architectures are discrete approximations of continuous differential equations. As shown in the architectural diagrams, the transition from standard layers to residual connections provides a formal geometric justification for the superior performance of ResNets. This "semi-parametric" flexibility allows the model to exploit a large parameter space while avoiding over-parametrization pitfalls through geometric constraints.

Experimental Results

The "forward-only" approach was evaluated on the MNIST handwritten digits database using 50,000 training images and 10,000 testing images. The method achieved competitive results across different configurations: plain iterations reached 97.4% accuracy, convolutional iterations reached 98.0%, and a 2-stage learning process achieved 98.7%.

Convergence and Stability

The algorithm demonstrates remarkable convergence speed. For specific step sizes, the model reached 96% accuracy in only 10 iterations. Figure 1 illustrates the accuracy trajectories for various step sizes (μ) and adaptive methods, highlighting the stability of the forward-only approach.

Figure 1: Accuracy vs. Iteration for different step sizes and training stages, highlighting rapid early-stage convergence and final performance levels.

Conclusion and Future Work

This research provides a powerful proof of concept that backpropagation-free training is viable. While the results on MNIST are slightly below state-of-the-art, the geometric framework offers a clearer mathematical foundation for deep learning. Future work will focus on scaling this approach to larger datasets like ImageNet, formalizing the universal approximation properties of modified ResNets, and reducing sensitivity to hyperparameters like momentum and step size.

Summary updated

BibTeX
@inproceedings{panahi2018demystifying,
  author = {Ashkan Panahi and Hamid Krim and Liyi Dai},
  title = {{Demystifying Deep Learning: a Geometric Approach to Iterative Projections}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8462687},
}
Figure