Abstract
Training deep neural networks typically relies on iterative Back-Propagation (BP), which is computationally expensive and often lacks theoretical transparency. This paper introduces ACnnL (Analytic Convolutional neural network Learning), a novel framework that enables training CNNs in a single pass using closed-form matrix algebra. By reformulating convolution as a linear system and employing label encoding for layer-wise targets, ACnnL achieves training speeds up to 113x faster than BP while maintaining competitive accuracy on standard benchmarks. Furthermore, the method provides a mathematical explanation for the superior generalization of CNNs compared to dense networks.
Introduction
The dominance of deep learning in pattern recognition is largely built upon Back-Propagation (BP). Despite its success, BP is an iterative "black box" process prone to vanishing gradients, sensitivity to initialization, and high resource consumption. Researchers have long sought "analytic" alternatives—methods that solve for weights directly—but applying this to the sparse, weight-sharing structure of CNNs has remained a significant challenge.
The ACnnL Framework
The proposed ACnnL method replaces the iterative gradient descent of BP with a direct, least-squares solution. The framework is built on three primary technical innovations:
1. Matrix Reformulation
To solve for weights analytically, the 3D convolution operation must be expressed as a standard linear equation. ACnnL converts the convolution into a 2D matrix multiplication, effectively flattening the local receptive fields into a feature matrix.
2. Label Encoding and Layer-wise Solving
Unlike BP, which propagates errors backward, ACnnL trains the network forward, layer by layer. Each layer requires a target to learn against. The authors utilize label encoding to project the final class labels into the hidden feature space of each layer. This provides "pseudo-labels" that guide the analytic solver at every step of the hierarchy.
3. Least Squares Solution
With inputs and targets defined for each layer, the optimal weights are calculated using the Moore-Penrose pseudoinverse. This finds the mathematically "best fit" weights in a single step, eliminating the need for hundreds of training epochs.
Experimental Results
The researchers evaluated ACnnL against standard BP (using the Adam optimizer) on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.
- Speed: ACnnL demonstrated massive efficiency gains. On a CPU, it trained a 5-layer CNN 113x faster than BP. On a GPU, the speedup was 17x.
- Accuracy: On MNIST, ACnnL achieved 99.31% accuracy, nearly matching BP's 99.43%. On the more complex CIFAR-10, it reached 70.49%, outperforming other non-BP methods like Deep Forest.
- Data Efficiency: ACnnL proved remarkably stable in small-sample regimes. When training data was limited to fewer than 1,000 samples, ACnnL often outperformed BP, which tends to overfit in such scenarios.
Theoretical Insights: Why CNNs Generalize
A key contribution of this work is the theoretical bridge between CNN architecture and linear regression. The authors demonstrate that a CNN is essentially a Multilayer Perceptron (MLP) subject to a vast number of localized linear constraints. This inherent structural regularization is what allows CNNs to generalize effectively, even when trained with a simple least-squares objective.
Conclusion and Future Work
ACnnL represents a significant step toward making deep learning more efficient and mathematically interpretable. While it currently lags behind BP on very high-dimensional datasets like CIFAR-100, its speed and stability make it an ideal candidate for rapid prototyping or as a sophisticated weight initialization technique. Future research will explore extending this analytic approach to modern architectures like ResNets and incorporating advanced regularization techniques.
Summary updated