Analytic Learning of Convolutional Neural Network For Pattern Recognition

Huiping Zhuang, Zhiping Lin, Yimin Yang, Kar-Ann ToharXiv 2022 · arXiv

Abstract

Training deep neural networks typically relies on iterative Back-Propagation (BP), which is computationally expensive and often lacks theoretical transparency. This paper introduces ACnnL (Analytic Convolutional neural network Learning), a novel framework that enables training CNNs in a single pass using closed-form matrix algebra. By reformulating convolution as a linear system and employing label encoding for layer-wise targets, ACnnL achieves training speeds up to 113x faster than BP while maintaining competitive accuracy on standard benchmarks. Furthermore, the method provides a mathematical explanation for the superior generalization of CNNs compared to dense networks.

Introduction

The dominance of deep learning in pattern recognition is largely built upon Back-Propagation (BP). Despite its success, BP is an iterative "black box" process prone to vanishing gradients, sensitivity to initialization, and high resource consumption. Researchers have long sought "analytic" alternatives—methods that solve for weights directly—but applying this to the sparse, weight-sharing structure of CNNs has remained a significant challenge.

The ACnnL Framework

The proposed ACnnL method replaces the iterative gradient descent of BP with a direct, least-squares solution. The framework is built on three primary technical innovations:

1. Matrix Reformulation

To solve for weights analytically, the 3D convolution operation must be expressed as a standard linear equation. ACnnL converts the convolution into a 2D matrix multiplication, effectively flattening the local receptive fields into a feature matrix.

Figure 1: Reformulation of 3D convolution (a) into a 2D matrix multiplication (b), allowing for direct weight estimation.

2. Label Encoding and Layer-wise Solving

Unlike BP, which propagates errors backward, ACnnL trains the network forward, layer by layer. Each layer requires a target to learn against. The authors utilize label encoding to project the final class labels into the hidden feature space of each layer. This provides "pseudo-labels" that guide the analytic solver at every step of the hierarchy.

Figure 2: The complete ACnnL pipeline, showing the transition from input image to feature transformation, label encoding, and the analytic solution for CNN weights.

3. Least Squares Solution

With inputs and targets defined for each layer, the optimal weights are calculated using the Moore-Penrose pseudoinverse. This finds the mathematically "best fit" weights in a single step, eliminating the need for hundreds of training epochs.

Experimental Results

The researchers evaluated ACnnL against standard BP (using the Adam optimizer) on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.

  • Speed: ACnnL demonstrated massive efficiency gains. On a CPU, it trained a 5-layer CNN 113x faster than BP. On a GPU, the speedup was 17x.
  • Accuracy: On MNIST, ACnnL achieved 99.31% accuracy, nearly matching BP's 99.43%. On the more complex CIFAR-10, it reached 70.49%, outperforming other non-BP methods like Deep Forest.
  • Data Efficiency: ACnnL proved remarkably stable in small-sample regimes. When training data was limited to fewer than 1,000 samples, ACnnL often outperformed BP, which tends to overfit in such scenarios.

Theoretical Insights: Why CNNs Generalize

A key contribution of this work is the theoretical bridge between CNN architecture and linear regression. The authors demonstrate that a CNN is essentially a Multilayer Perceptron (MLP) subject to a vast number of localized linear constraints. This inherent structural regularization is what allows CNNs to generalize effectively, even when trained with a simple least-squares objective.

Conclusion and Future Work

ACnnL represents a significant step toward making deep learning more efficient and mathematically interpretable. While it currently lags behind BP on very high-dimensional datasets like CIFAR-100, its speed and stability make it an ideal candidate for rapid prototyping or as a sophisticated weight initialization technique. Future research will explore extending this analytic approach to modern architectures like ResNets and incorporating advanced regularization techniques.

Summary updated

Abstract

Training convolutional neural networks (CNNs) with back-propagation (BP) is time-consuming and resource-intensive particularly in view of the need to visit the dataset multiple times. In contrast, analytic learning attempts to obtain the weights in one epoch. However, existing attempts to analytic learning considered only the multilayer perceptron (MLP). In this article, we propose an analytic convolutional neural network learning (ACnnL). Theoretically we show that ACnnL builds a closed-form solution similar to its MLP counterpart, but differs in their regularization constraints. Consequently, we are able to answer to a certain extent why CNNs usually generalize better than MLPs from the implicit regularization point of view. The ACnnL is validated by conducting classification tasks on several benchmark datasets. It is encouraging that the ACnnL trains CNNs in a significantly fast manner with reasonably close prediction accuracies to those using BP. Moreover, our experiments disclose a unique advantage of ACnnL under the small-sample scenario when training data are scarce or expensive.
BibTeX
@misc{zhuang2022analytic,
  author = {Huiping Zhuang and Zhiping Lin and Yimin Yang and Kar-Ann Toh},
  title = {{Analytic Learning of Convolutional Neural Network For Pattern Recognition}},
  year = {2022},
  eprint = {2202.06504},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.06504},
}
Figure