Abstract
This paper proposes a novel approach to train deep neural networks by unlocking the layer-wise dependency of backpropagation training. The approach employs additional modules called local critic networks besides the main network model to be trained, which are used to obtain error gradients without complete feedforward and backward propagation processes. We propose a cascaded learning strategy for these local networks. In addition, the approach is also useful from multi-model perspectives, including structural optimization of neural networks, computationally efficient progressive inference, and ensemble classification for performance improvement. Experimental results show the effectiveness of the proposed approach and suggest guidelines for determining appropriate algorithm parameters.
Introduction
Standard deep learning training is governed by backpropagation (BP), which is inherently sequential and synchronous. This creates a "locking" problem: layers cannot process data until the previous layer finishes (Forward Locking), and weights cannot be updated until the entire forward pass is complete and gradients return from the output layer (Update Locking). This sequential nature hinders the distribution of models across multiple computing nodes. While previous methods like Decoupled Neural Interfaces (DNI) attempted to solve this by synthesizing gradients, they often suffered from significant performance degradation. This paper introduces Local Critic Training to decouple layers without sacrificing accuracy.
Methodology
The proposed method introduces auxiliary modules called Local Critics ($c_i$) attached to specific layer groups ($f_i$) within a main network.
Local Gradient Estimation
Instead of waiting for the global loss from the final output ($h_N$), a local critic takes intermediate features ($h_i$) and predicts the final output. This prediction generates a local loss ($L_i$), providing an immediate gradient for the local layer group. This mechanism allows the group to update its weights independently and asynchronously.
Cascaded Learning
To ensure critics are accurate without re-introducing global locking, a "cascaded" strategy is employed. Each critic $c_i$ is trained to match the training loss of the subsequent critic ($c_{i+1}$). This allows error information to flow backward through the chain of critics asynchronously, maintaining the decoupling of the main network layers.
Applications and Structural Optimization
Local critics effectively transform intermediate layers into functional "mini-classifiers" or sub-models. This enables several advanced training and inference strategies:
- Structural Optimization: Identifying the minimal sub-model that satisfies specific accuracy requirements.
- Progressive Inference: Allowing "easy" samples to exit the network early via a local critic, significantly reducing computational cost.
- Ensemble Inference: Boosting final accuracy by combining predictions from all local critics and the main network.
Experimental Results
The approach was evaluated on CIFAR-10 and CIFAR-100 benchmarks using a VGG-like CNN architecture.
Decoupled Training Performance
Local Critic training successfully trained networks with minimal accuracy loss. On CIFAR-10, it achieved 92.39% accuracy, compared to the BP baseline of 93.93%. A key finding is that sub-models trained jointly with the main model performed significantly better than if trained independently; a small sub-model reached ~85% accuracy via this method, versus ~74% when trained alone.
Efficiency and Ensembles
Using progressive inference, the model processed CIFAR-10 data using only 18.45% of the original FLOPs while maintaining 91.18% accuracy. Furthermore, an ensemble of sub-models and the main model improved performance to 92.79% on CIFAR-10. Memory complexity was also reduced in sub-models, with one example dropping from 7.87 million to 0.88 million parameters while retaining over 90% accuracy.
Conclusion
Local Critic Training effectively breaks the sequential constraints of backpropagation, enabling model-parallel learning and efficient inference. While an accuracy gap remains on more complex datasets like CIFAR-100 (75.14% BP vs. 69.91% Local Critics), the method offers a powerful framework for structural optimization and computational efficiency. Future work will focus on scaling these critics for higher-dimensional tasks and exploring their application in other domains such as Natural Language Processing.
Summary updated