Deep Neural Networks with Short Circuits for Improved Gradient Learning

Ming Yan, Xueli Xiao, Joey Tianyi Zhou, Yi Pan

Abstract

Deep neural networks often suffer from the vanishing gradient problem, where error signals diminish as they propagate backward through many layers, hindering the training of shallow layers. This paper proposes "Short Circuits" (SC), a parameter-free, unidirectional connection mechanism that injects error signals from deep layers directly into shallow ones. Unlike residual connections, SC only affects the backward pass, ensuring that inference remains computationally identical to the baseline model. Empirical results on CIFAR10, MNIST, and various NLP tasks show that SC significantly improves accuracy and convergence speed for architectures like ResNet and BERT.

Introduction

As neural networks increase in depth to capture more complex features, they become progressively harder to optimize. The primary bottleneck is the vanishing gradient problem: the chain rule causes gradients to be multiplied repeatedly, often leading to signals that effectively disappear before reaching early layers. While residual connections (ResNet) mitigate this by providing bidirectional shortcuts, they alter both the forward and backward passes. Other solutions like Neural Architecture Search (NAS) are computationally prohibitive. This work introduces Short Circuits (SC), a "plug-and-play" enhancement that improves gradient learning without changing the model's predictive architecture or increasing inference costs.

The Short Circuit Mechanism

The core innovation of this work is the unidirectional Short Circuit connection. In standard backpropagation, the sensitivity (error signal) of a layer depends on the product of sensitivities from all subsequent layers. SC bypasses this chain by creating a direct path for the error signal from a deep layer to a shallow one.

Mechanism of Short Circuit (SC) in backpropagation compared to standard feedforward and backpropagation.

As shown in Figure 1, the forward pass remains unchanged, preserving the original model's inference efficiency. During the backward pass, the SC forces the chain-rule multiplier across skipped layers to be approximately 1, ensuring shallow layers receive a robust training signal. This approach is inspired by gradient truncation in Recurrent Neural Networks (RNNs), adapted here to prevent vanishing gradients across spatial layers in feedforward networks.

Experimental Setup

The effectiveness of SC was evaluated across diverse domains:

  • Computer Vision: Experiments were conducted on CIFAR10 and MNIST datasets.
  • Natural Language Processing: The researchers tested SC on Multiple Choice Question Answering (MCQA) tasks using the DREAM (dialogue-based) and SemEval-2018 Task 11 (narrative-based) datasets.

Key Results and Discussion

The addition of Short Circuits led to substantial performance gains across all tested architectures. A notable result was observed with ResNet-1202 on CIFAR10, where accuracy improved from 92.07% to 94.46%. This suggests that SC helps overcome the "depth limit" where standard ResNets typically begin to degrade.

Training loss curves comparing standard ResNet and SCResNet architectures across different depths.

As illustrated in Figure 5, SCResNet models achieve lower loss levels significantly faster than their baseline counterparts. In the NLP domain, SCBERT and SCRoberta outperformed state-of-the-art models like XLNet. For instance, SCRoberta achieved 86.3% accuracy on the DREAM dataset. Furthermore, gradient analysis confirmed that SC maintains a higher mean gradient in early layers, preventing the signal from hitting zero as it does in standard fully connected networks.

Conclusion

Short Circuits provide a simple yet effective way to improve the training of deep neural networks. By decoupling the training enhancement from the inference architecture, SC allows for better-performing models without any additional deployment cost. While "one-to-multi" connections can increase complexity in shallower models, the benefits for very deep architectures are clear. Future research will explore SC application to larger datasets like ImageNet and optimize layer-connection strategies.

Summary updated

BibTeX
@misc{yan2020deep,
  author = {Ming Yan and Xueli Xiao and Joey Tianyi Zhou and Yi Pan},
  title = {{Deep Neural Networks with Short Circuits for Improved Gradient Learning}},
  year = {2020},
  eprint = {2009.11719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.11719},
}
Figure