Deep Pyramidal Residual Networks with Separated Stochastic Depth

Yoshihiro Yamada, Masakazu Iwamura, Koichi KisearXiv 2016 · OpenReview

TL;DR

This paper introduces PyramidSepDrop, a novel deep learning architecture that combines the strengths of PyramidNet and ResDrop to achieve state-of-the-art accuracy on object recognition tasks. By strategically applying stochastic depth to different parts of the residual blocks, PyramidSepDrop effectively mitigates the vanishing gradient problem and improves learning in very deep networks.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (DCNNs) have revolutionized object recognition, with architectures like ResNet breaking accuracy records. However, as networks get deeper to improve their discriminative power, they often suffer from the vanishing gradient problem. This makes training difficult, as gradients become too small to effectively update network weights. While ResNet's residual blocks help, and improvements like ResDrop and PyramidNet offer solutions, their individual effectiveness can be limited, and combining them hasn't always yielded expected gains. This paper aims to bridge this gap by proposing a method that successfully integrates the benefits of stochastic depth (from ResDrop) with the architectural advantages of PyramidNet.

Methods

The core contribution is the PyramidSepDrop architecture. It builds upon PyramidNet, which increases network depth by gradually increasing the number of channels in residual blocks, avoiding the performance degradation sometimes seen with abrupt channel increases in standard ResNet. The paper then integrates ResDrop's concept of stochastic depth. Stochastic depth works by randomly dropping (treating as identity mappings) some residual blocks during training. This effectively makes the network appear shallower, helping to combat the vanishing gradient problem.

What's novel is the "Separated Stochastic Depth" in PyramidSepDrop. Unlike a simple combination, PyramidSepDrop applies stochastic depth independently to different parts of the residual block's function $F(x)$. Specifically, it separates $F(x)$ into an upper part (for increased channels) and a lower part (with the same channels as the input), applying the random drop mechanism to both. This fine-grained control over which parts of the residual computation are regularized is the key innovation.

Datasets

The experiments were conducted on the CIFAR-10 and CIFAR-100 datasets. These are standard benchmarks for image classification tasks, commonly used for evaluating deep learning models.

Key Insights and Ideas

  • Synergy of Architectural Design and Regularization: The paper demonstrates that simply combining existing improvements (like PyramidDrop) isn't always optimal. A more nuanced integration, like the separated stochastic depth in PyramidSepDrop, is crucial for unlocking performance gains.
  • Fine-grained Regularization Matters: Applying stochastic depth to specific components of the residual block (channel-increasing vs. identity-preserving parts) proved more effective than a blanket application. This suggests that understanding the role of different computational paths within a residual block is key to designing better regularization strategies.
  • Deeper is Better, with the Right Tools: The results show that increasing the number of layers in PyramidSepDrop consistently improved accuracy, reinforcing the idea that deeper networks are more powerful, provided that training challenges like vanishing gradients are effectively managed.

Key Results

The proposed PyramidSepDrop achieved a state-of-the-art error rate of 16.18% on CIFAR-100. This is a significant improvement over PyramidNet (18.29%) and competitive with ResNeXt (17.31%). Experiments also showed that increasing the number of layers in PyramidSepDrop generally led to lower error rates. Furthermore, when using multi-model learning, PyramidSepDrop's error rate decreased as the number of models increased, unlike PyramidNet, where it started to increase.

Limitations and Future Work

The paper acknowledges that the initial attempt to combine ResDrop and PyramidNet directly (named PyramidDrop) did not yield the expected accuracy improvements, highlighting the need for the more sophisticated "separated" approach. While the paper focuses on CIFAR-10 and CIFAR-100, the authors suggest that the proposed method could be beneficial for other datasets and tasks. Future work could explore different ways to separate and apply stochastic depth within residual blocks or investigate its application in even larger and more complex network architectures.

Summary updated

Abstract

On general object recognition, Deep Convolutional Neural Networks (DCNNs) achieve high accuracy. In particular, ResNet and its improvements have broken the lowest error rate records. In this paper, we propose a method to successfully combine two ResNet improvements, ResDrop and PyramidNet. We confirmed that the proposed network outperformed the conventional methods; on CIFAR-100, the proposed network achieved an error rate of 16.18% in contrast to PiramidNet achieving that of 18.29% and ResNeXt 17.31%.
BibTeX
@misc{yamada2016deep,
  author = {Yoshihiro Yamada and Masakazu Iwamura and Koichi Kise},
  title = {{Deep Pyramidal Residual Networks with Separated Stochastic Depth}},
  year = {2016},
  eprint = {1612.01230},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.01230},
}

Loading related content...

Figure