Conflicting Bundles: Adapting Architectures Towards the Improved Training of Deep Neural Networks

David Peer, Sebastian Stabinger, Antonio Rodriguez-SanchezWACV 2021

TL;DR

This paper introduces the concept of "conflicting bundles" – a phenomenon where different inputs produce identical outputs at a specific layer in a neural network, hindering learning. They propose a metric to detect these conflicts early in training and demonstrate that removing layers causing these conflicts can significantly reduce model size and inference time without sacrificing accuracy, even outperforming standard architectures.

What Problem Does This Paper Address?

Designing effective neural network architectures is notoriously difficult, often relying on trial-and-error or extensive grid searches. While many factors influence training success (initialization, activation functions, regularization), the paper focuses on a fundamental problem: identifying which specific layers are detrimental to a model's ability to learn. They observe that some architectures, even when seemingly well-designed, fail to train effectively. This paper aims to provide a theoretical framework and practical tools to pinpoint these problematic layers and offer a solution for creating more efficient and trainable deep neural networks.

Methods

The core contribution is the introduction of the "conflicting bundle" problem. Two input samples are considered "bundled" by a layer if they produce identical outputs after passing through that layer, considering floating-point precision. If these bundled samples have different labels, they form a "conflicting bundle," and the layer causing this is a "conflicting layer."

To detect this, they propose a "bundle entropy" metric ($H^1$). This metric quantifies how mixed the labels are within bundles formed at a specific layer. High bundle entropy indicates a conflicting bundle. They show that this metric can be evaluated early in training, even after just one epoch, by analyzing the deepest network.

Based on this detection, they propose an "auto-tuning" algorithm. This algorithm identifies conflicting layers and prunes them from the network. The process involves training a deep network, detecting conflicting layers, removing them, and then retraining the pruned architecture. This is repeated until no conflicting layers are found.

Datasets

The paper uses the following datasets:

  • Mnist: For evaluating fully connected networks and demonstrating the conflicting bundle boundary.
  • Cifar: Used for evaluating VGG nets and residual networks.
  • Svhn: Used for comparison with ResNet-50.
  • Imagenette: Used for evaluating VGG nets.
  • Toy dataset: A custom dataset with two classes used for controlled experiments to demonstrate the effects of fully conflicting bundles.

Key Insights and Ideas

  • Conflicting Bundles as a Root Cause of Poor Training: The paper posits that "conflicting bundles" are a primary reason why some neural networks fail to train effectively, leading to performance degradation or complete inability to learn.
  • Early Detection is Possible: A key insight is that conflicting bundles can be detected very early in training, even after a single epoch, by analyzing the deepest network. This makes the detection process computationally cheap.
  • Conflicting Bundles vs. Shattered Gradients: The paper distinguishes conflicting bundles from the "shattered gradients" problem, showing that gradients in conflicting bundles are highly correlated with their input, unlike shattered gradients.
  • Residual Connections as a Solution: A significant finding is that residual connections effectively resolve conflicting bundles, explaining why residual networks are easier to train and can be made much deeper.
  • Pruning is Effective: Removing layers that cause conflicting bundles leads to competitive accuracy while drastically reducing model size and inference time, suggesting that these layers are not essential for learning.

Key Results

  • Hypothesis 1 Confirmation: In controlled experiments with a toy dataset, networks trained with fully conflicting bundles failed to learn, with output neurons firing at a constant value (reflecting class imbalance), as predicted.
  • Hypothesis 2 Confirmation: Across various network types (fully connected, VGG, ResNet) and datasets, a negative correlation was observed between bundle entropy and test accuracy, confirming that conflicting bundles worsen performance.
  • Auto-tuning Algorithm Performance: The auto-tuning algorithm, which prunes conflicting layers, achieved competitive test accuracy compared to state-of-the-art ResNets on Imagenette, Cifar, Svhn, and Mnist.
  • Significant Efficiency Gains: The pruned networks achieved substantial reductions in memory consumption and inference time. For example, on Imagenette, an auto-tuned network with 19 layers achieved similar accuracy to a 50-layer ResNet but with significantly less memory and faster inference.
  • Residual Networks Eliminate Conflicts: Experiments showed that residual networks consistently had zero bundle entropy, indicating that residual connections effectively solve the conflicting bundle problem.

Limitations and Future Work

  • Focus on Removal, Not Fixing: The current approach focuses on removing conflicting layers. Future work could explore methods to directly "fix" these layers rather than prune them, potentially using approaches other than residual connections.
  • Floating-Point Precision: The paper acknowledges that the resolution of floating-point representations can influence bundling. Future work could investigate how coarser floating-point resolutions might increase the probability of bundling.
  • Data Augmentation and Sampling: The authors suggest exploring novel data augmentation strategies or intelligent mini-batch sampling as potential methods to avoid conflicting bundles during training.
  • Generalizability of Auto-tuning: While the auto-tuning algorithm is shown to be efficient, further research could explore its application to a wider range of architectures and tasks.

Summary updated

BibTeX
@inproceedings{peer2021conflicting,
  author = {David Peer and Sebastian Stabinger and Antonio Rodriguez-Sanchez},
  title = {{Conflicting Bundles: Adapting Architectures Towards the Improved Training of Deep Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure