Intra-Model Collaborative Learning of Neural Networks

Shijie Fang, Tong LinIJCNN 2021 · DOI

TL;DR

This paper introduces a novel "intra-model" collaborative learning framework that enhances neural network performance without increasing inference cost or model size. By enabling different parts of a single network to learn from each other, the authors achieve significant improvements in accuracy and robustness, particularly on noisy datasets, while drastically reducing the memory overhead compared to previous collaborative learning methods.

What Problem Does This Paper Address?

Deep neural networks have achieved remarkable success, but improving their performance often comes at a cost: either by ensembling multiple networks (increasing inference cost) or by using larger, more complex models (increasing memory footprint). Collaborative learning, as proposed by Song and Chai [4], offers a way to boost generalization by training multiple "subnets" within a single network. However, their approach typically involves duplicating network modules, leading to a substantial increase in memory usage during training, which is prohibitive for large-capacity models on memory-constrained hardware like GPUs. This paper tackles the challenge of achieving the benefits of collaborative learning with minimal memory overhead, by exploring how different components within a single network can collaborate.

Methods

The core contribution is a new framework for "intra-model" collaborative learning, which proposes four distinct ways for different parts of a single neural network to learn from each other:

  • Collaboration of Output Layer: Instead of duplicating network paths, this method leverages the widely used dropout technique. By randomly dropping units, the network is effectively split into multiple "thinner" sub-networks. The consistency of outputs from these dropout-induced sub-networks is regularized to improve robustness. This is a more memory-efficient alternative to the multi-instance or tree structures used previously.
  • Collaboration of Multiple Intermediate Layers: This approach introduces local classifiers at intermediate layers. These classifiers make predictions using the feature maps from that layer. The framework then regularizes the consistency between these local predictions and the ground-truth labels, as well as between the predictions of different intermediate layers. This helps propagate error signals more effectively and mitigates the vanishing gradient problem.
  • Collaboration of Intermediate Representation with Inputs and Targets: This method focuses on the relationship between intermediate layer representations and both the input data and the target labels. It uses cosine similarity to measure the similarity of feature maps within a mini-batch. The objective is to "pull" representations of data with the same label closer together and "push" them away from representations of input data, encouraging better feature extraction.
  • Collaboration Inside One Convolution Layer: This technique targets redundancy within a single convolution layer. It proposes regularizing the covariance matrix of the convolution kernel weights. The intuition is that by controlling the covariance, the redundancy in feature maps can be reduced, leading to more efficient representations and mitigating overfitting.

Datasets

The proposed methods were evaluated on several standard image classification datasets:

  • CIFAR-10: A dataset with 10 classes.
  • CIFAR-100: A dataset with 100 classes.
  • ImageNet32: A downsampled version of ImageNet, with images resized to 32x32 pixels.
  • STL-10: A dataset with 10 object classes.

Experiments on CIFAR-100 were also conducted with noisy labels to assess the robustness of the proposed methods.

Key Insights and Ideas

  • Collaborative learning can happen within a single network: The paper demonstrates that the benefits of collaborative learning are not limited to multi-head or multi-instance architectures. By cleverly utilizing techniques like dropout and intermediate layer supervision, a single network can achieve consensus among its own components.
  • Dropout as a memory-efficient collaborative tool: The use of dropout for output layer collaboration is a particularly elegant solution, offering significant memory savings compared to duplicating network paths.
  • Intermediate layers are crucial for collaboration: The paper highlights the importance of involving intermediate layers in the collaborative learning process, not just the output layer, to improve gradient flow and feature learning.
  • Robustness to noisy labels: The proposed methods show a marked improvement in performance when trained on datasets with corrupted labels, suggesting that collaborative learning can inherently enhance model generalization and resilience.
  • Combinations are powerful, but not always additive: While combining different collaboration strategies generally leads to better results, the paper shows that not all combinations are equally beneficial, and some might even degrade performance (e.g., adding Lkernel to VGG-16). This suggests a need for careful tuning and selection of collaboration strategies.

Key Results

  • Significant accuracy improvements: The proposed methods consistently reduced test errors across all tested datasets. For example, on STL-10, test errors were decreased by 9.28% for ResNet-18 and 5.45% for VGG-16 when using the best combinations.
  • Memory efficiency: The hierarchical dropout method for output layer collaboration resulted in a memory footprint almost identical to the original network, drastically reducing the overhead compared to previous methods (e.g., a ResNet-101 with hierarchical dropout used 8.78GB, while a tree-structure version used 13.16GB).
  • Robustness to label noise: On CIFAR-100 with 50% noisy labels, the Lout + Lmid + Lpull-push combination offered an improvement of 3.53% over the baseline, demonstrating enhanced tolerance to label corruption.
  • Competitive performance with fewer filters: The Lmid method alone achieved competitive accuracy with a baseline model having half the number of filters, indicating improved efficiency.

Limitations and Future Work

  • Optimal combination tuning: The paper notes that the best performance is achieved through specific combinations of the proposed methods, and that simply stacking all strategies is not always optimal. Finding these optimal combinations might require careful grid-search or more advanced hyperparameter optimization.
  • Future Directions: The authors plan to extend collaborative learning to other fields such as semantic segmentation, object detection, and person re-identification. They also suggest exploring selective combinations of their proposed methods for single deployment.

Summary updated

BibTeX
@inproceedings{fang2021intra,
  author = {Shijie Fang and Tong Lin},
  title = {{Intra-Model Collaborative Learning of Neural Networks}},
  booktitle = {IJCNN},
  year = {2021},
  doi = {10.1109/ijcnn52387.2021.9533324},
}
Figure