Understanding and Exploring the Network with Stochastic Architectures

Zhijie Deng, Yinpeng Dong, Shifeng Zhang, Jun ZhuNeurIPS 2020 · OpenReview

Abstract

This paper decouples the training of networks with stochastic architectures (NSA) from the typical Neural Architecture Search (NAS) pipeline to study them as a standalone tool. By identifying and fixing fundamental issues like Batch Normalization (BN) disparity and "function mode collapse," the authors transform NSA into a powerful framework for model ensembling, uncertainty estimation, and semi-supervised learning.

Introduction: The Standalone Potential of Stochastic Networks

In the realm of Neural Architecture Search (NAS), researchers frequently employ "weight-sharing" to train a single "supernet" from which numerous different architectures can be sampled. Typically, this is merely a means to an end: finding the single best architecture to train from scratch.

However, this paper poses a different question: What if the stochastic network itself is the end goal?

A network capable of "plugging and playing" different architectures at inference time could serve as a powerful regularizer or an efficient ensemble. Historically, this approach has struggled due to two primary hurdles:

  1. Training/Test Disparity: Stochastic networks often exhibit strong performance during training but fail during testing because Batch Normalization (BN) statistics become unstable when architectures change dynamically.
  2. Function Mode Collapse: Because weights are shared across many architectures, the network tends to learn to produce nearly identical outputs regardless of the specific architecture used, effectively defeating the purpose of having an ensemble.
Training curves and histograms illustrating the severe training/test disparity in vanilla NSA.

Methodology: Stabilizing and Diversifying NSA

To address these challenges, the authors conduct a systematic investigation and propose two key improvements to the vanilla NSA framework:

Instance-Specific NSA (NSA-i)

In standard training, a single architecture is sampled for an entire mini-batch. The authors discovered that this creates highly correlated features that disrupt BN statistics. By sampling a different architecture for every single image within a batch, they significantly reduce this correlation. This instance-specific sampling leads to stable BN behavior and substantially higher test accuracy, effectively resolving the training/test disparity.

Architecture-Dependent NSA (NSA-id)

To combat "function mode collapse," the authors introduce a minimal set of architecture-specific parameters—specifically, aggregation coefficients and BN affine parameters. This modification allows the shared weights to remain general while imbuing each architecture with just enough "personality" to produce diverse, complementary predictions, thereby creating a genuine ensemble effect.

Wiring-Based Search Space

Instead of relying on the popular DARTS-style sub-graph search space, the study utilizes a "wiring-based" space inspired by Randomly Wired Neural Networks. In this space, different architectures activate different skip-connections among a fixed set of operations.

Examples of different architectures within the wiring-based search space, showing varied skip-connection activations.

Experimental Setup

The researchers evaluated their proposed approach on standard computer vision benchmarks:

  • CIFAR-10 & CIFAR-100: Utilized for general classification, ensembling, and semi-supervised learning tasks.
  • SVHN (Street View House Numbers): Employed as an "out-of-distribution" (OOD) dataset to rigorously test the model's uncertainty estimation capabilities.

Key Insights

  • The Batch Normalization Bottleneck: The instability of stochastic networks is not a failure of the shared weights, but rather a failure of Batch Normalization. When an entire batch uses a single architecture, the BN mean and variance become biased toward that specific structure. Instance-specific sampling (NSA-i) acts as the crucial fix that aligns training and test performance.
  • Generalization to Unseen Architectures: If an NSA is trained on a sufficiently large variety of architectures (e.g., 500+), it demonstrates remarkable generalization to architectures it has never seen during training. This indicates that the shared weights learn a universal representation compatible with the entire architectural manifold.
  • Architecture as Structured Noise: NSA can be conceptualized as a structured form of "noise," akin to Dropout. However, while Dropout is local and unstructured, architecture stochasticity is global, offering a more principled and effective method for estimating model uncertainty.

Results and Applications

The improved NSA framework demonstrates exceptional performance across several domains:

  • State-of-the-Art Ensembling: On CIFAR-10, an NSA-id ensemble achieved a 2.75% error rate, outperforming strong baselines like Wide ResNets (4.00%) and even some models specifically optimized via NAS.
  • Superior Calibration: The model achieved an Expected Calibration Error (ECE) of 0.0032, significantly lower than standard networks or Monte Carlo Dropout. This indicates highly reliable confidence estimates.
  • Robust Uncertainty Estimation: In OOD detection (distinguishing CIFAR-10 from SVHN), NSA-id achieved an AUC of 0.970, surpassing MC Dropout (0.935).
  • Semi-Supervised Learning: By enforcing predictive consistency between different architectures for the same unlabeled image, NSA-id reached 86.96% accuracy on CIFAR-10 using only 4,000 labels, outperforming the popular $\Pi$-model baseline.

Limitations and Future Work

  • Implementation Complexity: While instance-specific architectures (NSA-i) are highly effective for wiring-based spaces, they present computational challenges in "sub-graph" spaces (like DARTS), where varying architectures may have different memory footprints.
  • Scaling Architecture-Specific Weights: The NSA-id method necessitates storing specific BN parameters for each architecture in the ensemble. While feasible for a small ensemble (e.g., 5-10 architectures), scaling this to thousands of architectures requires a more efficient parameterization strategy.
  • Search Space Sensitivity: The current findings are closely tied to the Erdos-Rényi random wiring space. Future research is needed to determine if these insights hold true for more constrained or specialized search spaces.

Summary updated

Abstract

There is an emerging trend to train a network with stochastic architectures to enable various architectures to be plugged and played during inference. However, the existing investigation is highly entangled with neural architecture search (NAS), limiting its widespread use across scenarios. In this work, we decouple the training of a network with stochastic architectures (NSA) from NAS and provide a first systematical investigation on it as a stand-alone problem. We first uncover the characteristics of NSA in various aspects ranging from training stability, convergence, predictive behaviour, to generalization capacity to unseen architectures. We identify various issues of the vanilla NSA, such as training/test disparity and function mode collapse, and further propose the solutions to these issues with theoretical and empirical insights. We believe that these results could also serve as good heuristics for NAS. Given these understandings, we further apply the NSA with our improvements into diverse scenarios to fully exploit its promise of inference-time architecture stochasticity, including model ensemble, uncertainty estimation and semi-supervised learning. Remarkable performance (e.g., 2.75% error rate and 0.0032 expected calibration error on CIFAR-10) validate the effectiveness of such a model, providing new perspectives of exploring the potential of the network with stochastic architectures, beyond NAS.
BibTeX
@inproceedings{deng2020understanding,
  author = {Zhijie Deng and Yinpeng Dong and Shifeng Zhang and Jun Zhu},
  title = {{Understanding and Exploring the Network with Stochastic Architectures}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=zqtmScdCCqU},
}
Figure