Stochastic Neural Network with Kronecker Flow.

Chin-Wei Huang, Ahmed Touati, Pascal Vincent, Gintare Karolina Dziugaite, Alexandre Lacoste, Aaron C. Courville

Abstract

Stochastic Neural Networks (SNNs) offer a robust framework for uncertainty estimation by treating weights as probability distributions. However, modeling complex dependencies between millions of parameters remains a significant computational hurdle. This paper introduces Kronecker Flow, a scalable method that generalizes the linear Kronecker product into a non-linear normalizing flow. By applying invertible mappings row-wise and column-wise to weight matrices, Kronecker Flow captures intricate parameter correlations without the memory explosion associated with full-covariance models. Experimental results across image classification and contextual bandit tasks demonstrate that Kronecker Flow provides superior uncertainty calibration and significantly tighter theoretical generalization bounds.

Introduction: The Scalability Challenge in SNNs

Stochastic Neural Networks are essential for safety-sensitive applications like autonomous driving, where knowing when a model "doesn't know" is critical. The primary challenge in SNNs is the trade-off between simplicity and expressivity. Standard "mean-field" approaches assume weight independence, which is computationally simple but often leads to poorly calibrated uncertainty. Conversely, modeling full correlations between millions of weights is mathematically impossible on current hardware. While the linear Kronecker product has been used as a middle ground, it is often too restrictive to capture the true complexity of a neural network's parameter space.

Methodology: Kronecker Flow

The authors propose Kronecker Flow to bridge the gap between independent noise and expensive full-covariance models through three key innovations:

  1. Normalizing Flows: The method utilizes sequences of invertible mappings to transform a simple distribution (like a Gaussian) into a highly complex, expressive one.
  2. The Kronecker Trick: To ensure scalability, these invertible mappings are applied row-wise and then column-wise to weight matrices. This structured approach allows the model to learn non-linear dependencies between parameters efficiently.
  3. PAC-Bayes Concentration: Since normalizing flows often lack simple analytical formulas for KL divergence, the authors derived a new mathematical tail bound to ensure that empirical estimates of these bounds are reliable and do not underestimate the risk of overfitting.

Experimental Setup

The method was evaluated on several standard machine learning benchmarks:

  • Image Classification: MNIST (handwritten digits) and CIFAR-10 (natural images).
  • Contextual Bandits: Six real-world datasets (e.g., Mushroom, Financial, Adult) used to test the model's ability to handle the exploration-exploitation tradeoff.
  • Generalization Benchmarks: Specific setups for calculating PAC-Bayes bounds on MNIST using Multi-Layer Perceptrons (MLPs) and LeNet-5.

Key Insights: Beyond Linearity

A central finding of the research is that standard linear Kronecker products are fundamentally limited; they cannot even represent all possible diagonal matrices. By moving to non-linear flows, the authors unlock significantly higher expressivity.

KL divergence comparison for different weight matrix and tensor shapes, demonstrating the increased expressivity of Kronecker-structured flows.

Capturing this non-linear structure in parameter space directly leads to tighter PAC-Bayes generalization bounds. This suggests that the "generalization gap" between training and test performance is partly hidden in the complex correlations between weights. Furthermore, Kronecker Flow is efficient enough to scale to larger architectures like VGG-16, unlike many other advanced Bayesian methods.

Key Results

  • Predictive Accuracy: On MNIST, the "K-Nonlinear" (Kronecker Flow) model achieved a test error of 0.60%, outperforming standard factorized Gaussians (0.90%) and even more complex multiplicative flows (0.70%).
  • Tighter Generalization Bounds: In PAC-Bayes experiments, Kronecker Flow produced significantly lower (tighter) generalization bounds than previous methods. For a LeNet-5 model, it reduced the complexity term (KL) significantly while maintaining low empirical risk.
Comparison of PAC-Bayes bounds and KL terms, showing that Kronecker-Nonlinear (K-N) flows achieve tighter bounds and lower complexity across different architectures.
  • Decision Making: In contextual bandit tasks, Kronecker Flow proved highly competitive, often achieving the lowest cumulative regret on datasets like 'Covertype' and 'Financial,' proving that its uncertainty estimates are useful for real-world decision-making.

Discussion and Future Work

While powerful, Kronecker Flow faces some practical hurdles. Certain parameterizations (like K-Diag) can be unstable during optimization and may diverge on specific datasets. Additionally, the theoretical guarantees rely on the Lipschitz smoothness of the flow, which adds a layer of mathematical constraint to model design. Future research may explore even more complex "triangular maps" to increase connectivity between parameters and apply these techniques to large-scale reinforcement learning problems.

Summary updated

Abstract

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and better than the baselines.
BibTeX
@inproceedings{huang2020stochastic,
  author = {Chin-Wei Huang and Ahmed Touati and Pascal Vincent and Gintare Karolina Dziugaite and Alexandre Lacoste and Aaron C. Courville},
  title = {{Stochastic Neural Network with Kronecker Flow.}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=6FeL2WvBGZs},
}
Figure