Stochastic Neural Network with Kronecker Flow

Chin-Wei Huang, Ahmed Touati, Pascal Vincent, Gintare Karolina Dziugaite, Alexandre Lacoste, Aaron C. Courville

Abstract

Stochastic Neural Networks (SNNs) are essential for robust uncertainty estimation and deriving theoretical generalization bounds. However, modeling complex correlations between millions of parameters is computationally daunting, often leading researchers to rely on overly simplistic "mean-field" independence assumptions. This paper introduces Kronecker Flow, a scalable framework that generalizes the Kronecker product into a non-linear normalizing flow. By capturing intricate dependencies within weight matrices while maintaining computational tractability, Kronecker Flow provides better-calibrated uncertainty, tighter PAC-Bayes generalization bounds, and improved performance in decision-making tasks.

Introduction

Modern deep learning models often suffer from overconfidence and poor calibration because they treat weights as fixed point estimates. Stochastic Neural Networks address this by treating weights as random variables. The primary challenge lies in the covariance structure: a network with $N$ parameters requires an $N \times N$ covariance matrix. For large models, this is trillions of entries.

Existing solutions typically assume parameter independence (diagonal covariance), which fails to capture the structural correlations inherent in neural network layers. Kronecker Flow addresses this gap by providing a middle ground: an expressive yet efficient way to model parameter dependencies by transforming the rows and columns of weight matrices.

Methodology: Kronecker Flow

The authors propose two primary variants of their method:

  1. Linear Kronecker Flow: This builds on the matrix-variate normal distribution. Parameters are correlated by multiplying the weight matrix from the left and right by smaller learnable matrices. While efficient, its linear nature limits expressivity.
  2. Nonlinear Kronecker Flow: This replaces linear multiplications with Normalizing Flows—sequences of invertible, non-linear mappings (e.g., RealNVP). This allows the model to capture complex, non-linear dependencies between weights.

To ensure these flows can be used for theoretical analysis, the authors derived a new PAC-Bayes concentration proof. Since the KL divergence cannot always be calculated analytically for non-linear flows, they provide a high-probability bound based on the Lipschitz properties of the transformation.

Empirical Evaluation

The method was evaluated across image classification (MNIST, CIFAR-10), generalization theory, and contextual bandits.

Capturing Parameter Structure

A key motivation for Kronecker Flow is reducing the information gap (KL divergence) between the approximate and true posterior. As shown in the figure below, methods that account for Kronecker structure (K-Linear) consistently achieve lower KL divergence across various matrix and tensor shapes compared to standard diagonal (Diag) or K-Diag approaches.

KL divergence comparison for random Gaussian matrices and 3D tensors across different shapes.

Performance in Contextual Bandits

In decision-making scenarios where exploration is driven by uncertainty, Kronecker Flow demonstrates significant advantages. On six standard benchmarks, Kronecker Flow variants (K-Linear and K-Nonlinear) were competitive with or outperformed state-of-the-art methods like Functional Bayesian Neural Networks (fBNN) and standard SGD, achieving the best performance in 4 out of 6 tasks.

Performance comparison on contextual bandit benchmarks. Bold values indicate the best performance.

Key Insights and Results

  • Breaking the Independence Bottleneck: The results confirm that the "mean-field" assumption is a major performance bottleneck. By modeling row and column dependencies, the network learns a more realistic distribution of weights.
  • Tighter Generalization Bounds: In PAC-Bayes experiments, Kronecker Flow produced significantly tighter generalization bounds than standard Gaussian methods. This suggests that the learned weight structure is more stable and theoretically sound.
  • Scalable Expressivity: On a VGG-16 architecture, Kronecker Flow achieved a 9.88% error rate on CIFAR-10, outperforming the strong "Noisy K-FAC" baseline (10.65%).

Conclusion

Kronecker Flow offers a powerful alternative to independent weight noise in SNNs. By leveraging the efficiency of Kronecker structures and the expressivity of normalizing flows, it bridges the gap between simple diagonal approximations and intractable full-covariance models. Future work includes extending these structured distributions to convolutional kernels and complex reinforcement learning agents.

Summary updated

Abstract

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and better than the baselines.
BibTeX
@inproceedings{huang2020stochastic,
  author = {Chin-Wei Huang and Ahmed Touati and Pascal Vincent and Gintare Karolina Dziugaite and Alexandre Lacoste and Aaron C. Courville},
  title = {{Stochastic Neural Network with Kronecker Flow}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=6FeL2WvBGZs},
}
Figure