Abstract
Stochastic Neural Networks (SNNs) are essential for robust uncertainty estimation and deriving theoretical generalization bounds. However, modeling complex correlations between millions of parameters is computationally daunting, often leading researchers to rely on overly simplistic "mean-field" independence assumptions. This paper introduces Kronecker Flow, a scalable framework that generalizes the Kronecker product into a non-linear normalizing flow. By capturing intricate dependencies within weight matrices while maintaining computational tractability, Kronecker Flow provides better-calibrated uncertainty, tighter PAC-Bayes generalization bounds, and improved performance in decision-making tasks.
Introduction
Modern deep learning models often suffer from overconfidence and poor calibration because they treat weights as fixed point estimates. Stochastic Neural Networks address this by treating weights as random variables. The primary challenge lies in the covariance structure: a network with $N$ parameters requires an $N \times N$ covariance matrix. For large models, this is trillions of entries.
Existing solutions typically assume parameter independence (diagonal covariance), which fails to capture the structural correlations inherent in neural network layers. Kronecker Flow addresses this gap by providing a middle ground: an expressive yet efficient way to model parameter dependencies by transforming the rows and columns of weight matrices.
Methodology: Kronecker Flow
The authors propose two primary variants of their method:
- Linear Kronecker Flow: This builds on the matrix-variate normal distribution. Parameters are correlated by multiplying the weight matrix from the left and right by smaller learnable matrices. While efficient, its linear nature limits expressivity.
- Nonlinear Kronecker Flow: This replaces linear multiplications with Normalizing Flows—sequences of invertible, non-linear mappings (e.g., RealNVP). This allows the model to capture complex, non-linear dependencies between weights.
To ensure these flows can be used for theoretical analysis, the authors derived a new PAC-Bayes concentration proof. Since the KL divergence cannot always be calculated analytically for non-linear flows, they provide a high-probability bound based on the Lipschitz properties of the transformation.
Empirical Evaluation
The method was evaluated across image classification (MNIST, CIFAR-10), generalization theory, and contextual bandits.
Capturing Parameter Structure
A key motivation for Kronecker Flow is reducing the information gap (KL divergence) between the approximate and true posterior. As shown in the figure below, methods that account for Kronecker structure (K-Linear) consistently achieve lower KL divergence across various matrix and tensor shapes compared to standard diagonal (Diag) or K-Diag approaches.
Performance in Contextual Bandits
In decision-making scenarios where exploration is driven by uncertainty, Kronecker Flow demonstrates significant advantages. On six standard benchmarks, Kronecker Flow variants (K-Linear and K-Nonlinear) were competitive with or outperformed state-of-the-art methods like Functional Bayesian Neural Networks (fBNN) and standard SGD, achieving the best performance in 4 out of 6 tasks.
Key Insights and Results
- Breaking the Independence Bottleneck: The results confirm that the "mean-field" assumption is a major performance bottleneck. By modeling row and column dependencies, the network learns a more realistic distribution of weights.
- Tighter Generalization Bounds: In PAC-Bayes experiments, Kronecker Flow produced significantly tighter generalization bounds than standard Gaussian methods. This suggests that the learned weight structure is more stable and theoretically sound.
- Scalable Expressivity: On a VGG-16 architecture, Kronecker Flow achieved a 9.88% error rate on CIFAR-10, outperforming the strong "Noisy K-FAC" baseline (10.65%).
Conclusion
Kronecker Flow offers a powerful alternative to independent weight noise in SNNs. By leveraging the efficiency of Kronecker structures and the expressivity of normalizing flows, it bridges the gap between simple diagonal approximations and intractable full-covariance models. Future work includes extending these structured distributions to convolutional kernels and complex reinforcement learning agents.
Summary updated