TRADI: Tracking Deep Neural Network Weight Distributions

Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, Isabelle Bloch

Abstract

Modern deep neural networks (DNNs) often exhibit overconfidence, particularly when faced with out-of-distribution (OOD) data. While Deep Ensembles provide robust uncertainty estimation, their computational cost is often prohibitive. We present TRADI (TRacking of the weight DIstribution), a method that leverages Kalman Filtering to track the trajectory of weights during the optimization process. By treating stochastic gradient descent (SGD) as a random walk, TRADI captures the mean and variance of weights over time, enabling the sampling of an ensemble at test time with the computational overhead of training only a single model.

Introduction

Deep Neural Networks are frequently deployed in safety-critical environments, yet they struggle to provide reliable uncertainty estimates. Epistemic uncertainty—the uncertainty regarding model parameters—is typically addressed through Deep Ensembles or MC Dropout. However, ensembles are computationally expensive, and MC Dropout can be inconsistent across different architectures.

TRADI addresses these challenges by utilizing the information contained within the optimization trajectory. Rather than focusing solely on the final weight configuration, TRADI monitors the path taken by weights during training to build a distribution that reflects the geometry of the weight space.

Methodology

The core innovation of TRADI lies in its treatment of the optimization process as a linear dynamic system suitable for Kalman Filtering.

Weight Trajectory Tracking

The stochastic nature of mini-batch optimization causes weight updates to resemble a random walk. TRADI tracks this trajectory to estimate the underlying distribution of the weights.

Weight trajectory during optimization on a loss landscape.

Kalman Filtering for Weights

A Kalman Filter is employed to maintain a running estimate of the mean ($\mu$) and variance ($\sigma^2$) for each weight. As new data is observed, the filter updates these statistics, effectively capturing the "oscillations" of the weights during training.

Graphical model and evolution of weight distributions over time.

To handle the high dimensionality of modern networks, TRADI uses a tractable covariance approximation via Random Fourier Features. This reduces the complexity from $O(K^2)$ to $O(K \times N)$, where $K$ is the number of parameters and $N$ is a small number of random features. At inference time, weights are sampled from the resulting distribution to form an ensemble.

Experimental Results

TRADI was evaluated across regression, classification, and OOD detection tasks, including large-scale semantic segmentation for autonomous driving.

Performance and Calibration

On UCI regression benchmarks, TRADI consistently outperformed MC Dropout and Deep Ensembles in terms of RMSE and Log-Likelihood. In OOD detection tasks, such as the StreetHazards dataset, TRADI demonstrated superior calibration, achieving lower Expected Calibration Error (ECE).

Computational Efficiency

A significant advantage of TRADI is its efficiency. In semantic segmentation experiments on the BDD Anomaly dataset, TRADI achieved OOD detection performance comparable to Deep Ensembles while requiring significantly less training time (~21 hours vs. ~189 hours).

Conclusion

TRADI provides a principled, Bayesian-inspired approach to uncertainty estimation by tracking weight distributions during training. While it introduces some memory overhead to store weight statistics, its ability to provide high-quality uncertainty estimates at the cost of a single training run makes it a compelling alternative to traditional ensemble methods. Future research will focus on extending TRADI to transfer learning and refining hyperparameter selection for diverse architectures.

Summary updated

Abstract

During training, the weights of a Deep Neural Network (DNN) are optimized from a random initialization towards a nearly optimum value minimizing a loss function. Only this final state of the weights is typically kept for testing, while the wealth of information on the geometry of the weight space, accumulated over the descent towards the minimum is discarded. In this work we propose to make use of this knowledge and leverage it for computing the distributions of the weights of the DNN. This can be further used for estimating the epistemic uncertainty of the DNN by aggregating predictions from an ensemble of networks sampled from these distributions. To this end we introduce a method for tracking the trajectory of the weights during optimization, that does neither require any change in the architecture, nor in the training procedure. We evaluate our method, TRADI, on standard classification and regression benchmarks, and on out-of-distribution detection for classification and semantic segmentation. We achieve competitive results, while preserving computational efficiency in comparison to ensemble approaches.
BibTeX
@inproceedings{franchi2020tradi,
  author = {Gianni Franchi and Andrei Bursuc and Emanuel Aldea and Séverine Dubuisson and Isabelle Bloch},
  title = {{TRADI: Tracking Deep Neural Network Weight Distributions}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58520-4\_7},
}
Figure