Abstract
Modern deep neural networks (DNNs) often exhibit overconfidence, particularly when faced with out-of-distribution (OOD) data. While Deep Ensembles provide robust uncertainty estimation, their computational cost is often prohibitive. We present TRADI (TRacking of the weight DIstribution), a method that leverages Kalman Filtering to track the trajectory of weights during the optimization process. By treating stochastic gradient descent (SGD) as a random walk, TRADI captures the mean and variance of weights over time, enabling the sampling of an ensemble at test time with the computational overhead of training only a single model.
Introduction
Deep Neural Networks are frequently deployed in safety-critical environments, yet they struggle to provide reliable uncertainty estimates. Epistemic uncertainty—the uncertainty regarding model parameters—is typically addressed through Deep Ensembles or MC Dropout. However, ensembles are computationally expensive, and MC Dropout can be inconsistent across different architectures.
TRADI addresses these challenges by utilizing the information contained within the optimization trajectory. Rather than focusing solely on the final weight configuration, TRADI monitors the path taken by weights during training to build a distribution that reflects the geometry of the weight space.
Methodology
The core innovation of TRADI lies in its treatment of the optimization process as a linear dynamic system suitable for Kalman Filtering.
Weight Trajectory Tracking
The stochastic nature of mini-batch optimization causes weight updates to resemble a random walk. TRADI tracks this trajectory to estimate the underlying distribution of the weights.
Kalman Filtering for Weights
A Kalman Filter is employed to maintain a running estimate of the mean ($\mu$) and variance ($\sigma^2$) for each weight. As new data is observed, the filter updates these statistics, effectively capturing the "oscillations" of the weights during training.
To handle the high dimensionality of modern networks, TRADI uses a tractable covariance approximation via Random Fourier Features. This reduces the complexity from $O(K^2)$ to $O(K \times N)$, where $K$ is the number of parameters and $N$ is a small number of random features. At inference time, weights are sampled from the resulting distribution to form an ensemble.
Experimental Results
TRADI was evaluated across regression, classification, and OOD detection tasks, including large-scale semantic segmentation for autonomous driving.
Performance and Calibration
On UCI regression benchmarks, TRADI consistently outperformed MC Dropout and Deep Ensembles in terms of RMSE and Log-Likelihood. In OOD detection tasks, such as the StreetHazards dataset, TRADI demonstrated superior calibration, achieving lower Expected Calibration Error (ECE).
Computational Efficiency
A significant advantage of TRADI is its efficiency. In semantic segmentation experiments on the BDD Anomaly dataset, TRADI achieved OOD detection performance comparable to Deep Ensembles while requiring significantly less training time (~21 hours vs. ~189 hours).
Conclusion
TRADI provides a principled, Bayesian-inspired approach to uncertainty estimation by tracking weight distributions during training. While it introduces some memory overhead to store weight statistics, its ability to provide high-quality uncertainty estimates at the cost of a single training run makes it a compelling alternative to traditional ensemble methods. Future research will focus on extending TRADI to transfer learning and refining hyperparameter selection for diverse architectures.
Summary updated