Abstract
Direct Loss Minimization (DLM) has been proposed as a pseudo-Bayesian method motivated by regularized loss minimization. By replacing the standard Evidence Lower Bound (ELBO) loss term with the predictive log loss—the same metric used for evaluation—DLM theoretically aligns training more closely with performance goals. While successful in simpler models like Gaussian Processes, this paper demonstrates that DLM consistently fails to outperform ELBO in the context of Bayesian Neural Networks (BNNs). Surprisingly, ELBO optimization often minimizes the DLM objective more effectively than the DLM algorithm itself.
The Objective Mismatch in BNNs
In probabilistic machine learning, a fundamental mismatch often exists between training and evaluation. Bayesian models are typically trained using Variational Inference (VI) to maximize the Evidence Lower Bound (ELBO). However, at test time, performance is measured by the Negative Log-Loss (NLL) of the predictive distribution.
Direct Loss Minimization (DLM) was designed to bridge this gap. Instead of a surrogate like the ELBO, DLM optimizes the predictive log-loss directly during training, utilizing the Kullback–Leibler (KL) divergence solely as a regularizer. Despite theoretical bounds suggesting DLM should be superior, its practical application to high-dimensional BNNs reveals significant optimization challenges.
Empirical Performance Gap
The authors evaluated DLM and ELBO across standard image classification benchmarks, including CIFAR-10, CIFAR-100, STL-10, and SVHN, using AlexNet and PreResNet20 architectures. The study utilized a mean-field diagonal Gaussian variational distribution and Monte Carlo sampling to estimate expectations.
Across every dataset and architecture tested, DLM resulted in higher (worse) test NLL compared to ELBO. As shown in Figure 1, the difference in test loss (DLM minus ELBO) was consistently positive, indicating a significant performance deficit for the DLM approach.
The "Better Optimizer" Paradox
One of the most startling findings of the research is that ELBO optimization is actually a better way to minimize the DLM objective than the DLM algorithm itself. In path analysis and trajectory tracking, the authors observed that the ELBO objective is much better aligned with test set performance.
As illustrated in the trajectories above, the ELBO-trained model (blue line) consistently achieves a lower DLM loss than the model trained directly on the DLM objective (orange line). Furthermore, when DLM was initialized with a "good" solution from an ELBO-trained model, performance actually degraded, suggesting that the DLM objective is either inherently difficult to optimize or highly prone to overfitting in the high-dimensional landscape of neural networks.
Conclusion
The failure of DLM in BNNs highlights a disconnect between theoretical risk bounds and practical optimization. While DLM offers tighter theoretical guarantees, the ELBO acts as a "better-behaved" surrogate, providing smoother gradients and better generalization for deep learning optimizers. Future work is required to understand why the success of DLM in sparse Gaussian Processes does not translate to the complex, non-convex landscapes of deep Bayesian models.
Summary updated