On the Performance of Direct Loss Minimization for Bayesian Neural Networks

Yadi Wei, Roni Khardon

Abstract

Direct Loss Minimization (DLM) has been proposed as a pseudo-Bayesian method motivated by regularized loss minimization. By replacing the standard Evidence Lower Bound (ELBO) loss term with the predictive log loss—the same metric used for evaluation—DLM theoretically aligns training more closely with performance goals. While successful in simpler models like Gaussian Processes, this paper demonstrates that DLM consistently fails to outperform ELBO in the context of Bayesian Neural Networks (BNNs). Surprisingly, ELBO optimization often minimizes the DLM objective more effectively than the DLM algorithm itself.

The Objective Mismatch in BNNs

In probabilistic machine learning, a fundamental mismatch often exists between training and evaluation. Bayesian models are typically trained using Variational Inference (VI) to maximize the Evidence Lower Bound (ELBO). However, at test time, performance is measured by the Negative Log-Loss (NLL) of the predictive distribution.

Direct Loss Minimization (DLM) was designed to bridge this gap. Instead of a surrogate like the ELBO, DLM optimizes the predictive log-loss directly during training, utilizing the Kullback–Leibler (KL) divergence solely as a regularizer. Despite theoretical bounds suggesting DLM should be superior, its practical application to high-dimensional BNNs reveals significant optimization challenges.

Empirical Performance Gap

The authors evaluated DLM and ELBO across standard image classification benchmarks, including CIFAR-10, CIFAR-100, STL-10, and SVHN, using AlexNet and PreResNet20 architectures. The study utilized a mean-field diagonal Gaussian variational distribution and Monte Carlo sampling to estimate expectations.

Across every dataset and architecture tested, DLM resulted in higher (worse) test NLL compared to ELBO. As shown in Figure 1, the difference in test loss (DLM minus ELBO) was consistently positive, indicating a significant performance deficit for the DLM approach.

Difference in test Negative Log-Loss (NLL) between DLM and ELBO across various datasets and architectures. Positive values indicate DLM underperforms compared to ELBO.

The "Better Optimizer" Paradox

One of the most startling findings of the research is that ELBO optimization is actually a better way to minimize the DLM objective than the DLM algorithm itself. In path analysis and trajectory tracking, the authors observed that the ELBO objective is much better aligned with test set performance.

Training trajectories for CIFAR-10 on AlexNet showing (a) test loss, (b) DLM objective, and (c) ELBO objective. ELBO optimization (blue) consistently achieves lower DLM loss than the DLM algorithm itself (orange).

As illustrated in the trajectories above, the ELBO-trained model (blue line) consistently achieves a lower DLM loss than the model trained directly on the DLM objective (orange line). Furthermore, when DLM was initialized with a "good" solution from an ELBO-trained model, performance actually degraded, suggesting that the DLM objective is either inherently difficult to optimize or highly prone to overfitting in the high-dimensional landscape of neural networks.

Conclusion

The failure of DLM in BNNs highlights a disconnect between theoretical risk bounds and practical optimization. While DLM offers tighter theoretical guarantees, the ELBO acts as a "better-behaved" surrogate, providing smoother gradients and better generalization for deep learning optimizers. Future work is required to understand why the success of DLM in sparse Gaussian Processes does not translate to the complex, non-convex landscapes of deep Bayesian models.

Summary updated

Abstract

Direct Loss Minimization (DLM) has been proposed as a pseudo-Bayesian method motivated as regularized loss minimization. Compared to variational inference, it replaces the loss term in the evidence lower bound (ELBO) with the predictive log loss, which is the same loss function used in evaluation. A number of theoretical and empirical results in prior work suggest that DLM can significantly improve over ELBO optimization for some models. However, as we point out in this paper, this is not the case for Bayesian neural networks (BNNs). The paper explores the practical performance of DLM for BNN, the reasons for its failure and its relationship to optimizing the ELBO, uncovering some interesting facts about both algorithms.
BibTeX
@inproceedings{wei2022performance,
  author = {Yadi Wei and Roni Khardon},
  title = {{On the Performance of Direct Loss Minimization for Bayesian Neural Networks}},
  booktitle = {ICBINB @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=_lzq1tghrG},
}
Figure