Variational Inference based on Robust Divergences

Futoshi Futami, Issei Sato, Masashi Sugiyama

Abstract

Robustness to outliers is a central issue in real-world machine learning applications. While replacing a model to a heavy-tailed one (e.g., from Gaussian to Student-t) is a standard approach for robustification, it can only be applied to simple models. In this paper, based on Zellner’s optimization and variational formulation of Bayesian inference, we propose an outlier-robust pseudo-Bayesian variational method by replacing the Kullback-Leibler divergence used for data fitting to a robust divergence such as the beta- and gamma-divergences. An advantage of our approach is that superior but complex models such as deep networks can also be handled. We theoretically prove that, for deep networks with ReLU activation functions, the influence function in our proposed method is bounded, while it is unbounded in the ordinary variational inference. This implies that our proposed method is robust to both of input and output outliers, while the ordinary variational method is not. We experimentally demonstrate that our robust variational method outperforms ordinary variational inference in regression and classification with deep networks.

Introduction

Standard Bayesian inference, and specifically its computationally efficient implementation via Variational Inference (VI), is notoriously sensitive to outliers. This fragility arises because VI typically minimizes the Kullback-Leibler (KL) divergence between the posterior and the data distribution. In this framework, a single extreme outlier can exert an unbounded "pull" on model parameters, dragging the predictive distribution away from the true underlying pattern. While heavy-tailed distributions (like the Student-t) are often used to mitigate this, they are difficult to scale to complex architectures like deep neural networks. This paper proposes a way to make the inference process itself robust, regardless of the specific model architecture.

Methodology: Robust Divergences

The authors leverage Zellner’s optimization formulation of Bayesian inference to replace the standard KL divergence in the Evidence Lower Bound (ELBO) with robust alternatives: $β$-divergence or $γ$-divergence.

These divergences possess a "redescending" property: as a data point becomes more of an outlier (i.e., its probability under the model decreases), its contribution to the gradient of the objective function vanishes. This effectively acts as an automated weighting mechanism that ignores improbable data points during training. The resulting objective functions, termed $β$-VI and $γ$-VI, can be optimized using standard black-box variational techniques like the reparameterization trick.

Theoretical Robustness and the ReLU Vulnerability

A significant theoretical insight of this work concerns the use of ReLU activations. Because ReLUs do not saturate, an outlier in the input space can cause internal activations to grow infinitely, leading to unbounded influence in standard VI. By analyzing the Influence Function (IF)—a measure of how much a small contamination at a specific point changes model parameters—the authors prove that the proposed robust divergences bound this influence even for ReLU networks. This provides finite-sample guarantees of stability that were previously lacking in robust Bayesian methods.

Experimental Results

The proposed method was rigorously tested against standard VI across various regression and classification tasks with artificially injected outliers (10% and 20%).

Regression Performance

In linear regression toy experiments, standard VI is easily skewed by outlier clusters, whereas the proposed robust VI remains aligned with the primary data trend.

Comparison of Ordinary VI and Proposed VI in a regression task with outliers. The robust method (right) effectively ignores the outlier cluster at the top, maintaining a fit consistent with the main data.

On real-world UCI benchmarks like the Concrete dataset with 20% outliers, standard VI resulted in an RMSE of 16.8, while $β$-VI maintained a significantly lower RMSE of 8.15.

Classification Stability

Similar results were observed in classification. Standard VI decision boundaries are heavily distorted by outliers, leading to poor generalization.

Comparison of decision boundaries in a classification task. Ordinary VI (left) is significantly pulled by the blue star outliers, while the proposed VI (right) maintains a boundary that better separates the primary classes.

On the Spam dataset with 20% contamination, the accuracy of standard VI plummeted to 60.6%, whereas $β$-VI remained robust at 92.2%.

Conclusion

By rethinking the objective function of Variational Inference, this research provides a scalable framework for training deep networks that are inherently resistant to "dirty" data. While the introduction of hyperparameters like $β$ requires careful tuning, the bounded influence function and empirical stability make this a powerful tool for real-world applications where data quality cannot be guaranteed.

Summary updated

Abstract

Robustness to outliers is a central issue in real-world machine learning applications. While replacing a model to a heavy-tailed one (e.g., from Gaussian to Student-t) is a standard approach for robustification, it can only be applied to simple models. In this paper, based on Zellner’s optimization and variational formulation of Bayesian inference, we propose an outlier-robust pseudo-Bayesian variational method by replacing the Kullback-Leibler divergence used for data fitting to a robust divergence such as the beta- and gamma-divergences. An advantage of our approach is that superior but complex models such as deep networks can also be handled. We theoretically prove that, for deep networks with ReLU activation functions, the influence function in our proposed method is bounded, while it is unbounded in the ordinary variational inference. This implies that our proposed method is robust to both of input and output outliers, while the ordinary variational method is not. We experimentally demonstrate that our robust variational method outperforms ordinary variational inference in regression and classification with deep networks.
BibTeX
@inproceedings{futami2018variational,
  author = {Futoshi Futami and Issei Sato and Masashi Sugiyama},
  title = {{Variational Inference based on Robust Divergences}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=f5tDwHpVEhq},
}
Figure