VFunc: a Deep Generative Model for Functions

Philip Bachman, Riashat Islam, Alessandro Sordoni, Zafarali AhmedarXiv 2018 · OpenReview

Abstract

We introduce a deep generative model for functions, VFunc. Our model provides a joint distribution $p(f, z)$ over functions $f$ and latent variables $z$, allowing for efficient sampling from the marginal $p(f)$ and maximization of a variational lower bound on the functional entropy $H(f)$. This framework enables the optimization of objectives that combine task-specific rewards or likelihoods with functional diversity. We demonstrate that VFunc offers significant benefits for Bayesian deep learning in function space and reinforcement learning, providing robust uncertainty representations and the discovery of diverse strategies.

Introduction: From Parameters to Functions

In traditional Bayesian Deep Learning, uncertainty is typically modeled by placing a prior distribution over the parameters (weights) of a neural network. However, the high dimensionality and over-parameterization of neural networks make the parameter posterior extremely complex and multi-modal, as many distinct weight configurations can represent the identical mathematical function.

VFunc addresses this by modeling uncertainty directly in function space. By treating functions as the primary objects of interest, we can achieve a more parsimonious representation. While Gaussian Processes (GPs) provide a classic approach to functional modeling, they often struggle with scalability and non-linear representation. VFunc bridges this gap, combining the functional reasoning of GPs with the flexibility of deep generative models.

Methodology

The core of VFunc is a joint distribution $p(f, z)$ where a latent variable $z$ indexes a specific function $f$.

  1. The Generative Process: A function is sampled by first drawing $z \sim p(z)$ and then conditioning a prediction network $p(y|x, z)$.
  2. Entropy Maximization: To ensure the model captures a diverse range of valid functions rather than collapsing to a single mode, we maximize the functional entropy $H(f)$.
  3. Variational Bound: As $H(f)$ is generally intractable, we utilize a lower bound based on mutual information $I(f; z)$, requiring a recognition network $q(z|f)$.
  4. Universal Function Encoding: To represent a function $f$ within the recognition network, VFunc uses a "diff gradient" approach. It computes the gradient of the prediction loss with respect to a baseline latent variable based on a set of observed input-output pairs. This gradient acts as a universal feature vector capturing the unique characteristics of the function.

Regression and Uncertainty Estimation

In regression tasks, VFunc aims to represent the distribution of functions that could plausibly fit the observed data. Unlike standard baselines such as Bayes by Backprop or MC-Dropout, which often produce overconfident or counter-intuitive uncertainty estimates in data-sparse regions, VFunc provides robust predictive distributions.

Comparison of uncertainty estimates in 1D regression. VFunc (represented here by Bayes by Hypernet) shows more intuitive uncertainty growth in regions lacking training data compared to MC-Dropout and Bayes by Backprop.

Experiments on toy 1D data and the Mauna Loa CO2 dataset demonstrate that VFunc effectively "knows what it doesn't know," with uncertainty bands that naturally expand as the model moves away from the training distribution.

Reinforcement Learning and Policy Diversity

In Reinforcement Learning (RL), maximizing functional entropy $H(p(\pi))$ serves as a formal mechanism to encourage policy diversity. Instead of converging to a single optimal path, VFunc learns a distribution of policies where different regions of the latent space $z$ correspond to different valid strategies.

Latent variable interpolation in a double-slit gridworld environment. By smoothly varying the latent parameter α, the model transitions between distinct navigation strategies (e.g., taking the top path vs. the bottom path).

In complex environments like the "double-slit" maze or "Four Rooms," VFunc successfully discovers multiple ways to reach a goal. As shown in the interpolation experiments, the latent space is well-structured; moving through $z$-space results in smooth, meaningful transitions between different high-performing behaviors.

Conclusion

VFunc provides a scalable and flexible framework for modeling distributions over functions. By shifting the focus from parameter space to function space and leveraging variational entropy bounds, it achieves superior uncertainty estimation and discovers diverse strategies in RL. Future work will focus on scaling these representations to high-dimensional inputs and optimizing the selection of informative points for function encoding.

Summary updated

Abstract

We introduce a deep generative model for functions. Our model provides a joint distribution p(f, z) over functions f and latent variables z which lets us efficiently sample from the marginal p(f) and maximize a variational lower bound on the entropy H(f). We can thus maximize objectives of the form E_{f~p(f)}[R(f)] + c*H(f), where R(f) denotes, e.g., a data log-likelihood term or an expected reward. Such objectives encompass Bayesian deep learning in function space, rather than parameter space, and Bayesian deep RL with representations of uncertainty that offer benefits over bootstrapping and parameter noise. In this short paper we describe our model, situate it in the context of prior work, and present proof-of-concept experiments for regression and RL.
BibTeX
@misc{bachman2018vfunc,
  author = {Philip Bachman and Riashat Islam and Alessandro Sordoni and Zafarali Ahmed},
  title = {{VFunc: a Deep Generative Model for Functions}},
  year = {2018},
  eprint = {1807.04106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.04106},
}
Figure