Abstract
Stochastic processes provide a mathematically elegant way model complex data. In theory, they provide flexible priors over function classes that can encode a wide range of interesting assumptions. In practice, however, efficient inference by optimisation or marginalisation is difficult, a problem further exacerbated with big data and high dimensional input spaces. We propose a novel variational autoencoder (VAE) called the prior encoding variational autoencoder ($\pi$VAE). The $\pi$VAE is finitely exchangeable and Kolmogorov consistent, and thus is a continuous stochastic process. We use $\pi$VAE to learn low dimensional embeddings of function classes. We show that our framework can accurately learn expressive function classes such as Gaussian processes, but also properties of functions to enable statistical inference (such as the integral of a log Gaussian process). For popular tasks, such as spatial interpolation, $\pi$VAE achieves state-of-the-art performance both in terms of accuracy and computational efficiency. Perhaps most usefully, we demonstrate that the low dimensional independently distributed latent space representation learnt provides an elegant and scalable means of performing Bayesian inference for stochastic processes within probabilistic programming languages such as Stan.
Introduction
Stochastic processes, such as Gaussian Processes (GPs), are considered the "gold standard" for modeling functions with uncertainty. However, they suffer from two major limitations: scalability and inference difficulty. Standard GPs scale cubically ($O(n^3)$) with the number of data points, making them impractical for large datasets. Furthermore, performing fully Bayesian inference (MCMC) on these models is challenging because the parameter spaces are high-dimensional and highly correlated, leading to poor sampler performance.
While Deep Learning offers scalability, it often lacks principled uncertainty quantification. Existing "Neural Processes" attempt to address this but often lack formal mathematical properties like Kolmogorov consistency. This paper introduces $\pi$VAE, a framework that bridges the gap between deep learning's flexibility and the mathematical rigor of stochastic processes by "pre-encoding" complex priors into a low-dimensional latent space.
Methodology
The authors propose a two-stage approach to enable efficient Bayesian inference.
Prior Encoding
Instead of training on data immediately, the model first trains a Variational Autoencoder (VAE) to learn a specific class of functions (the prior). The architecture is inspired by the Karhunen-Loève expansion, representing a function as $f(s) = d(Z)^\top \Phi(s)$, where $\Phi(s)$ is a trainable feature mapping (basis functions) and $d(Z)$ is a decoder that transforms a low-dimensional latent vector $Z$ into weights for those bases.
The $\pi$VAE loss function combines mean squared error terms for approximation and reconstruction with a KL-divergence regularization term to ensure the latent space is well-behaved.
Bayesian Inference
Once the $\pi$VAE is trained, the decoder is fixed. To fit new data, inference is performed only on the latent variable $Z$. Because $Z$ is low-dimensional (typically <50) and designed to be nearly uncorrelated, standard MCMC samplers like Hamiltonian Monte Carlo (HMC) in Stan can explore the posterior distribution extremely efficiently, achieving near-perfect convergence diagnostics ($\hat{R} \approx 1$).
Experiments and Results
The model was evaluated on several diverse benchmarks, including 1-D regression, Log-Gaussian Cox Processes, and large-scale real-world datasets.
Spatial Interpolation and Scalability
On the East Africa Temperature dataset (~89,000 points), $\pi$VAE achieved a testing MSE of 0.38, vastly outperforming a full-rank GP (MSE 2.47) and a low-rank GMRF (MSE 4.36). The authors attribute this to $\pi$VAE's ability to capture non-stationary, fine-scale features that traditional kernels often miss. On the Kin40K dataset, $\pi$VAE provided competitive results (RMSE 0.112) compared to state-of-the-art GP approximations while providing a full Bayesian treatment.
Image Completion
In the MNIST image completion task, treated as a 2-D spatial regression problem, the model could generate plausible digit reconstructions even when observing only 10% of the pixels, demonstrating its robustness and expressive power.
Discussion and Conclusion
A key insight of $\pi$VAE is the decoupling of the prior from the likelihood. By learning the prior structure upfront, the actual inference becomes a simple sampling problem in a well-behaved latent space. Furthermore, $\pi$VAE can learn to output function properties, such as integrals, simultaneously with function values—a task that is traditionally computationally expensive.
While the upfront cost of training the $\pi$VAE can be significant, the resulting model is reusable for many different inference tasks. Future work will explore extending the framework to higher-dimensional input spaces and other types of stochastic processes, such as Dirichlet processes.
Summary updated