Exploiting Inferential Structure in Neural Processes

Dharmesh Tailor, Mohammad Emtiyaz Khan, Eric T. Nalisnick

Introduction

Neural Processes (NPs) represent a powerful class of models that combine the flexibility of neural networks with the uncertainty quantification of Gaussian Processes. They are designed for "fast adaptation," allowing them to ingest a small set of context points and immediately predict values for target points. However, standard NPs often suffer from underfitting because they rely on a simplistic bottleneck: they typically aggregate context information into a single, global latent variable governed by a simple Gaussian distribution. This paper addresses this limitation by introducing Structured Inference Networks (SIN), a framework that incorporates Probabilistic Graphical Models (PGMs) into the NP architecture.

The Problem: The Representation Bottleneck

In traditional Neural Processes, the aggregation of context points is often handled by heuristic operations like mean-pooling. While computationally efficient, this approach assumes the context set can be summarized by a unimodal Gaussian distribution. In real-world scenarios, data is rarely so well-behaved; it may be multi-modal (containing multiple distinct patterns) or corrupted by heavy-tailed noise (outliers).

Previous attempts to improve NP performance have focused on increasing the complexity of the neural encoders—for instance, by using Self-Attention or Transformers. This paper takes a different approach: instead of simply making the "black box" larger, it refines the underlying probabilistic mathematics to better reflect the inherent structure of the data.

Methods: Structured Inference Networks (SIN)

The authors propose that the aggregation of context points should be treated as a formal inference problem on a PGM. By choosing specific PGM priors, the model can automatically determine how to combine evidence based on probabilistic rules. The paper highlights two primary implementations:

  1. Mixture Bayesian Aggregation (mBA): By using a Mixture of Gaussians prior, the model can represent multi-modal data. This allows the NP to capture multiple potential underlying functions simultaneously, which is particularly useful for complex datasets like handwritten characters.
  2. Robust Bayesian Aggregation (rBA): By employing a Student-t distribution (modeled via a hierarchical Gamma-Gaussian structure), the model becomes resilient to outliers. The inference rules of the PGM naturally learn to downweight context points that deviate significantly from the expected pattern.

To ensure these models remain trainable within a deep learning pipeline, the authors utilize Variational Message Passing (VMP). This allows for the integration of PGM-based inference with standard stochastic gradient descent used for the neural network components.

Experimental Evaluation

The researchers evaluated the SIN framework across two distinct domains:

  • 1-D Regression: Using synthetic data from Gaussian Processes, the authors tested the models against heavy-tail noise. The robust model (rBA) maintained a significantly lower Root Mean Square Error (RMSE) compared to standard NPs, demonstrating its ability to ignore noise that would otherwise skew a standard model's predictions.
  • 2-D Image Completion (EMNIST): Using the EMNIST dataset, the models were trained on digits and then tested on both digits and unseen letters to evaluate out-of-distribution (OOD) performance. The mixture model (mBA) showed consistent improvements in predictive log-likelihood as the number of mixture components increased.

Key Insights and Results

  • Aggregation as Inference: The central contribution of this work is the realization that the "pooling" step in a Neural Process is not just an architectural choice, but a mathematically grounded inference step.
  • Efficiency over Scale: The PGM-based models proved to be highly parameter-efficient. For example, the NP-mBA model outperformed the much larger Attentive NP (NP+SA) on OOD image tasks while utilizing approximately 40% fewer parameters. This suggests that the right probabilistic structure can be more effective than raw model scaling.
  • Automatic Weighting: In the rBA version, the model naturally assigns weights to context points. If a point is identified as an outlier by the PGM’s inference rules, it is automatically given less influence over the final prediction, providing built-in robustness without manual tuning.

Conclusion and Future Work

By exploiting inferential structure, the authors demonstrate that Neural Processes can be made more robust and expressive without a massive increase in parameter count. While the current work focuses on conditionally conjugate models to simplify the mathematics, future research may extend this to more complex PGM structures, such as temporal models for time-series analysis or causal graphs to capture underlying physical laws. This framework opens a new path for designing neural architectures that are both mathematically principled and practically powerful.

Summary updated

BibTeX
@inproceedings{tailor2023exploiting,
  author = {Dharmesh Tailor and Mohammad Emtiyaz Khan and Eric T. Nalisnick},
  title = {{Exploiting Inferential Structure in Neural Processes}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=MbQKovZFHIH},
}
Figure