Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling

Tung Nguyen, Aditya Grover

This paper introduces Transformer Neural Processes (TNPs), a framework that reimagines meta-learning and uncertainty estimation through the lens of sequence modeling. By swapping out complex latent variable models for the expressive power of Transformers, the authors achieve state-of-the-art results in tasks ranging from simple regression to complex Bayesian optimization.

TL;DR

Transformer Neural Processes (TNPs) cast uncertainty-aware meta-learning as a sequence modeling problem, bypassing the underfitting and intractable likelihoods of traditional Neural Processes. By using a custom-masked Transformer architecture, TNPs achieve superior accuracy and uncertainty quantification across regression, image completion, and sequential decision-making tasks.

What Problem Does This Paper Address?

Meta-learning aims to create models that can adapt to new tasks using only a few examples (the "context"). Neural Processes (NPs) were originally designed to combine the best of two worlds: the flexibility and scalability of deep learning with the principled uncertainty estimation of Gaussian Processes (GPs).

However, traditional NPs have two major pain points:

  1. Intractable Likelihoods: Most NPs rely on latent variables and Variational Autoencoders (VAEs). This requires optimizing a surrogate objective (the Evidence Lower Bound, or ELBO) rather than the actual likelihood, which can lead to poor representations.
  2. Underfitting: NPs often struggle to capture the nuances of the data distribution. This leads to overconfident or inaccurate predictions, which is especially problematic in sequential decision-making where precise uncertainty is vital.

Methods

The core insight of this paper is that meta-learning can be treated as an autoregressive sequence modeling task. Instead of using a global latent variable to represent an entire function, TNPs directly model the joint distribution of target labels conditioned on context points.

Key Architectural Innovations

The TNP architecture processes context and target points as unified tokens through a Transformer encoder.
  • Unified Tokens: To respect the relationship between inputs ($x$) and labels ($y$), the model concatenates them into a single token. This allows the authors to remove positional embeddings, ensuring the model is invariant to the order of context points—a strict mathematical requirement for NPs.
  • Custom Masking: The authors designed a specific attention mask to maintain autoregressive integrity. Context points can only attend to themselves, while target points can attend to all context points and any previously predicted target points.
  • Three Variants:
    1. TNP-A (Autoregressive): The most expressive version; it predicts target points one by one.
  • TNP-D (Diagonal): Assumes target points are independent given the context. It is much faster but less expressive.
  • TNP-ND (Non-Diagonal): Uses a Cholesky decomposition to model correlations between target points without full autoregression, striking a balance between speed and performance.

Key Insights and Ideas

  • Abandoning Latents: The paper demonstrates that explicit latent variables are not strictly necessary to model functional uncertainty. Autoregressive modeling in the label space is sufficient and often more effective.
  • Baking in Inductive Biases: Transformers are not naturally "Neural Processes." The authors show that by carefully removing positional encodings and using specific masking techniques, they can force a Transformer to satisfy the mathematical requirements of a stochastic process (context invariance and target equivariance).
  • Attention is (Almost) All You Need: While previous "Attentive" NPs added attention mechanisms on top of a VAE-like structure, TNPs replace the entire stack with a Transformer, significantly reducing underfitting.

Key Results

Image completion results on EMNIST showing TNP variants producing sharper, more accurate reconstructions compared to the BANP baseline.
  • Qualitative Sharpness: In image completion tasks (like EMNIST and CelebA), TNPs produced much sharper and more diverse samples than standard NPs, which often suffer from blurry, "averaged" outputs.
  • Superior Likelihood: In 1-D regression, TNP-A achieved significantly higher log-likelihoods than all baselines (e.g., 1.63 vs. 0.82 for the RBF kernel).
  • Better Decision Making: In contextual bandits, TNP variants achieved drastically lower cumulative regret. For the hardest "Wheel Bandit" settings ($\delta=0.999$), TNP-A had a regret of 6.13, while the best non-TNP baseline (NP) sat at 62.73.
  • Efficiency Trade-offs: While TNP-A is slower due to its autoregressive nature, the TNP-D and TNP-ND variants offer competitive performance with much faster prediction times.

Limitations and Future Work

  • Prediction Latency: The most accurate model, TNP-A, has a high prediction time ($337s$ vs. $\sim2s$ for baselines in the test set) because it must generate target points sequentially.
  • High-Dimensional Scaling: While successful on images and low-dimensional Bayesian Optimization, scaling the autoregressive approach to very high-dimensional input spaces remains a computational challenge.
  • Uncertainty Decomposition: The authors note that future work could focus on better separating "functional" uncertainty (uncertainty about the underlying rule) from "point" uncertainty (noise in specific observations).

Summary updated

Abstract

Neural Processes (NPs) are a popular class of approaches for meta-learning. Similar to Gaussian Processes (GPs), NPs define distributions over functions and can estimate uncertainty in their predictions. However, unlike GPs, NPs and their variants suffer from underfitting and often have intractable likelihoods, which limit their applications in sequential decision making. We propose Transformer Neural Processes (TNPs), a new member of the NP family that casts uncertainty-aware meta learning as a sequence modeling problem. We learn TNPs via an autoregressive likelihood-based objective and instantiate it with a novel transformer-based architecture that respects the inductive biases inherent to the problem structure, such as invariance to the observed data points and equivariance to the unobserved points. We further design knobs within the TNP architecture to tradeoff the increase in expressivity of the decoding distribution with extra computation. Empirically, we show that TNPs achieve state-of-the-art performance on various benchmark problems, outperforming all previous NP variants on meta regression, image completion, contextual multi-armed bandits, and Bayesian optimization.
BibTeX
@inproceedings{nguyen2022transformer,
  author = {Tung Nguyen and Aditya Grover},
  title = {{Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=NNtjQXbb4i},
}
Figure