Autoencoding Conditional Neural Processes for Representation Learning

Victor Prokhorov, Ivan Titov, Siddharth N

TL;DR

The Partial Pixel Space Variational Autoencoder (PPS-VAE) moves away from randomly selecting pixels for image completion, instead learning to identify the most informative "context points" in an image. This approach creates a sparse, semantically meaningful representation that not only improves image reconstruction but also proves highly effective for downstream classification tasks, even in out-of-distribution scenarios.

What Problem Does This Paper Address?

Conditional Neural Processes (CNPs) are powerful models used for tasks like contextual image completion, where they observe a small "context set" of pixels to predict the rest of the image. Traditionally, these context pixels are chosen at random or via simple heuristics, such as picking areas of high variance.

The authors identify a missed opportunity in this standard approach: if a model can choose which parts of the data to observe, it might learn a more efficient and meaningful representation of the underlying signal. They ask two core questions:

  1. Which pixels would a CNP prefer to see to perform better?
  2. Do these chosen pixels tell us something semantically important about the image?

Methods

To answer these questions, the authors introduce the Partial Pixel Space Variational Autoencoder (PPS-VAE). This framework treats the selection of context points as a latent variable problem, learned through amortized variational inference.

PPS-VAE Architecture
  • The Latent Variable $a$: Unlike standard VAEs that use abstract vectors, PPS-VAE introduces an "abstractive" latent variable $a$ that provides smooth control over the arrangement and values of the context set.
  • Autoregressive Encoder: The model uses a CNN-based encoder that selects $M$ pixel locations and values one by one. To make this discrete selection process differentiable, they employ the Gumbel-Softmax trick.
  • ConvCNP Decoder: The decoder is a Convolutional Conditional Neural Process. By using a CNN-based architecture, the model gains a "translation equivariance" inductive bias, which the authors show is crucial for the model to learn meaningful spatial arrangements rather than just treating the context as a lookup table.
  • Dynamic Capacity: A unique feature of this model is its ability to change the number of context points ($M$) at inference time (e.g., training on 128 points but evaluating on 256) without retraining.

Datasets

The model was evaluated on four standard computer vision benchmarks:

  • FER2013: Grayscale images of facial expressions.
  • CelebA: Large-scale face attributes dataset.
  • CLEVR: Synthetic scenes containing various geometric objects (cubes, spheres, cylinders).
  • Tiny ImageNet: A subset of the classic ImageNet dataset with 200 classes at 64x64 resolution.

Key Insights and Ideas

Context points selected on CelebA faces
  • Semantic "Clues": Visual inspection reveals that the model doesn't just pick random noise. It focuses on boundary points (defining shapes), interior points (capturing color/texture), and background points (capturing global context).
  • Data as its Own Codebook: In models like VQ-VAE, the "codebook" is an abstract set of vectors. In PPS-VAE, the "codebook" is a subset of the actual pixels. This makes the representation inherently more interpretable.
  • The "Aha" Moment in Classification: The authors found that a classifier trained only on the sparse context points (roughly 3% of the total pixels) could perform nearly as well as one trained on the full image. On the CLEVR dataset, the context points actually provided better features for classification than the original image, likely because the model learned to ignore redundant background noise and focus on the objects.
  • Generalization: Representations learned on one dataset (e.g., Tiny ImageNet) remained effective for classification on entirely different datasets (e.g., CelebA), suggesting the model learns a general strategy for "what makes an image informative."

Key Results

  • Model Fit: PPS-VAE achieved significantly higher log-marginal likelihoods compared to baselines using random points (ConvCNP) or fixed global selection (PPS-CAE). For example, on CelebA, PPS-VAE scored 14210 vs. 12064 for the standard ConvCNP.
  • Classification Accuracy: On CLEVR, using only 128 context points, the model achieved an F1-macro score of 90.21, compared to 91.90 for the full image.
  • Scalability: The authors demonstrated that the model could scale to larger 256x256 images by using a "tiling" approach, where the encoder selects points within non-overlapping 8x8 blocks.

Limitations and Future Work

  • Human Interpretability: While the pixel clouds are more interpretable than abstract vectors, they are still "noisy" to a human eye. The authors suggest that future versions could learn to select closed regions or patches instead of individual pixels to improve interpretability.
  • Fixed $M$ during Training: Currently, the number of points $M$ is a hyperparameter set before training. The authors note that allowing the model to dynamically decide how many points it needs for a specific image would be a valuable extension.
  • Inductive Biases: The success of the model relies heavily on the CNN architecture. Exploring how different architectures (like Transformers) might change the "preferred" context points is an open question.

Summary updated

Abstract

Conditional neural processes (CNPs) are a flexible and efficient family of models that learn to learn a stochastic process from data. They have seen particular application in contextual image completion - observing pixel values at some locations to predict a distribution over values at other unobserved locations. However, the choice of pixels in learning CNPs is typically either random or derived from a simple statistical measure (e.g. pixel variance). Here, we turn the problem on its head and ask: which pixels would a CNP like to observe - do they facilitate fitting better CNPs, and do such pixels tell us something meaningful about the underlying image? To this end we develop the Partial Pixel Space Variational Autoencoder (PPS-VAE), an amortised variational framework that casts CNP context as latent variables learnt simultaneously with the CNP. We evaluate PPS-VAE over a number of tasks across different visual data, and find that not only can it facilitate better-fit CNPs, but also that the spatial arrangement and values meaningfully characterise image information - evaluated through the lens of classification on both within and out-of-data distributions. Our model additionally allows for dynamic adaption of context-set size and the ability to scale-up to larger images, providing a promising avenue to explore learning meaningful and effective visual representations.
BibTeX
@inproceedings{prokhorov2024autoencoding,
  author = {Victor Prokhorov and Ivan Titov and Siddharth N},
  title = {{Autoencoding Conditional Neural Processes for Representation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=XuQPA4D396},
}
Figure