Abstract
Conditional neural processes (CNPs) are a flexible and efficient family of models that learn to learn a stochastic process from data. They have seen particular application in contextual image completion - observing pixel values at some locations to predict a distribution over values at other unobserved locations. However, the choice of pixels in learning CNPs is typically either random or derived from a simple statistical measure (e.g. pixel variance). Here, we turn the problem on its head and ask: which pixels would a CNP like to observe - do they facilitate fitting better CNPs, and do such pixels tell us something meaningful about the underlying image? To this end we develop the Partial Pixel Space Variational Autoencoder (PPS-VAE), an amortised variational framework that casts CNP context as latent variables learnt simultaneously with the CNP. We evaluate PPS-VAE over a number of tasks across different visual data, and find that not only can it facilitate better-fit CNPs, but also that the spatial arrangement and values meaningfully characterise image information - evaluated through the lens of classification on both within and out-of-data distributions. Our model additionally allows for dynamic adaption of context-set size and the ability to scale-up to larger images, providing a promising avenue to explore learning meaningful and effective visual representations.
Introduction: Beyond Random Sampling in CNPs
Conditional Neural Processes (CNPs) have emerged as a powerful paradigm for few-shot learning and image completion. By observing a small "context set" of pixels, these models predict the distribution of values for the remaining unobserved pixels. Traditionally, the selection of this context set has been treated as a secondary concern, often relying on random sampling or basic heuristics like pixel variance.
This research introduces the Partial Pixel Space Variational Autoencoder (PPS-VAE), which flips the script: instead of asking how to complete an image from random pixels, it asks which pixels are most informative for representing the image. By learning to identify these critical points, the model creates a sparse, interpretable "pixel codebook" that serves as a high-quality latent representation for downstream tasks.
The PPS-VAE Framework
The core innovation of PPS-VAE is treating the context set—both the locations ($x_M$) and their corresponding values ($y_M$)—as latent variables within a variational framework.
The model consists of two primary stages:
- Generative Model: An "abstractive" latent variable ($a$) provides global control. The model first samples context locations and values, then employs a CNP to impute the rest of the image.
- Inference Model: An encoder learns to predict the optimal context points for a given image. Crucially, the authors utilize a ConvCNP architecture. This incorporates translation equivariance, ensuring that the model understands spatial relationships rather than treating the context set as an unordered list.
The entire system is trained using amortized variational inference to maximize the Evidence Lower Bound (ELBO), effectively discovering the most efficient way to summarize a data distribution using its own pixels.
Representation Learning and Results
The learned representations were evaluated across several benchmarks, including FER2013, CelebA, CLEVR, and Tiny ImageNet.
Meaningful Selection Patterns
One of the most striking findings is that the model does not select points randomly. As the number of points ($M$) increases, the PPS-VAE focuses on structural boundaries, internal textures, and complex background elements that define the object's identity.
Quantitative Performance
- Reconstruction: PPS-VAE consistently achieves higher log marginal likelihoods than random or heuristic baselines.
- Classification: The learned pixel sets prove to be highly discriminative. On the CLEVR dataset, classification based on PPS-VAE points reached an F1-macro score of ~90.21%, dwarfing the ~36.17% achieved with random points.
- Robustness: Features learned on Tiny ImageNet transferred effectively to CelebA, indicating that the model captures general visual primitives rather than dataset-specific noise.
Conclusion and Future Directions
PPS-VAE demonstrates that sparse pixel sets can function as powerful, interpretable latent codes. Unlike standard VAEs that map images to abstract vector spaces, PPS-VAE keeps the representation grounded in the original data space.
Future work aims to address current limitations, such as the fixed number of context points during training and the computational cost of autoregressive encoding. Moving toward inferring "closed regions" or objects rather than individual points could further align these models with human-centric visual concepts, bridging the gap between low-level pixels and high-level semantics.
Summary updated