On Contrastive Representations of Stochastic Processes

Emile Mathieu, Adam Foster, Yee Whye Teh

Introduction

This paper introduces CRESP (Contrastive Representations of Stochastic Processes), a unifying framework that leverages contrastive learning to model stochastic processes without relying on exact data reconstruction. By operating entirely in representation space, CRESP successfully handles high-dimensional observations (like images) and complex noise distributions that typically break traditional likelihood-based methods such as Neural Processes.

The Problem with Exact Reconstruction

In machine learning, many types of data can be modeled as stochastic processes (SPs)—families of random variables where a "covariate" $x$ (like time, space, or camera viewpoint) maps to an "observation" $y$ (like a sensor reading or a 2D image).

Examples of stochastic processes

Traditionally, models like Neural Processes (NPs) or Conditional Neural Processes (CNPs) learn representations of these processes by trying to reconstruct the observations. They take a context set of $(x, y)$ pairs and use an explicit likelihood function (usually a Gaussian) to predict the exact observation at a new target $x^*$.

However, this exact reconstruction approach breaks down when the observations are high-dimensional (like images) or when the noise is complex and non-Gaussian (like multi-modal noise or color distortion). Forcing a model to perfectly reconstruct a high-dimensional image with complex noise is incredibly difficult and often unnecessary, especially if the ultimate goal is simply to classify the object or predict a high-level feature (like whether two objects have collided).

The CRESP Framework

To address these limitations, the authors propose CRESP, which abandons explicit likelihoods and pixel-perfect reconstruction in favor of self-supervised contrastive learning, specifically using an InfoNCE-style objective.

The architecture employs a flexible encoder that processes covariates and observations separately (e.g., using CNNs for images and Random Fourier Features for coordinates), merges them, and then uses a Transformer with self-attention to aggregate the context set into a single representation.

Crucially, the authors split downstream tasks into two categories and design specific training methods for each:

  1. Targeted CRESP: Designed for tasks that depend on a specific query covariate $x^*$ (e.g., "What is happening at time $t$?"). The model learns to match a predicted representation (based on the context and $x^*$) with the actual representation of the true observation $y^*$, contrasting it against negative samples from other realizations.
  2. Untargeted CRESP: Designed for tasks that depend on the entire process realization (e.g., "What 3D object is this?"). The model splits a context set in half and trains the encoder to make the representations of the two halves similar to each other, contrasting them against representations of entirely different objects.

Datasets and Evaluation

The authors evaluate CRESP on three datasets designed to test robustness to high dimensionality and complex noise:

  • Sinusoids (Synthetic): 1D periodic functions injected with bi-modal noise, which breaks the standard Gaussian assumption.
  • ShapeNet: A standard 3D object dataset treated as a stochastic process where the covariate is the camera viewpoint and the observation is a 2D image. The authors added heavy, randomized color distortion to simulate naturalistic, non-Gaussian noise.
  • Snooker Dynamical Process: A custom physics simulation of two balls bouncing in a 2D box. The covariate is time, and the observation is a 28x28 RGB image. The downstream task is predicting whether the balls are colliding at a specific time.

Key Insights and Results

  • Reconstruction is a bottleneck: For many downstream tasks like classification or collision detection, knowing the exact RGB values of every pixel is unnecessary. Contrastive learning allows the model to capture the semantic essence of the stochastic process without wasting capacity on high-dimensional generative modeling.
  • Attention beats pooling: Previous contrastive approaches for functions used simple mean-pooling to aggregate context points. CRESP demonstrates that using self-attention to weight and combine different views or points is critical for learning high-quality representations.
  • Robustness to Noise: When color distortion was added to ShapeNet, the performance of reconstructive models (CNP, ACNP) plummeted. CRESP, however, remained robust and actually saw slight performance gains from mild distortion, acting as a helpful data augmentation.

Quantitative Highlights: * Sinusoids: Under bi-modal noise, the Attentive Conditional Neural Process (ACNP) completely failed to recover the function parameters. CRESP handled the multi-modal noise easily, and its use of self-attention reduced Mean Squared Error (MSE) by orders of magnitude compared to baseline contrastive methods. * ShapeNet: Untargeted CRESP achieved ~85% accuracy on 13-way object classification, significantly outperforming CNP/ACNP (which hovered around 60-70% and degraded sharply as color distortion increased). * Snooker: On the targeted task of predicting collisions from a few context frames, Targeted CRESP achieved 96.8% accuracy, outperforming CNP (85.3%), FCLR (85.6%), and MetaCDE (87.7%). It successfully learned to interpolate and extrapolate semantic features smoothly over time.

Limitations and Future Work

While powerful, CRESP has some limitations:

  • No Generative Capabilities: Because CRESP bypasses exact reconstruction, it cannot be directly used to generate or predict raw observations in the $Y$ space (e.g., it cannot draw the missing frames of a video).
  • Data Hungry: The self-supervised contrastive encoder requires a large quantity of unlabelled data to train effectively before it can be used for semi-supervised downstream tasks.
  • Lack of Explicit Uncertainty: Unlike Neural Processes, which use latent variables to model uncertainty over the process realization, CRESP produces deterministic embeddings. Extending the InfoNCE loss to handle distributions (stochastic embeddings) is left for future work.

The authors suggest CRESP's robust representations could be highly valuable for reinforcement learning in real-world environments, where naturalistic noise typically cripples explicit likelihood models.

Summary updated

Abstract

Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.
BibTeX
@inproceedings{mathieu2021contrastive,
  author = {Emile Mathieu and Adam Foster and Yee Whye Teh},
  title = {{On Contrastive Representations of Stochastic Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=sAaymAJB_OW},
}
Figure