On Contrastive Representations of Stochastic Processes

Emile Mathieu, Adam Foster, Yee Whye Teh

Abstract

Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.

Introduction

Stochastic processes (SPs) provide a powerful mathematical framework for modeling systems with inherent variability, such as the varying viewpoints of a 3D object, the evolution of a physical system over time, or the fluctuations of financial markets. In machine learning, the goal is often to learn a compact representation—a latent code—that captures the underlying generative rules of these processes.

Traditionally, the Neural Process (NP) family has been the dominant approach. These models are trained using a reconstructive objective: given a set of "context" observations, the model attempts to predict the value of the process at a new "target" location. However, this reliance on explicit likelihood (typically Gaussian) makes them fragile. When faced with high-dimensional data like images or complex, non-Gaussian noise, reconstructive models often produce blurry results or fail to capture the true essence of the process.

Stochastic processes can represent a wide variety of data types, from 1D functions to 3D objects and video sequences.

The CReSP Framework

To overcome the limitations of reconstruction, we introduce CReSP (Contrastive Representations of Stochastic Processes). Instead of predicting raw observations, CReSP learns to distinguish between observations that belong to the same realization of a process and those that do not. By employing the InfoNCE contrastive loss, the model learns a latent space where representations of the same process are clustered together.

Targeted vs. Untargeted Representations

A key contribution of our work is the distinction between two types of representation needs:

  1. Targeted CReSP: This variant learns a representation for a specific query point within the process. For example, in a 3D object task, it might represent what the object looks like from a specific, unseen camera angle.
  2. Untargeted CReSP: This variant produces a global summary of the entire process, capturing its overall identity regardless of specific query points (e.g., identifying the type of 3D object).
The CReSP framework supports both targeted (left) and untargeted (right) representation learning using a contrastive objective.

The architecture utilizes a flexible encoder that combines domain-specific networks (like CNNs for images) with Transformer-based self-attention. This attention mechanism allows the model to dynamically weigh the importance of different context observations, a significant improvement over the simple mean-pooling used in prior contrastive methods like FCLR.

Experimental Evaluation

We evaluated CReSP across three diverse domains to demonstrate its robustness and versatility.

3D Objects (ShapeNet)

Using the ShapeNet dataset, we modeled 3D objects as stochastic processes where each observation is a 2D viewpoint. To test robustness, we introduced randomized color distortions. While traditional reconstructive models (CNP and ACNP) saw their performance degrade significantly as noise increased, CReSP remained stable, significantly outperforming them in downstream classification tasks.

Dynamical Systems (Snooker)

In the "Snooker" task, we modeled the motion of two balls in a box. The goal was to predict whether the balls overlap at a specific future time based on a few initial snapshots. Targeted CReSP achieved an impressive 96.8% accuracy, far surpassing specialized baselines like MetaCDE (87.7%) and standard CNPs (85.3%).

Periodic Functions

We tested CReSP on synthetic sinusoids with bi-modal noise—a scenario where Gaussian-based reconstructive models inherently fail. CReSP successfully captured the underlying structure of the functions by bypassing the need to model the complex noise distribution explicitly.

Discussion and Future Work

CReSP demonstrates that for many critical tasks—such as classification, collision detection, or system identification—perfect reconstruction is not only unnecessary but can be a hindrance. By focusing on contrastive signals, CReSP achieves superior robustness to noise and high-dimensional data.

However, the framework currently lacks the explicit uncertainty quantification provided by traditional Neural Processes. Furthermore, as a purely discriminative model, it cannot be used for data synthesis. Future research will explore integrating probabilistic elements into the contrastive objective to regain uncertainty benefits without sacrificing the robustness of the latent representation.

Summary updated

Abstract

Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.
BibTeX
@inproceedings{mathieu2021contrastive,
  author = {Emile Mathieu and Adam Foster and Yee Whye Teh},
  title = {{On Contrastive Representations of Stochastic Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=sAaymAJB_OW},
}
Figure