Palm up: Playing in the Latent Manifold for Unsupervised Pretraining

Hao Liu, Tom Zahavy, Volodymyr Mnih, Satinder Singh

TL;DR

PALM (Playing in the Latent Manifold) bridges the gap between static vision datasets and interactive reinforcement learning by turning a pretrained generative model (like StyleGAN) into an interactive environment. By letting an unsupervised RL agent explore this latent space, PALM generates naturally evolving trajectories that train strong visual representations without relying on manual data augmentations.

What Problem Does This Paper Address?

Modern AI excels at learning from massive, static datasets of images or text. However, intelligent creatures learn primarily through interaction—actively manipulating their surroundings to observe how the world changes.

While we can train reinforcement learning (RL) agents in simulated 3D environments (like Habitat or AI2THOR), these simulators are tedious to hand-design and often lack the visual diversity and complexity of the real world. PALM tackles this dilemma by asking: Can we use the rich, diverse distributions already captured by deep generative models as an interactive sandbox for representation learning?

Methods

The authors introduce a framework that replaces hand-crafted 3D simulation engines with a pretrained conditional generative model (such as StyleGAN).

  1. Latent Environment Dynamics: At each timestep, the agent outputs an action $a_t$. The environment mixes this action with a randomly sampled latent vector $z$ and applies an Exponential Moving Average (EMA) to ensure temporal smoothness. This creates a continuous, persistent trajectory through the latent space.
  2. Observation Generation: The resulting latent vector is passed through the pretrained generator (conditioned on a fixed class prompt for the duration of the episode) to render an image observation $s_t$.
  3. Unsupervised Exploration: The RL agent explores this latent environment using an intrinsic reward based on particle entropy maximization (APT). This explicitly incentivizes the agent to discover novel, unseen images within the generator's capability.
  4. Representation Learning: Using consecutive frames from the agent's trajectory, a Siamese network (SimSiam framework) learns visual representations by maximizing similarity across timesteps. Crucially, PALM doesn't require standard data augmentations (like cropping or color jittering) because navigating the latent space naturally simulates environmental changes like lighting shifts and object deformations.
Overview of the PALM framework showing agent-environment interaction, latent transition dynamics, and the representation learning architecture.

Datasets

  • Pretraining Environments: StyleGAN models trained on CIFAR-10 and CIFAR-100. For reinforcement learning tasks, StyleGAN was trained on a dataset of 50 million frames collected from Atari games using a Double DQN agent.
  • Downstream Evaluation: Linear probing on CIFAR-10 and CIFAR-100.
  • Out-of-Distribution (OOD) Benchmarks: SVHN, LSUN, CelebA, and an interpolated CIFAR dataset.
  • RL Evaluation: The Atari 100k benchmark (evaluating sample efficiency within a strict 100k interaction limit).

Key Insights and Ideas

  • Generative Models as Simulators: Pretrained GANs contain highly complex semantic manifolds. Defining simple transition dynamics over their latent spaces allows us to turn any static image dataset into an interactive environment without any manual 3D modeling.
  • Natural Augmentation via Time: Traditional self-supervised learning relies heavily on artificial, hand-crafted image transformations. PALM demonstrates that the temporal persistence of an agent moving through a latent space provides a "natural" curriculum of visual variations (e.g., smooth transitions in shape, color, and occlusion).
  • The Power of Agency: Active interaction matters. Ablation studies showed that if the agent's actions are replaced with random noise, or if the environment lacks temporal memory, the quality of the learned representation drops significantly. Active exploration forces the model to learn robust data invariances.

Key Results

  • Data-Augmentation Free Performance: On CIFAR-10 linear classification, PALM achieves 76% accuracy without using any standard data augmentations during representation learning. When standard augmentations are added back in, PALM matches or outperforms top-tier static self-supervised methods like SimSiam and SimCLR.
Figure 3: Linear classification results from different methods in CIFAR datasets. Model is ResNet-18 except for CLIP [55] which is based on ResNet-50. Supervised ResNet [22] is trained from scratch. SimCLR [10] and SimSiam [11] are pretrained on CIFAR dataset. CLIP [55] is pretrained on paired image-text datasets. We use reported scores of each baseline for comparison. All results of our method are the average of three runs with different seeds. PALM (our method) actively collect synthesized data from deep generative models based environment. The representation is learned without using data augmentation from explored experience but still achieves competitive results. Our PALM, which is based on SimSiam [11], is compatible with data augmentation, by adding the same data augmentation used in SimSiam, PALM + data aug matches or outperforms state-of-the-art static representation learning algorithms.
  • Superior OOD Detection: PALM outperforms the state-of-the-art CSI method on 4 out of 5 out-of-distribution datasets, achieving 97.9% AUROC on SVHN compared to CSI's 96.5%.
  • Downstream RL Boost: Pretraining with PALM on synthetic Atari environments significantly improves sample efficiency on the Atari 100k benchmark, achieving a higher median and mean Human Normalized Score (HNS) than training from scratch with DrQ.

Limitations and Future Work

  • Computational Expense: Generating samples from deep generative models like StyleGAN at every RL step is computationally heavy and significantly slower than highly optimized physics engines (like MuJoCo).
  • Simplistic Latent Dynamics: The current transition dynamics (linear mixing + EMA) are hand-crafted and simple. Learning more realistic or structured transition dynamics remains an open challenge.
  • Alternative Generative Models: The authors suggest extending PALM to language-conditioned generative models (like DALL-E) to allow agents to interact with environments driven entirely by natural language prompts.

Summary updated

Abstract

Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence. Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment. In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets. Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space. The transition dynamics simply mixes an action and a random sampled latent. It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator. We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data. We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning. Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.
BibTeX
@inproceedings{liu2022palm,
  author = {Hao Liu and Tom Zahavy and Volodymyr Mnih and Satinder Singh},
  title = {{Palm up: Playing in the Latent Manifold for Unsupervised Pretraining}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=6y0lgLb9tny},
}
Figure