Time Reversal as Self-Supervision

Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, Vikash KumararXiv 2018 · arXiv

Introduction

Robot learning for complex manipulation tasks, such as block assembly or organization, is often hindered by the "exploration problem." In high-dimensional state spaces, an autonomous agent is unlikely to discover the precise sequence of movements required to assemble complex shapes through random motion alone. Traditionally, overcoming this challenge requires extensive human demonstrations, manually engineered reward functions, or access to privileged information like exact object coordinates—all of which are difficult to scale or unavailable in real-world settings.

This research introduces TRASS (Time Reversal as Self-Supervision), a framework that enables robots to acquire manipulation skills autonomously. The core insight is that while solving a problem (assembly) is difficult, "un-solving" it (disassembly) is often trivial. By recording the disruption of a goal state and reversing the trajectory, the robot generates its own supervision.

Methodology

The TRASS framework operates through a Time-Reversal Model (TRM) and follows a three-phase process:

  1. Data Collection (Disassembly): The robot begins at a goal state (e.g., blocks already mated) and applies random forces to disrupt the scene. This "disassembly" trajectory is recorded as a sequence of images.
  2. TRM Training: The recorded trajectories are reversed. A video prediction model is then trained to take a disrupted state and predict the sequence of frames leading back to the goal. By predicting states (images) rather than actions, the model decouples high-level planning from low-level execution.
  3. Execution (Visual MPC): During testing, the robot encounters a messy scene. The TRM predicts a "guiding trajectory" of images toward the goal. Simultaneously, a separate Visual Dynamics Model predicts the outcomes of potential robot actions. A Cross Entropy Method (CEM) planner then selects actions that best align the predicted outcomes with the TRM's guiding trajectory.
The TRASS pipeline: from disassembly data collection to time-reversed model training and visual control.

Experimental Evaluation

The researchers evaluated TRASS on three distinct tasks: 2-block mating, 3-block mating, and tower stacking. Training was conducted in a MuJoCo simulation environment using domain randomization to vary colors, lighting, and textures, facilitating transfer to physical hardware. Real-world testing was performed on a KUKA IIWA-7 robot arm using only uncalibrated RGB camera input.

Results and Key Insights

TRASS demonstrated significant improvements over traditional self-supervised methods. In simulated 2-block mating, the model achieved a 49% success rate, more than doubling the performance of a semi-supervised baseline using shaped rewards (19%). Notably, TRASS performed nearly as well as models provided with a ground-truth goal image (57%), despite having to "imagine" the goal state itself.

Performance comparison of TRM against shaped rewards and ground truth goals across seen and unseen configurations.

The "reversibility heuristic" proves particularly effective for tasks like assembly, where the backward trajectory provides a natural curriculum for alignment and approach. Furthermore, the model showed robust generalization; when transferred to the real KUKA robot, it maintained a 75% success rate on seen block pairs and a 50% success rate on entirely unseen shapes.

Limitations and Future Work

While powerful, TRASS is limited to tasks with reversible dynamics. It cannot currently be applied to irreversible processes like mixing ingredients or breaking objects. Additionally, the presence of the robot's gripper in training frames can interfere with the time-reversal logic if its interaction is not carefully managed. Future research may focus on planning in latent spaces to mitigate video blurriness over long horizons and extending the framework to more complex, non-random exploration strategies.

Summary updated

Abstract

A longstanding challenge in robot learning for manipulation tasks has been the ability to generalize to varying initial conditions, diverse objects, and changing objectives. Learning based approaches have shown promise in producing robust policies, but require heavy supervision to efficiently learn precise control, especially from visual inputs. We propose a novel self-supervision technique that uses time-reversal to learn goals and provide a high level plan to reach them. In particular, we introduce the time-reversal model (TRM), a self-supervised model which explores outward from a set of goal states and learns to predict these trajectories in reverse. This provides a high level plan towards goals, allowing us to learn complex manipulation tasks with no demonstrations or exploration at test time. We test our method on the domain of assembly, specifically the mating of tetris-style block pairs. Using our method operating atop visual model predictive control, we are able to assemble tetris blocks on a physical robot using only uncalibrated RGB camera input, and generalize to unseen block pairs. sites.google.com/view/time-reversal
BibTeX
@misc{nair2018time,
  author = {Suraj Nair and Mohammad Babaeizadeh and Chelsea Finn and Sergey Levine and Vikash Kumar},
  title = {{Time Reversal as Self-Supervision}},
  year = {2018},
  eprint = {1810.01128},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.01128},
}
Figure