Unsupervised Learning of Visual 3D Keypoints for Control

Boyuan Chen, Pieter Abbeel, Deepak Pathak

Abstract

Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. Prior works show that structured latent space such as visual keypoints often outperforms unstructured representations for robotic control. However, most of these representations, whether structured or unstructured are learned in a 2D space even though the control tasks are usually performed in a 3D environment. In this work, we propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. The input images are embedded into latent 3D keypoints via a differentiable encoder which is trained to optimize both a multi-view consistency loss and downstream task objective. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space. The proposed approach outperforms prior state-of-art methods across a variety of reinforcement learning benchmarks.

Introduction

Learning to control a robot directly from raw pixels is notoriously difficult. Standard "unstructured" representations, such as those generated by typical Convolutional Neural Networks (CNNs), often fail to capture the fine-grained geometry required for precise manipulation. While "structured" representations like 2D keypoints offer improvements, they struggle with 3D reasoning, occlusions, and changes in camera perspective.

The ideal solution is a 3D keypoint representation. However, manually labeling 3D coordinates for every frame of a robot's training is prohibitively expensive and does not scale. Keypoint3D addresses this by asking: Can a robot learn to identify and track 3D keypoints entirely on its own, just by looking at the world from a few different angles?

Methodology

Keypoint3D is an end-to-end unsupervised framework centered around a "differentiable 3D keypoint bottleneck." The pipeline operates as follows:

  1. Multi-view Encoding: The system processes images from $N$ different cameras. A shared encoder predicts 2D heatmaps (locating keypoints in the image plane) and corresponding depth maps.
  2. Unprojection: Utilizing known camera geometry, these 2D coordinates and depths are "unprojected" into a unified 3D world coordinate space.
  3. Consistency & Attention: To handle occlusions, an attention mechanism weights the most confident predictions, averaging them into a single set of global 3D keypoints.
  4. Reconstruction (The Training Signal): To ensure these keypoints are semantically meaningful, the model projects them back into 2D and uses a decoder to reconstruct the original images. Accurate reconstruction implies the keypoints are tracking stable features like robot joints or object handles.
  5. Joint RL Training: Keypoints are learned simultaneously with the reinforcement learning (RL) policy. The task reward helps guide the keypoints toward features that are most useful for control.
Architecture of the Keypoint3D framework showing the multi-view encoding, unprojection, and reconstruction pipeline.

Experiments and Results

The researchers evaluated Keypoint3D across several challenging simulated environments, including Meta-world (8 manipulation tasks), Pybullet Ant (locomotion), and Deformable Manipulation (wrapping a soft scarf around a mannequin).

Keypoint3D demonstrated state-of-the-art performance, outperforming strong baselines like RAD and CURL on the majority of Meta-world tasks. In the scarf-wrapping task, it was the only method capable of consistently solving the problem, as the 3D keypoints effectively "pinned" the deformable fabric in 3D space.

Performance comparison showing Episode Return versus Environment Steps across eight Meta-world tasks. Keypoint3D (green) consistently outperforms or matches baselines.

Key Insights

  • 3D Stability: Unlike 2D representations, which change as an object moves relative to the camera, 3D keypoints remain consistent, making it easier for the RL policy to learn stable control rules.
  • Emergent Semantics: The model naturally discovers joints and handles without explicit labels, as these points serve as the most stable anchors for scene reconstruction across different views.
  • Robustness to Texture: Testing on a "Colorless Ant" environment proved that the model relies on geometry rather than simple color tracking.
  • Generalization: When pre-trained on the ML45 benchmark, Keypoint3D showed significantly better transfer to unseen tasks compared to 2D-based methods.

Conclusion and Future Work

Keypoint3D provides a robust method for discovering 3D structure from pixels without human supervision. While it currently requires known camera matrices and struggles with highly dynamic backgrounds, it represents a significant step toward scalable, geometry-aware robotic control. Future work will focus on dynamically determining the necessary number of keypoints and adapting to "in-the-wild" settings with unknown camera parameters.

Summary updated

Abstract

Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. Prior works show that structured latent space such as visual keypoints often outperforms unstructured representations for robotic control. However, most of these representations, whether structured or unstructured are learned in a 2D space even though the control tasks are usually performed in a 3D environment. In this work, we propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. The input images are embedded into latent 3D keypoints via a differentiable encoder which is trained to optimize both a multi-view consistency loss and downstream task objective. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space. The proposed approach outperforms prior state-of-art methods across a variety of reinforcement learning benchmarks. Code and videos at https://buoyancy99.github.io/unsup-3d-keypoints/
BibTeX
@inproceedings{chen2021unsupervised,
  author = {Boyuan Chen and Pieter Abbeel and Deepak Pathak},
  title = {{Unsupervised Learning of Visual 3D Keypoints for Control}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=AWVUTNaM1I},
}
Figure