Unsupervised Learning of Visual 3D Keypoints for Control

Boyuan Chen, Pieter Abbeel, Deepak Pathak

Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. For a robot to perform complex tasks—like picking up a cup or navigating an environment—it must derive a robust understanding of its surroundings from visual data.

Most current deep reinforcement learning (RL) methods rely on either "unstructured" representations (like raw CNN features) or 2D keypoints. Unstructured features often fail to capture fine-grained geometry and struggle to generalize when the visual environment changes. While 2D keypoints offer an improvement, they suffer from a fundamental flaw: they exist in a flat image space. They cannot easily reason about depth, occlusions, or the true 3D kinematic structure of a robot. Existing 3D keypoint methods, on the other hand, often require expensive human labeling or are not trained end-to-end with the actual control task, meaning the keypoints might not track the parts of the scene most critical for the job.

To bridge this gap, researchers introduce Keypoint3D, a framework that learns structured 3D keypoint representations from multi-view images without any manual labels. By enforcing geometric consistency across different camera views and training jointly with RL, the model discovers task-relevant 3D structures that significantly improve a robot's ability to manipulate objects and navigate complex environments.

Methodology: The Keypoint3D Pipeline

The authors propose an end-to-end pipeline that transforms multi-view images into a sparse set of 3D coordinates.

  1. Differentiable 3D Bottleneck: For each camera view, an encoder predicts 2D probability heatmaps and depth maps. The system calculates the "expected" 2D position and depth, which are then "unprojected" into 3D world coordinates using known camera geometry.
  2. Multi-view Aggregation: Because multiple cameras observe the same scene, the model employs an attention mechanism to weight the most confident predictions, averaging them into a single set of global 3D keypoints.
  3. Unsupervised Losses: To learn without manual labels, the model leverages three signals:
  4. Multi-view Consistency: Forces different camera views to agree on the same 3D location for a given keypoint.
  5. Reconstruction: A decoder attempts to rebuild the original images from the reprojected keypoints.
  6. Separation: Prevents all keypoints from collapsing into a single point.
  7. Joint RL Training: Crucially, the encoder is trained simultaneously with the RL policy (e.g., PPO). This allows the task's "reward" signal to influence which keypoints the model decides to track, ensuring they are functionally relevant.
The Keypoint3D framework: Multi-view images are encoded into 3D keypoints, which are then used for both image reconstruction and RL control.

Key Insights and Inductive Biases

By baking the mathematics of 3D projection directly into the neural network, the authors give the model a geometric "head start" in understanding the physical world. Because the model is trained jointly with RL, it discovers useful keypoints rather than arbitrary ones (e.g., naturally learning to track a door handle during a door-opening task).

Furthermore, unlike many computer vision methods that fail on smooth, featureless surfaces, Keypoint3D’s multi-view consistency allows it to track the underlying "skeleton" of a robot even if it lacks distinct color textures. Ultimately, the experiments consistently demonstrate that 3D keypoints outperform 2D ones by providing a consistent coordinate frame that does not "warp" as the robot moves relative to the camera.

Experimental Evaluation and Results

The researchers evaluated Keypoint3D across several challenging benchmarks, including Meta-World (8 distinct robotic manipulation tasks), the PyBullet Ant (a high-dimensional quadruped locomotion task), a custom Deformable Manipulation environment, and ML45 (a multi-task benchmark for transfer learning).

Keypoint3D outperformed state-of-the-art baselines like RAD and CURL across the board:

  • Manipulation: In Meta-World, Keypoint3D achieved significantly higher success rates and faster learning on 5 out of 8 tasks, matching the best baselines on the remaining tasks.
  • Locomotion: On the PyBullet Ant, Keypoint3D was the only method to reach high performance consistently. This was especially true in a "colorless" setting where other visual methods failed, proving the model relies on true geometry rather than just color textures.
  • Generalization: In the ML45 transfer learning test, pre-training with Keypoint3D allowed the robot to learn new tasks much faster than methods lacking a structured 3D bottleneck.
Performance comparison on Meta-World tasks. Keypoint3D (green) consistently reaches higher episode returns faster than baselines.

Limitations and Future Work

The primary limitation of Keypoint3D is its reliance on multiple synchronized cameras with known extrinsic and intrinsic parameters (calibration). In many real-world, "in-the-wild" scenarios, this precise information might not be available.

Future work could involve learning to estimate camera parameters on the fly or extending the framework to handle dynamic backgrounds more robustly. Additionally, while the keypoints are sparse and computationally efficient, scaling the system to track hundreds of keypoints for extremely complex, cluttered scenes remains an open computational challenge.

Summary updated

Abstract

Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. Prior works show that structured latent space such as v...
BibTeX
@inproceedings{chen2021unsupervised,
  author = {Boyuan Chen and Pieter Abbeel and Deepak Pathak},
  title = {{Unsupervised Learning of Visual 3D Keypoints for Control}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=AWVUTNaM1I},
}
Figure