kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation

Lucas Manuelli, Wei Gao, Peter R. Florence, Russ Tedrake

Abstract: We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations.

Introduction

In robotic manipulation, "pick and place" is a fundamental task. However, traditional industrial pipelines often rely on precise 3D models or 6-DOF pose estimation. While effective for specific parts on an assembly line, this approach fails when robots encounter the vast diversity of everyday objects. For instance, "putting mugs on a shelf" involves mugs of infinite shapes, sizes, and handle configurations. A pose defined for a tall travel mug does not translate to a short espresso cup.

Overview of kPAM: Semantic keypoints enable category-level manipulation by mapping diverse object instances to target configurations via geometric constraints.

To address this, researchers from MIT CSAIL proposed kPAM (KeyPoint Affordances for category-level Manipulation). This framework shifts the focus from rigid pose estimation to semantic 3D keypoints, allowing robots to handle entire categories of objects by representing them through a small set of task-relevant points.

The Challenge of Intra-Category Variation

Traditional "template-based" approaches attempt to align an observed object to a "canonical" 3D model. This fails due to two primary reasons:

  1. Geometric Incompatibility: Proportional differences (e.g., between a sneaker and a boot) can cause the robot to place objects in physically impossible states, such as penetrating the floor or floating in mid-air.
  2. Task Failure: High-precision tasks, like hanging a mug by its handle, require knowing the exact location of specific features. If a new mug's handle differs from the template, pose-based methods often miss the target.

kPAM argues that a full 6-DOF pose or dense mesh is unnecessary. Instead, knowing the locations of a few semantic points—even those not on the object's surface, like the "top center" of a mug—is sufficient for successful manipulation.

The kPAM Framework

The kPAM pipeline factors the manipulation problem into four distinct sub-problems, leveraging specialized networks and optimization techniques.

The kPAM pipeline: From instance segmentation and 3D keypoint detection to optimization-based action planning and execution.
  1. Instance Segmentation: The system uses Mask R-CNN to identify and isolate the target object within an RGB-D image.
  2. 3D Keypoint Detection: An "Integral Network" detects semantic keypoints (e.g., heel and toe for shoes; rim and bottom for mugs). These points provide a consistent representation across the category.
  3. Optimization-Based Planning: Rather than a target pose, the goal is specified as geometric costs and constraints. For example, a constraint might require the mug's bottom center to be on the table, while a cost function minimizes the deviation of the mug's axis from the vertical. The system solves for a rigid transformation ($T_{action}$) that satisfies these requirements.
  4. Execution: An object-agnostic grasp planner picks up the object, and the robot applies the calculated $T_{action}$ to reach the goal state.

Key Insights and Results

By focusing on semantic points (e.g., "where the foot enters"), the robot ignores task-irrelevant details like sole thickness or decorative patterns. This approach also handles symmetry naturally; because kPAM detects specific keypoints (like "bottom center"), the orientation is resolved implicitly without complex symmetry-breaking logic.

The system was evaluated using a Kuka IIWA robot arm on a variety of objects, including 20 shoes and 40 mugs with significant variations in appearance and topology:

  • Shoes on a Rack: Achieved a 98% success rate over 100 trials, with an average heel placement error of just 1.09 cm.
  • Mugs on a Shelf: Achieved a 97-100% success rate, correctly orienting mugs even when they started in horizontal positions.
  • Hanging Mugs: Despite small tolerances (2-3 cm handles), the robot achieved a 100% success rate for regular mugs.

Conclusion

kPAM demonstrates that semantic keypoints are a powerful bridge between perception and action. While current limitations include an assumption of object rigidity and the need for manual keypoint labeling, the system's ability to generalize to never-before-seen objects marks a significant step toward robust, category-level robotic manipulation.

Summary updated

Abstract

We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and size of a coffee mug relative to some canonical mug is not sufficient to successfully hang it on a rack by its handle. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Using this formulation, we factor the manipulation policy into instance segmentation, 3D keypoint detection, optimization-based robot action planning and local dense-geometry-based action execution. This factorization allows us to leverage advances in these sub-problems and combine them into a general and effective perception-to-action manipulation pipeline. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations.
BibTeX
@inproceedings{manuelli2019kpam,
  author = {Lucas Manuelli and Wei Gao and Peter R. Florence and Russ Tedrake},
  title = {{kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation}},
  booktitle = {ISRR},
  year = {2019},
  url = {https://openreview.net/forum?id=VfyltB_wR-},
}
Figure