KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation

Lucas Manuelli, Wei Gao, Peter R. Florence, Russ Tedrake

In the world of robotic manipulation, "pick and place" is a bread-and-butter task. However, most industrial pipelines rely on knowing the exact 3D model of an object or estimating its precise 6-DOF pose. This works great for a specific bolt on an assembly line, but it falls apart when you ask a robot to "put the mugs on the shelf." Mugs come in infinite shapes, sizes, and handle configurations. A pose defined for a tall travel mug won't translate to a short espresso cup.

In this paper, researchers from MIT CSAIL present kPAM (KeyPoint Affordances for category-level Manipulation), a framework that moves away from rigid pose estimation in favor of semantic 3D keypoints.

TL;DR

kPAM enables robots to manipulate entire categories of objects (like shoes or mugs) by representing them as a small set of semantic 3D keypoints. By defining manipulation tasks as geometric costs and constraints on these keypoints, the system achieves high-precision placement across significant intra-category shape variations without needing instance-specific tuning.

What Problem Does This Paper Address?

The core challenge is intra-category variation. Traditional robotic pipelines typically use a "template-based" approach: they have a 3D model of a "canonical" mug and try to align the observed mug to that template.

This fails for two reasons:

  1. Geometric Incompatibility: A transformation that works for a sneaker might result in a boot penetrating the floor or floating in mid-air because their proportions differ.
  2. Task Failure: To hang a mug on a rack, the robot needs to know exactly where the handle is. If the handle on the new mug is higher or smaller than the template, a pose-based approach will likely miss the hook.

The authors argue that we don't need a full 6-DOF pose or a dense 3D mesh to perform a task. We just need to know the locations of a few task-relevant points.

Methods

The kPAM framework factors the manipulation problem into four distinct sub-problems:

  1. Instance Segmentation: Using Mask R-CNN to find the object in an RGB-D image.
  2. 3D Keypoint Detection: The heart of the system. They use an "Integral Network" to detect semantic keypoints (e.g., the heel and toe of a shoe, or the rim and bottom of a mug). Crucially, these keypoints don't even have to be on the object's surface—the "top center" of a mug is often in empty space.
  3. Optimization-Based Planning: Instead of a target pose, the user specifies the goal as geometric constraints. For example: "The mug bottom center must be on the table" (equality constraint) and "The mug axis must be within 5 degrees of vertical" (cost function). The system solves for a rigid transformation ($T_{action}$) that satisfies these requirements.
  4. Execution: An object-agnostic grasp planner picks up the object, and the robot applies the calculated $T_{action}$.

Datasets

The researchers developed a custom pipeline to generate training data efficiently:

  • Data Collection: They used a "LabelFusion" style approach, performing 3D reconstructions of static scenes and manually labeling keypoints on the 3D model once. These labels were then projected back into over 100,000 2D images.
  • Training Set: 117 scenes (43 for shoes, 74 for mugs).
  • Evaluation: They tested the system on a held-out set of 20 shoes and 40 mugs with significant variations in appearance, size, and topology (e.g., different types of heels or handle shapes).

Key Insights and Ideas

  • Semantic > Geometric: By focusing on semantic points (like "where the foot goes in"), the robot ignores task-irrelevant details like the thickness of a shoe's sole or the decorative pattern on a mug.
  • Handling Symmetry Naturally: Pose estimation often struggles with "flipped" orientations for symmetric objects. Because kPAM detects specific keypoints (like "bottom center"), the orientation is resolved implicitly without complex symmetry-breaking logic.
  • The "Affordance" of Keypoints: The paper treats keypoints as a bridge between perception and action. You don't just "see" a point; you see a point that constrains how the robot can move.

Key Results

The authors demonstrated the system on a Kuka IIWA robot arm with impressive results:

  • Shoes on a Rack: Achieved a 98% success rate across 100 trials. The average error for heel placement was just 1.09 cm.
  • Mugs on a Shelf: Achieved a 97-100% success rate. The system correctly oriented mugs even when they started in horizontal positions.
  • Hanging Mugs by Handles: This was the most difficult task due to the small tolerances (2-3 cm handles). The robot achieved a 100% success rate for regular mugs and 50% for "small" mugs.
  • Generalization: The system successfully manipulated objects it had never seen before, proving that the keypoint representation generalizes across the category.

Limitations and Future Work

  • The "Grasp Shift" Problem: Currently, the robot perceives the object, then grasps it. If the object shifts slightly during the grasp, the robot doesn't "re-perceive" it. Adding a wrist-mounted camera to update keypoint locations after grasping is a suggested fix.
  • Rigid Assumption: The current formulation assumes the object is rigid. Extending this to deformable objects (like clothes) would require moving beyond a single rigid transformation ($T_{action}$).
  • Human in the Loop: The system still requires humans to define which keypoints are important and to label them in the initial 3D reconstructions. Moving toward self-supervised keypoint discovery is a promising future direction.

Summary updated

Abstract

We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task – e.g. knowing the pose and size of a coffee mug relative to some canonical mug is not sufficient to successfully hang it on a rack by its handle. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Using this formulation, we factor the manipulation policy into instance segmentation, 3D keypoint detection, optimization-based robot action planning and local dense-geometry-based action execution. This factorization allows us to leverage advances in these sub-problems and combine them into a general and effective perception-to-action manipulation pipeline. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations. The video, supplementary material and source code are available on our project page https://sites.google.com/view/kpam .
BibTeX
@inproceedings{manuelli2019kpam,
  author = {Lucas Manuelli and Wei Gao and Peter R. Florence and Russ Tedrake},
  title = {{KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation}},
  booktitle = {ISRR},
  year = {2019},
  url = {https://openreview.net/forum?id=VfyltB_wR-},
}

Loading related content...

Figure