kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation

Wei Gao, Russ Tedrake

TL;DR

kPAM 2.0 introduces a closed-loop robotic manipulation framework that generalizes across entire categories of objects (e.g., different shapes of pegs or erasers) rather than just specific instances. By representing objects using "oriented 3D keypoints" and defining actions and force feedback relative to these points, the system can perform precise, contact-rich tasks regardless of the specific object instance, its initial configuration, or how the robot grasps it.

What Problem Does This Paper Address?

In robotic manipulation, performing precise, contact-rich tasks (like inserting a peg into a tight hole or wiping a whiteboard) typically requires closed-loop feedback. Traditionally, pipelines achieve this by estimating the 6-DOF (Degrees of Freedom) pose of an object based on a known, fixed geometric template. However, this approach breaks down when dealing with categories of objects that have large intra-category shape variations (e.g., mugs of different heights, or erasers of different sizes). A 6-DOF pose template simply cannot map cleanly to objects with different geometries.

Alternatively, end-to-end visuomotor policies (like those trained via RL) can learn contact-rich behaviors, but they often struggle to generalize to new object instances, different camera positions, or novel robot grasp poses without extensive retraining. This paper bridges the gap, offering a perception-to-action framework that handles significant shape variations while maintaining the precision required for contact-rich feedback control.

Methods

The authors build upon their previous work, kPAM (KeyPoint Affordance-based Manipulation), which used 3D semantic keypoints for open-loop pick-and-place tasks. kPAM 2.0 extends this to closed-loop, contact-rich tasks via three main innovations:

  1. Oriented Keypoints: Instead of just 3D coordinates, the authors augment semantic keypoints with local orientation information (e.g., defining the $z$-axis along the shaft of a peg).
  2. Object-Centric Action Representation: Instead of commanding the robot's end-effector, the control policy outputs desired linear/angular velocities or forces/torques at the oriented keypoint.
  3. Virtual Force/Torque Sensors: Raw force/torque (F/T) measurements from a wrist-mounted sensor are highly dependent on the robot's grasp pose and the object's specific geometry. The authors mathematically transform these raw F/T readings into the coordinate frame of the oriented keypoint. This acts as a "virtual sensor" mounted directly at the point of interaction, making the sensory feedback invariant to how the object is grasped.

Because both the state (oriented keypoints) and the actions (velocity/force at the keypoints) are defined locally, standard joint-space controllers (like Jacobian-based controllers) can easily map these object-centric commands to robot joint commands.

Datasets

The paper does not rely on standard offline benchmark datasets (like ImageNet or ShapeNet) for training. Instead, the framework is evaluated through real-world hardware experiments using custom sets of objects grouped by category:

  • Whiteboard erasers (various shapes and sizes)
  • 3D-printed pegs and holes (with tight 0.2 mm clearances)
  • LEGO blocks
  • USB drives and ports

Key Insights and Ideas

  • Locality is a feature, not a bug: In many manipulation tasks, only a local part of the object actually interacts with the environment (e.g., the tip of a peg, the bottom of a mug). By using local keypoints rather than global 6-DOF poses, the system naturally ignores task-irrelevant geometric details (like the shape of the handle the robot is holding).
  • Grasp-Agnostic Control: Because the action space and the force feedback are centered on the object's keypoints rather than the robot's end-effector, the control policy becomes completely agnostic to the robot's grasp pose. You can grab a tool differently, and the same policy still works.
  • Simplifying Pick-and-Place: Adding orientation to keypoints significantly simplifies the optimization problem used in the original kPAM. Instead of setting up complex geometric constraints to align objects, you can simply command the relative transformation between an oriented keypoint and its target configuration.

Key Results

The framework was tested on a real robot performing whiteboard wiping and peg-hole insertions (printed pegs, LEGOs, and USBs). It was compared against an open-loop baseline and a standard 6-DOF pose baseline.

  • Whiteboard Wiping: kPAM 2.0 achieved a 4% failure rate, compared to 80% for the open-loop baseline and 40% for the 6-DOF baseline. (Open-loop fails because it cannot regulate contact force; 6-DOF fails because template mismatch causes the eraser to collide with the table or miss the board).
  • Printed Peg-Hole (0.2mm clearance): kPAM 2.0 achieved a 26% failure rate, massively outperforming the open-loop (95%) and 6-DOF (55%) baselines.
  • LEGO Blocks: kPAM 2.0 achieved an 8% failure rate (vs. 35% open-loop, 36% 6-DOF).
  • USB Ports: The most visually demanding task. kPAM 2.0 achieved a 45% failure rate, which, while high, was vastly superior to the open-loop (100%) and 6-DOF (85%) baselines.

Limitations and Future Work

  • Dense Geometry Tasks: Because keypoints are sparse representations, the framework cannot handle tasks that require knowledge of the object's dense geometry, such as collision-free trajectory planning in tight spaces, or realizing that a specific peg is simply too fat to fit into a specific hole. The authors suggest integrating shape completion algorithms to solve this.
  • Static Grasp Assumption: The math assumes the object is rigidly attached to the gripper (no in-hand slipping). If the object moves within the grasp, the keypoint tracking fails. Future work could use external cameras to detect and compensate for relative motion between the gripper and the object.
  • Policy Generation: The current experiments used hand-written agents (like compliance controllers) acting on the keypoints. The authors note that combining this representation with imitation learning or reinforcement learning is a highly promising next step.

Summary updated

Abstract

In this letter, we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a realtime 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in [13] for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile - we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment. Video demonstration, source code and supplemental materials are available on https://sites.google.com/view/kpam2/home.
BibTeX
@article{gao2021kpam,
  author = {Wei Gao and Russ Tedrake},
  title = {{kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=G0yhLGfGi8o},
}

Loading related content...

Figure