kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation

Wei Gao, Russ Tedrake

Abstract

In this paper, we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a real-time 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in kPAM for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile -- we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment.

Introduction

Precise robotic manipulation in contact-rich environments, such as inserting a peg into a tight hole or wiping a whiteboard, necessitates robust closed-loop feedback. Traditional pipelines achieve this by estimating the 6-DOF (Degrees of Freedom) pose of an object based on a known, fixed geometric template. However, this approach breaks down when dealing with entire categories of objects that exhibit large intra-category shape variations (e.g., erasers of different sizes or mugs of different heights). A static 6-DOF pose template simply cannot map cleanly to objects with varying geometries.

While end-to-end visuomotor policies can learn contact-rich behaviors, they often struggle to generalize to new object instances or novel robot grasp poses without extensive retraining. kPAM 2.0 bridges this gap, offering a perception-to-action framework that handles significant shape variations while maintaining the precision required for contact-rich feedback control.

Methodology

The kPAM 2.0 framework builds upon the original KeyPoint Affordance-based Manipulation (kPAM) system, extending it from open-loop pick-and-place to closed-loop, contact-rich tasks through three primary innovations:

  1. Oriented Keypoints: Semantic keypoints are augmented with local orientation information (e.g., defining the $z$-axis along the shaft of a peg). This provides a local coordinate frame for interaction.
  2. Object-Centric Action Representation: Instead of commanding the robot's end-effector directly, the control policy outputs desired linear/angular velocities or forces/torques at the oriented keypoint.
  3. Virtual Force/Torque Sensors: Raw force/torque (F/T) measurements from a wrist-mounted sensor are mathematically transformed into the coordinate frame of the oriented keypoint. This "virtual sensor" makes sensory feedback invariant to how the object is grasped.
Overview of object categories and manipulation tasks.

By focusing on local interaction points, the system naturally ignores task-irrelevant geometric details (like the shape of a handle) and becomes completely agnostic to the robot's grasp pose. Standard joint-space controllers can then map these object-centric commands to robot joint commands.

Experimental Setup

The framework was evaluated through real-world hardware experiments using custom sets of objects grouped by category, each presenting unique geometric challenges:

  • Whiteboard Erasers: Various shapes and sizes requiring consistent contact force.
  • 3D-Printed Pegs and Holes: Custom sets with tight 0.2 mm clearances.
  • LEGO Blocks: Standardized blocks of different colors and sizes.
  • USB Drives and Ports: Objects with high visual and geometric complexity.

Results and Discussion

kPAM 2.0 was compared against an open-loop baseline and a standard 6-DOF pose baseline across several demanding tasks.

Whiteboard Wiping

The system achieved a 4% failure rate, significantly outperforming the open-loop (80%) and 6-DOF (40%) baselines. Open-loop control fails due to a lack of force regulation, while 6-DOF templates often lead to collisions or missed contact due to geometric mismatch.

Robot performing the whiteboard wiping task using oriented keypoints.

Peg-Hole Insertion and Assembly

  • Printed Pegs (0.2mm clearance): kPAM 2.0 reduced the failure rate to 26%, compared to 95% for open-loop and 55% for 6-DOF.
  • LEGO Blocks: The framework achieved an 8% failure rate, whereas baselines hovered around 35-36%.
  • USB Ports: In this visually demanding task, kPAM 2.0 achieved a 45% failure rate, which was vastly superior to the open-loop (100%) and 6-DOF (85%) results.

Conclusion and Future Work

kPAM 2.0 demonstrates that local, oriented keypoints provide a robust representation for category-level manipulation. However, limitations remain. Sparse keypoints do not capture dense geometry, which is necessary for collision-free planning in tight spaces. Additionally, the current model assumes a static grasp; future iterations could incorporate external sensing to handle in-hand slippage. Integrating this representation with learning-based policy generation remains a promising avenue for future research.

Summary updated

Abstract

In this paper, we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a real-time 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in kPAM for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile -- we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment.
BibTeX
@article{gao2021kpam,
  author = {Wei Gao and Russ Tedrake},
  title = {{kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=G0yhLGfGi8o},
}
Figure