TL;DR
kPAM 2.0 introduces a closed-loop robotic manipulation framework that generalizes across entire categories of objects (e.g., different shapes of pegs or erasers) rather than just specific instances. By representing objects using "oriented 3D keypoints" and defining actions and force feedback relative to these points, the system can perform precise, contact-rich tasks regardless of the specific object instance, its initial configuration, or how the robot grasps it.
What Problem Does This Paper Address?
In robotic manipulation, performing precise, contact-rich tasks (like inserting a peg into a tight hole or wiping a whiteboard) typically requires closed-loop feedback. Traditionally, pipelines achieve this by estimating the 6-DOF (Degrees of Freedom) pose of an object based on a known, fixed geometric template. However, this approach breaks down when dealing with categories of objects that have large intra-category shape variations (e.g., mugs of different heights, or erasers of different sizes). A 6-DOF pose template simply cannot map cleanly to objects with different geometries.
Alternatively, end-to-end visuomotor policies (like those trained via RL) can learn contact-rich behaviors, but they often struggle to generalize to new object instances, different camera positions, or novel robot grasp poses without extensive retraining. This paper bridges the gap, offering a perception-to-action framework that handles significant shape variations while maintaining the precision required for contact-rich feedback control.
Methods
The authors build upon their previous work, kPAM (KeyPoint Affordance-based Manipulation), which used 3D semantic keypoints for open-loop pick-and-place tasks. kPAM 2.0 extends this to closed-loop, contact-rich tasks via three main innovations:
- Oriented Keypoints: Instead of just 3D coordinates, the authors augment semantic keypoints with local orientation information (e.g., defining the $z$-axis along the shaft of a peg).
- Object-Centric Action Representation: Instead of commanding the robot's end-effector, the control policy outputs desired linear/angular velocities or forces/torques at the oriented keypoint.
- Virtual Force/Torque Sensors: Raw force/torque (F/T) measurements from a wrist-mounted sensor are highly dependent on the robot's grasp pose and the object's specific geometry. The authors mathematically transform these raw F/T readings into the coordinate frame of the oriented keypoint. This acts as a "virtual sensor" mounted directly at the point of interaction, making the sensory feedback invariant to how the object is grasped.
Because both the state (oriented keypoints) and the actions (velocity/force at the keypoints) are defined locally, standard joint-space controllers (like Jacobian-based controllers) can easily map these object-centric commands to robot joint commands.
Datasets
The paper does not rely on standard offline benchmark datasets (like ImageNet or ShapeNet) for training. Instead, the framework is evaluated through real-world hardware experiments using custom sets of objects grouped by category:
- Whiteboard erasers (various shapes and sizes)
- 3D-printed pegs and holes (with tight 0.2 mm clearances)
- LEGO blocks
- USB drives and ports
Key Insights and Ideas
- Locality is a feature, not a bug: In many manipulation tasks, only a local part of the object actually interacts with the environment (e.g., the tip of a peg, the bottom of a mug). By using local keypoints rather than global 6-DOF poses, the system naturally ignores task-irrelevant geometric details (like the shape of the handle the robot is holding).
- Grasp-Agnostic Control: Because the action space and the force feedback are centered on the object's keypoints rather than the robot's end-effector, the control policy becomes completely agnostic to the robot's grasp pose. You can grab a tool differently, and the same policy still works.
- Simplifying Pick-and-Place: Adding orientation to keypoints significantly simplifies the optimization problem used in the original kPAM. Instead of setting up complex geometric constraints to align objects, you can simply command the relative transformation between an oriented keypoint and its target configuration.
Key Results
The framework was tested on a real robot performing whiteboard wiping and peg-hole insertions (printed pegs, LEGOs, and USBs). It was compared against an open-loop baseline and a standard 6-DOF pose baseline.
- Whiteboard Wiping: kPAM 2.0 achieved a 4% failure rate, compared to 80% for the open-loop baseline and 40% for the 6-DOF baseline. (Open-loop fails because it cannot regulate contact force; 6-DOF fails because template mismatch causes the eraser to collide with the table or miss the board).
- Printed Peg-Hole (0.2mm clearance): kPAM 2.0 achieved a 26% failure rate, massively outperforming the open-loop (95%) and 6-DOF (55%) baselines.
- LEGO Blocks: kPAM 2.0 achieved an 8% failure rate (vs. 35% open-loop, 36% 6-DOF).
- USB Ports: The most visually demanding task. kPAM 2.0 achieved a 45% failure rate, which, while high, was vastly superior to the open-loop (100%) and 6-DOF (85%) baselines.
Limitations and Future Work
- Dense Geometry Tasks: Because keypoints are sparse representations, the framework cannot handle tasks that require knowledge of the object's dense geometry, such as collision-free trajectory planning in tight spaces, or realizing that a specific peg is simply too fat to fit into a specific hole. The authors suggest integrating shape completion algorithms to solve this.
- Static Grasp Assumption: The math assumes the object is rigidly attached to the gripper (no in-hand slipping). If the object moves within the grasp, the keypoint tracking fails. Future work could use external cameras to detect and compensate for relative motion between the gripper and the object.
- Policy Generation: The current experiments used hand-written agents (like compliance controllers) acting on the keypoints. The authors note that combining this representation with imitation learning or reinforcement learning is a highly promising next step.
Summary updated