Introduction
Specifying complex manipulation tasks for robots remains a significant challenge. While high-level language commands are intuitive for humans, they are difficult for robots to translate into precise physical actions. Conversely, low-level joint trajectories are rigid and lack the semantic grounding needed for generalization. Correspondence-Oriented Imitation Learning (COIL) addresses this gap by utilizing 3D keypoint trajectories as a flexible intermediate representation, bridging the gap between abstract intent and physical execution.
The Challenge of Task Specification
Traditional imitation learning often relies on fixed-rate, dense annotations or 2D image flows. 2D flows, while useful, suffer from depth ambiguity and occlusion issues. Furthermore, requiring dense annotations limits the scalability and ease of use. COIL introduces a 3D task representation that supports variable spatial and temporal granularity. This allows users to specify tasks ranging from sparse "start-to-goal" keypoints to dense motion flows, depending on the complexity of the manipulation.
The COIL Framework
The core of COIL is a policy conditioned on 3D keypoint correspondences. The architecture is designed to handle the inherent flexibility of these inputs through several key components:
- Spatio-Temporal Transformer: The policy employs a transformer architecture that interleaves attention across spatial and temporal dimensions. This allows the model to understand how individual points move over time and how they relate to the overall scene geometry and other keypoints.
- Flow-Matching Head: To handle the multi-modal nature of manipulation (where multiple paths can lead to the same goal), COIL utilizes a flow-matching objective. This generative approach enables the prediction of multi-step action sequences that are robust to variations in execution.
- Hindsight Relabeling: The model is trained entirely in simulation using self-supervised hindsight relabeling. By observing random heuristic motions and treating the resulting object trajectories as intended tasks, the system learns the underlying physics of manipulation without requiring manual expert demonstrations.
Experimental Evaluation
COIL was trained in the ManiSkill3 simulation environment using a diverse set of objects from the YCB and PartNet-Mobility datasets. Evaluation focused on zero-shot generalization to real-world tasks, including:
- Pick-and-Place: Inserting a shoe into a drawer.
- Sweeping: Moving a carrot into a dustpan using a brush.
- Folding: Manipulating a deformable scarf.
Results and Discussion
COIL demonstrated significant improvements over existing baselines like RT-Trajectory and Im2Flow2Act. In the Pick-and-Place task, COIL achieved an 80-90% success rate, even with sparse task specifications. Notably, the system maintained a 60-70% success rate on the zero-shot folding task, highlighting its ability to handle deformable objects and out-of-distribution scenarios.
A key finding is COIL's robustness to granularity; the policy's performance remains high whether provided with 3 or 32 keypoints. This suggests that the transformer effectively learns to infer missing geometric information. While the system currently depends on external 3D trackers and lacks tactile feedback, its ability to bridge 3D intent with robust visuomotor control marks a significant step toward more flexible and generalizable robotic manipulation.
Summary updated