Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning

Yunhao Cao, Zubin Bhaumik, Jessie Jia, Xingyi He, Kuan Fang

Introduction

Specifying complex manipulation tasks for robots remains a significant challenge. While high-level language commands are intuitive for humans, they are difficult for robots to translate into precise physical actions. Conversely, low-level joint trajectories are rigid and lack the semantic grounding needed for generalization. Correspondence-Oriented Imitation Learning (COIL) addresses this gap by utilizing 3D keypoint trajectories as a flexible intermediate representation, bridging the gap between abstract intent and physical execution.

The Challenge of Task Specification

Traditional imitation learning often relies on fixed-rate, dense annotations or 2D image flows. 2D flows, while useful, suffer from depth ambiguity and occlusion issues. Furthermore, requiring dense annotations limits the scalability and ease of use. COIL introduces a 3D task representation that supports variable spatial and temporal granularity. This allows users to specify tasks ranging from sparse "start-to-goal" keypoints to dense motion flows, depending on the complexity of the manipulation.

COIL supports varying levels of spatial and temporal granularity across different tasks like placing, sweeping, and folding.

The COIL Framework

The core of COIL is a policy conditioned on 3D keypoint correspondences. The architecture is designed to handle the inherent flexibility of these inputs through several key components:

  1. Spatio-Temporal Transformer: The policy employs a transformer architecture that interleaves attention across spatial and temporal dimensions. This allows the model to understand how individual points move over time and how they relate to the overall scene geometry and other keypoints.
  2. Flow-Matching Head: To handle the multi-modal nature of manipulation (where multiple paths can lead to the same goal), COIL utilizes a flow-matching objective. This generative approach enables the prediction of multi-step action sequences that are robust to variations in execution.
  3. Hindsight Relabeling: The model is trained entirely in simulation using self-supervised hindsight relabeling. By observing random heuristic motions and treating the resulting object trajectories as intended tasks, the system learns the underlying physics of manipulation without requiring manual expert demonstrations.
The COIL architecture features a Spatio-Temporal Attention mechanism and a Flow-Matching Head to process flexible 3D task specifications.

Experimental Evaluation

COIL was trained in the ManiSkill3 simulation environment using a diverse set of objects from the YCB and PartNet-Mobility datasets. Evaluation focused on zero-shot generalization to real-world tasks, including:

  • Pick-and-Place: Inserting a shoe into a drawer.
  • Sweeping: Moving a carrot into a dustpan using a brush.
  • Folding: Manipulating a deformable scarf.

Results and Discussion

COIL demonstrated significant improvements over existing baselines like RT-Trajectory and Im2Flow2Act. In the Pick-and-Place task, COIL achieved an 80-90% success rate, even with sparse task specifications. Notably, the system maintained a 60-70% success rate on the zero-shot folding task, highlighting its ability to handle deformable objects and out-of-distribution scenarios.

A key finding is COIL's robustness to granularity; the policy's performance remains high whether provided with 3 or 32 keypoints. This suggests that the transformer effectively learns to infer missing geometric information. While the system currently depends on external 3D trackers and lacks tactile feedback, its ability to bridge 3D intent with robust visuomotor control marks a significant step toward more flexible and generalizable robotic manipulation.

Summary updated

Abstract

We introduce Correspondence-Oriented Imitation Learning(COIL), a conditional policy learning framework for visuomotor control with a flexible task representation in 3D. At the core of our approach, each task is defined by the intended motion of keypoints selected on objects in the scene. Instead of assuming a fixed number of keypoints or uniformly spaced time intervals, COIL supports task specifications with variable spatial and temporal granularity, adapting to different user intents and task requirements. To robustly ground this correspondence-oriented task representation into actions, we design a conditional policy with a spatio-temporal attention mechanism that effectively fuses information across multiple input modalities. The policy is trained via a scalable self-supervised pipeline using demonstrations collected in simulation, with correspondence labels automatically generated in hindsight. COIL generalizes across tasks, objects, and motion patterns, achieving superior performance compared to prior methods on real-world manipulation tasks under both sparse and dense specifications.
BibTeX
@misc{cao2025correspondence,
  author = {Yunhao Cao and Zubin Bhaumik and Jessie Jia and Xingyi He and Kuan Fang},
  title = {{Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning}},
  year = {2025},
  eprint = {2512.05953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.05953},
}
Figure