Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning

Yunhao Cao, Zubin Bhaumik, Jessie Jia, Xingyi He, Kuan FangSubmission to ICLR 2026 · Withdrawn

TL;DR

Correspondence-Oriented Imitation Learning (COIL) introduces a flexible 3D task representation where manipulation tasks are specified as intended trajectories of arbitrary object keypoints. Conditioned via a spatio-temporal transformer and a flow-matching policy head, COIL effortlessly handles task specifications ranging from sparse start–goal pairs to dense motion flows. Trained entirely in simulation with hindsight correspondence labeling, the policy successfully transfers zero-shot to real-world tasks involving rigid objects, tool use, and even unseen deformable objects.

What Problem Does This Paper Address?

Specifying robotic tasks usually falls into two extremes:

  1. High-level instructions (language or goal images), which are intuitive for humans but difficult for policies to ground into precise, low-level physical interactions.
  2. Dense low-level trajectories (like 2D optical flows or end-effector paths), which are costly to specify, rigid, and prone to depth ambiguity or occlusion.

Existing 3D flow representations try to bridge this gap, but they typically rely on handcrafted motion controllers, assume fixed numbers of keypoints, or require strictly timed dense waypoints. COIL addresses this rigidity by creating a unified policy that can interpret 3D keypoint motion specifications of arbitrary spatial density (one point to many) and temporal resolution (a single target state up to a full continuous path).

Methods

COIL defines tasks as a tensor $c \in \mathbb{R}^{H \times K \times 3}$, tracking the 3D positions of $K$ object keypoints across $H$ sequential stages. Crucially, the policy only enforces that target coordinates are reached in sequential order, leaving timing and recovery behaviors flexible.

Observed Point Cloud (xt) ──┐
Tracked Keypoints   (ut) ───┼─► [Spatio-Temporal Transformer] ─► [Flow-Matching Head] ─► Multi-step Actions
Task Spec Waypoints  (c) ───┘     (Interleaved Spatial,             (Rectified Flow)
                                  Temporal & Cross-Attention)

The system operates in three main stages:

  • Online 3D Point Tracking: Initial keypoint locations are tracked across multi-view stereo cameras using a 2D point tracker (BootsTAP) fused with stereo depth measurements.
  • Spatio-Temporal Transformer Policy: The policy architecture encodes the point cloud $x_t$, tracked keypoints $u_t$, and task tensor $c$. It runs an interleaved attention mechanism:
  • Temporal self-attention over keypoint trajectories to locate task progress and predict motion.
  • Spatial self-attention across keypoints to capture multi-point geometric constraints.
  • Cross-attention with the observed 3D point cloud to ground motion plans into the physical scene.
  • Flow-Matching Action Head: Action chunks are predicted using a continuous-time diffusion/flow-matching objective (Rectified Flow), capturing multi-modal execution strategies.
  • Hindsight Correspondence Training: Training data is collected in simulation using simple randomized primitives (6DoF pick-and-place and tool use). Trajectories are automatically labeled in hindsight by tracking object points. To support variable granularity at test time, trajectories are heavily augmented by randomly subsampling keypoint counts and time horizons, while injecting tracking noise.

Datasets

  • Pre-training / Simulation: The policy is trained entirely in the ManiSkill3 simulator using assets from the YCB dataset, PartNet-Mobility, and open-source 3D models spanning manipulable items, containers, and tools.
  • Evaluation: Evaluated across three real-world physical robot tasks using a 7-DoF Franka Research 3 arm:
  • Pick-and-Place: Placing a baby shoe into an open drawer without moving the drawer.
  • Sweeping: Tool use using a brush to sweep a carrot into a dustpan.
  • Folding: Manipulating an out-of-distribution deformable scarf.

Key Insights and Ideas

  • Granularity-Agnostic Conditioning: By randomly subsampling spatial points ($K$) and temporal steps ($H$) during self-supervised simulation training, the model naturally learns to interpolate. A user can provide just two points (start and goal) or a complete dense trajectory through the same interface.
  • Flexible Timing Beats Strict Timestamps: Instead of forcing keypoints to reach coordinates at rigid clock cycles, ordering targets sequentially allows the policy to adjust its execution speed dynamically and recover gracefully if an object slips.
  • Sim-to-Real Generalization to Novel Dynamics: Despite being trained only on rigid objects and basic heuristic primitives in simulation, COIL achieves strong zero-shot real-world transfer—even folding deformable cloth—because the policy learns geometry-grounded point transport rather than category-specific behaviors.

Key Results

COIL was compared against state-of-the-art flow/trajectory baselines across three granularity settings: Sparse (3–5 keypoints, 2–5 steps), Medium (8–12 keypoints, 16 steps), and Dense (32 keypoints, 32 steps):

Method Pick-and-Place (Sp / Med / De) Sweeping (Sp / Med / De) Folding (Sp / Med / De)
RT-Trajectory 0/10 / 0/10 / 2/10 0/10 / 0/10 / 1/10 0/10 / 0/10 / 1/10
Im2Flow2Act 2/10 / 3/10 / 8/10 0/10 / 0/10 / 5/10 1/10 / 0/10 / 4/10
General-Flow — / — / 1/10 — / — / 0/10 — / — / 6/10
COIL (Ours) 8/10 / 8/10 / 9/10 6/10 / 6/10 / 7/10 6/10 / 7/10 / 6/10
VLM + COIL — / 7/10 / — — / 5/10 / — — / 4/10 / —
  • Robustness to Sparsity: Baseline methods failed almost completely under sparse specifications, whereas COIL maintained high success rates (e.g., 80% on sparse Pick-and-Place).
  • Ablations: In simulation ablations, removing flow randomization caused the largest drop in success rate (dropping from 82% to 21%), confirming that data augmentation is vital for multi-granularity generalization. Removing the Spatio-Temporal Attention dropped performance to 60%.

Limitations and Future Work

  • Tracking Bottlenecks: Real-world failure analysis showed most errors stemmed from point-tracking failures under occlusions or noisy stereo depth estimations.
  • External Task Generation: The policy cannot currently detect ambiguity in user intent or self-correct bad prompt trajectories; it executes whatever task representation it receives.
  • No Tactile/Force Sensing: Control relies entirely on 3D vision and proprioception. Incorporating force and tactile feedback will be essential for high-precision, contact-rich assembly tasks.

Summary updated

Abstract

We introduce Correspondence-Oriented Imitation Learning(COIL), a conditional policy learning framework for visuomotor control with a flexible task representation in 3D. At the core of our approach, each task is defined by the intended motion of keypoints selected on objects in the scene. Instead of assuming a fixed number of keypoints or uniformly spaced time intervals, COIL supports task specifications with variable spatial and temporal granularity, adapting to different user intents and task requirements. To robustly ground this correspondence-oriented task representation into actions, we design a conditional policy with a spatio-temporal attention mechanism that effectively fuses information across multiple input modalities. The policy is trained via a scalable self-supervised pipeline using demonstrations collected in simulation, with correspondence labels automatically generated in hindsight. COIL generalizes across tasks, objects, and motion patterns, achieving superior performance compared to prior methods on real-world manipulation tasks under both sparse and dense specifications.
BibTeX
@misc{cao2025correspondence,
  author = {Yunhao Cao and Zubin Bhaumik and Jessie Jia and Xingyi He and Kuan Fang},
  title = {{Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning}},
  year = {2025},
  eprint = {2512.05953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.05953},
}
Figure