TL;DR
Correspondence-Oriented Imitation Learning (COIL) introduces a flexible 3D task representation where manipulation tasks are specified as intended trajectories of arbitrary object keypoints. Conditioned via a spatio-temporal transformer and a flow-matching policy head, COIL effortlessly handles task specifications ranging from sparse start–goal pairs to dense motion flows. Trained entirely in simulation with hindsight correspondence labeling, the policy successfully transfers zero-shot to real-world tasks involving rigid objects, tool use, and even unseen deformable objects.
What Problem Does This Paper Address?
Specifying robotic tasks usually falls into two extremes:
- High-level instructions (language or goal images), which are intuitive for humans but difficult for policies to ground into precise, low-level physical interactions.
- Dense low-level trajectories (like 2D optical flows or end-effector paths), which are costly to specify, rigid, and prone to depth ambiguity or occlusion.
Existing 3D flow representations try to bridge this gap, but they typically rely on handcrafted motion controllers, assume fixed numbers of keypoints, or require strictly timed dense waypoints. COIL addresses this rigidity by creating a unified policy that can interpret 3D keypoint motion specifications of arbitrary spatial density (one point to many) and temporal resolution (a single target state up to a full continuous path).
Methods
COIL defines tasks as a tensor $c \in \mathbb{R}^{H \times K \times 3}$, tracking the 3D positions of $K$ object keypoints across $H$ sequential stages. Crucially, the policy only enforces that target coordinates are reached in sequential order, leaving timing and recovery behaviors flexible.
Observed Point Cloud (xt) ──┐
Tracked Keypoints (ut) ───┼─► [Spatio-Temporal Transformer] ─► [Flow-Matching Head] ─► Multi-step Actions
Task Spec Waypoints (c) ───┘ (Interleaved Spatial, (Rectified Flow)
Temporal & Cross-Attention)
The system operates in three main stages:
- Online 3D Point Tracking: Initial keypoint locations are tracked across multi-view stereo cameras using a 2D point tracker (BootsTAP) fused with stereo depth measurements.
- Spatio-Temporal Transformer Policy: The policy architecture encodes the point cloud $x_t$, tracked keypoints $u_t$, and task tensor $c$. It runs an interleaved attention mechanism:
- Temporal self-attention over keypoint trajectories to locate task progress and predict motion.
- Spatial self-attention across keypoints to capture multi-point geometric constraints.
- Cross-attention with the observed 3D point cloud to ground motion plans into the physical scene.
- Flow-Matching Action Head: Action chunks are predicted using a continuous-time diffusion/flow-matching objective (Rectified Flow), capturing multi-modal execution strategies.
- Hindsight Correspondence Training: Training data is collected in simulation using simple randomized primitives (6DoF pick-and-place and tool use). Trajectories are automatically labeled in hindsight by tracking object points. To support variable granularity at test time, trajectories are heavily augmented by randomly subsampling keypoint counts and time horizons, while injecting tracking noise.
Datasets
- Pre-training / Simulation: The policy is trained entirely in the ManiSkill3 simulator using assets from the YCB dataset, PartNet-Mobility, and open-source 3D models spanning manipulable items, containers, and tools.
- Evaluation: Evaluated across three real-world physical robot tasks using a 7-DoF Franka Research 3 arm:
- Pick-and-Place: Placing a baby shoe into an open drawer without moving the drawer.
- Sweeping: Tool use using a brush to sweep a carrot into a dustpan.
- Folding: Manipulating an out-of-distribution deformable scarf.
Key Insights and Ideas
- Granularity-Agnostic Conditioning: By randomly subsampling spatial points ($K$) and temporal steps ($H$) during self-supervised simulation training, the model naturally learns to interpolate. A user can provide just two points (start and goal) or a complete dense trajectory through the same interface.
- Flexible Timing Beats Strict Timestamps: Instead of forcing keypoints to reach coordinates at rigid clock cycles, ordering targets sequentially allows the policy to adjust its execution speed dynamically and recover gracefully if an object slips.
- Sim-to-Real Generalization to Novel Dynamics: Despite being trained only on rigid objects and basic heuristic primitives in simulation, COIL achieves strong zero-shot real-world transfer—even folding deformable cloth—because the policy learns geometry-grounded point transport rather than category-specific behaviors.
Key Results
COIL was compared against state-of-the-art flow/trajectory baselines across three granularity settings: Sparse (3–5 keypoints, 2–5 steps), Medium (8–12 keypoints, 16 steps), and Dense (32 keypoints, 32 steps):
| Method | Pick-and-Place (Sp / Med / De) | Sweeping (Sp / Med / De) | Folding (Sp / Med / De) |
|---|---|---|---|
| RT-Trajectory | 0/10 / 0/10 / 2/10 | 0/10 / 0/10 / 1/10 | 0/10 / 0/10 / 1/10 |
| Im2Flow2Act | 2/10 / 3/10 / 8/10 | 0/10 / 0/10 / 5/10 | 1/10 / 0/10 / 4/10 |
| General-Flow | — / — / 1/10 | — / — / 0/10 | — / — / 6/10 |
| COIL (Ours) | 8/10 / 8/10 / 9/10 | 6/10 / 6/10 / 7/10 | 6/10 / 7/10 / 6/10 |
| VLM + COIL | — / 7/10 / — | — / 5/10 / — | — / 4/10 / — |
- Robustness to Sparsity: Baseline methods failed almost completely under sparse specifications, whereas COIL maintained high success rates (e.g., 80% on sparse Pick-and-Place).
- Ablations: In simulation ablations, removing flow randomization caused the largest drop in success rate (dropping from 82% to 21%), confirming that data augmentation is vital for multi-granularity generalization. Removing the Spatio-Temporal Attention dropped performance to 60%.
Limitations and Future Work
- Tracking Bottlenecks: Real-world failure analysis showed most errors stemmed from point-tracking failures under occlusions or noisy stereo depth estimations.
- External Task Generation: The policy cannot currently detect ambiguity in user intent or self-correct bad prompt trajectories; it executes whatever task representation it receives.
- No Tactile/Force Sensing: Control relies entirely on 3D vision and proprioception. Incorporating force and tactile feedback will be essential for high-precision, contact-rich assembly tasks.
Summary updated