TL;DR
This paper details a third-place submission to the MetaFine Open Competition (CoRL 2026), focusing on fine-grained robotic manipulation. By combining a shared vision-language architecture with task-specific data refinement and inference strategies—such as uncertainty-triggered replanning and contact-aware filtering—the authors achieved an impressive overall score of 0.914 across five challenging manipulation tasks.
What Problem Does This Paper Address?
Fine-grained manipulation is a "boss level" challenge in robotics. It requires a robot to not only understand natural language instructions (instruction grounding) but also to perceive objects accurately under varying light and viewpoints, and execute movements with sub-centimeter precision (contact control).
The authors tackle the MetaFine benchmark, which consists of five distinct tasks:
- T1: Part Grasping (e.g., "grasp the handle")
- T2: Mug Transport
- T3: Switch Manipulation
- T4: Block Placement
- T5: Letter Insertion (the most difficult due to narrow tolerances)
The core challenge is that a "one-size-fits-all" model often fails to capture the nuances of these different tasks, leading to issues like stalling during tight insertions or failing to generalize across visual perturbations.
Methods
The team utilized a modular, "shared-yet-specialized" approach.
Architecture
The backbone is a multimodal Transformer that processes:
- Vision: Frozen DINOv2 features from base and wrist cameras.
- Language: Frozen CLIP text embeddings.
- Fusion: Feature-wise Linear Modulation (FiLM) layers that allow the language instruction to dynamically adjust the visual features.
- Action Heads: Depending on the task, the model uses either direct L1 regression or Conditional Flow Matching (FM) to generate 32-step action chunks.
Inference Strategies
The authors introduced several clever tricks at test-time:
- Photometric TTA: For grasping and switches, they run the model on eight different brightness/contrast variations of the same image and pick the "medoid" (most representative) action.
- Uncertainty-Triggered Replanning: For tasks like letter insertion, the model generates multiple action samples. If the samples disagree significantly (high uncertainty), the robot truncates its current action and requests a new plan from a fresh observation.
Datasets
The models were trained on the MetaFine competition datasets, but the authors performed significant data refinement:
- Contact-Aware Filtering (T1): They removed demonstrations involving premature contacts or collisions.
- DAgger-style Correction (T5): For the difficult letter insertion task, they used expert corrections to teach the model how to recover when it stalled near a slot.
- Augmentation: They used camera-view and photometric augmentations to ensure the models weren't brittle to lighting or viewpoint changes.
Key Insights and Ideas
- Task Specialization Wins: Even with a shared architecture, tailoring the action head (Regression vs. Flow Matching) and the control space (End-Effector vs. Joint Delta) to the specific physics of the task was vital.
- Replanning is a Double-Edged Sword: While uncertainty-triggered replanning was a game-changer for high-precision tasks like letter insertion, it actually hurt performance in simpler tasks like block placement by disrupting motion continuity.
- Data Quality > Quantity: Filtering out "noisy" demonstrations where the robot bumped into things was more effective than simply scaling up the dataset.
Key Results
The submission achieved an overall score of 0.914, ranking 3rd out of 15 teams.
- Near-Perfection: The model achieved perfect or near-perfect scores on Block Placement (1.000), Switch Manipulation (0.996), and Mug Transport (0.989).
- The Power of Replanning: In T5 (Letter Insertion), replanning boosted the success rate from 55% to 95%. For the particularly difficult "R" letter, success jumped from 30% to 90%.
Limitations and Future Work
- Hardest Task: T1 (Part Grasping) remains the weakest link (score: 0.762), suggesting that fine-grained semantic understanding of object parts is still a major hurdle.
- Final Seating: In letter insertion, failures still occur during the very last stage of "seating" the object, which might require even higher-frequency tactile feedback or specialized contact-rich policies.
- Adaptive Strategies: The authors suggest that future work should focus on more sophisticated ways to decide when to replan, rather than using a fixed uncertainty threshold.
Summary updated