Task-Specialized Language-Conditioned Models for Fine-Grained Robotic Manipulation

Chanho Kim, Soyun ChoiarXiv 2026 · arXiv

TL;DR

This paper details a third-place submission to the MetaFine Open Competition (CoRL 2026), focusing on fine-grained robotic manipulation. By combining a shared vision-language architecture with task-specific data refinement and inference strategies—such as uncertainty-triggered replanning and contact-aware filtering—the authors achieved an impressive overall score of 0.914 across five challenging manipulation tasks.

What Problem Does This Paper Address?

Fine-grained manipulation is a "boss level" challenge in robotics. It requires a robot to not only understand natural language instructions (instruction grounding) but also to perceive objects accurately under varying light and viewpoints, and execute movements with sub-centimeter precision (contact control).

The authors tackle the MetaFine benchmark, which consists of five distinct tasks:

  1. T1: Part Grasping (e.g., "grasp the handle")
  2. T2: Mug Transport
  3. T3: Switch Manipulation
  4. T4: Block Placement
  5. T5: Letter Insertion (the most difficult due to narrow tolerances)

The core challenge is that a "one-size-fits-all" model often fails to capture the nuances of these different tasks, leading to issues like stalling during tight insertions or failing to generalize across visual perturbations.

Methods

The team utilized a modular, "shared-yet-specialized" approach.

Architecture

The backbone is a multimodal Transformer that processes:

  • Vision: Frozen DINOv2 features from base and wrist cameras.
  • Language: Frozen CLIP text embeddings.
  • Fusion: Feature-wise Linear Modulation (FiLM) layers that allow the language instruction to dynamically adjust the visual features.
  • Action Heads: Depending on the task, the model uses either direct L1 regression or Conditional Flow Matching (FM) to generate 32-step action chunks.
Figure 1: Overall architecture of our approach. Frozen DINOv2 and CLIP encoders provide language-conditioned visual features through trainable FiLM modules. A multimodal Transformer drives action heads that produce end-effector targets for T1–T4 or joint-space actions for T5.

Inference Strategies

The authors introduced several clever tricks at test-time:

  • Photometric TTA: For grasping and switches, they run the model on eight different brightness/contrast variations of the same image and pick the "medoid" (most representative) action.
  • Uncertainty-Triggered Replanning: For tasks like letter insertion, the model generates multiple action samples. If the samples disagree significantly (high uncertainty), the robot truncates its current action and requests a new plan from a fresh observation.

Datasets

The models were trained on the MetaFine competition datasets, but the authors performed significant data refinement:

  • Contact-Aware Filtering (T1): They removed demonstrations involving premature contacts or collisions.
  • DAgger-style Correction (T5): For the difficult letter insertion task, they used expert corrections to teach the model how to recover when it stalled near a slot.
  • Augmentation: They used camera-view and photometric augmentations to ensure the models weren't brittle to lighting or viewpoint changes.
Table 1: Task-specific training and deployment settings. Episode counts include rendered camera views before applying training masks.

Key Insights and Ideas

  • Task Specialization Wins: Even with a shared architecture, tailoring the action head (Regression vs. Flow Matching) and the control space (End-Effector vs. Joint Delta) to the specific physics of the task was vital.
  • Replanning is a Double-Edged Sword: While uncertainty-triggered replanning was a game-changer for high-precision tasks like letter insertion, it actually hurt performance in simpler tasks like block placement by disrupting motion continuity.
  • Data Quality > Quantity: Filtering out "noisy" demonstrations where the robot bumped into things was more effective than simply scaling up the dataset.

Key Results

The submission achieved an overall score of 0.914, ranking 3rd out of 15 teams.

  • Near-Perfection: The model achieved perfect or near-perfect scores on Block Placement (1.000), Switch Manipulation (0.996), and Mug Transport (0.989).
  • The Power of Replanning: In T5 (Letter Insertion), replanning boosted the success rate from 55% to 95%. For the particularly difficult "R" letter, success jumped from 30% to 90%.
Figure 2: Task-dependent effects of replanning. Episode success rates with and without uncertainty- triggered replanning for (a) T4 block placement and (b) T5 letter insertion.
Table 2: Official MetaFine evaluation results. Scores are rounded for display.

Limitations and Future Work

  • Hardest Task: T1 (Part Grasping) remains the weakest link (score: 0.762), suggesting that fine-grained semantic understanding of object parts is still a major hurdle.
  • Final Seating: In letter insertion, failures still occur during the very last stage of "seating" the object, which might require even higher-frequency tactile feedback or specialized contact-rich policies.
  • Adaptive Strategies: The authors suggest that future work should focus on more sophisticated ways to decide when to replan, rather than using a fixed uncertainty threshold.

Summary updated

Abstract

Fine-grained robotic manipulation requires instruction grounding, robust perception, and precise contact control. We describe our submission to the MetaFine Open Competition at CoRL 2026. Five independently trained models use a common architecture with frozen DINOv2 and CLIP encoders, trainable language-conditioned feature modulation, and task-specific action heads. We improve demonstrations through contact-aware grasp filtering, instruction-matched data, and expert correction of insertion failures. At inference, task-dependent strategies include photometric test-time augmentation, stochastic action averaging, and uncertainty-triggered replanning. Our submission achieves an official overall score of 0.914 and ranks third among 15 teams. We analyze the models and their task-specific behavior.
BibTeX
@misc{kim2026task,
  author = {Chanho Kim and Soyun Choi},
  title = {{Task-Specialized Language-Conditioned Models for Fine-Grained Robotic Manipulation}},
  year = {2026},
  eprint = {2611.00955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00955},
}
Figure