TL;DR
The Demonstration Expansion and Execution Design (DEED) framework significantly improves fine-grained robotic manipulation by addressing data scarcity and execution mismatches. By pairing multiple instructions with the same scene and optimizing task-specific control parameters like execution horizons and gripper mappings, DEED achieved an 85.55% score on the MetaFine benchmark, nearly doubling the performance of the state-of-the-art $\pi_0$ baseline.
What Problem Does This Paper Address?
While Vision-Language-Action (VLA) models have shown impressive general-purpose capabilities, they often stumble when tasks require "fine-grained" precision—think of inserting a specific letter into a slot or toggling a small switch.
The authors identify three critical gaps in current VLA training:
- Scene-Instruction Confounding: If a specific instruction is always seen in a specific scene, the model might learn to associate the background with the task rather than truly understanding the command.
- Visual Vulnerability: Models trained on limited camera angles and lighting conditions fail when the environment changes slightly.
- The Execution Gap: There is often a mismatch between the "action chunks" a model predicts and how a robot actually needs to move (e.g., gripper semantics or the frequency of replanning).
Methods
DEED tackles these problems through a two-pronged approach: Demonstration Expansion and Execution Design.
1. Demonstration Expansion
To break the reliance on scene shortcuts, the authors used Instruction Pairing. They generated multiple different instructions (and their corresponding successful trajectories) starting from the exact same initial scene layout. This forces the model to attend to the specific objects mentioned in the text rather than the environment.
Furthermore, they used Physics-Consistent Visual Expansion. By re-rendering successful trajectories in the ManiSkill3 simulator, they created thousands of new training examples with randomized camera positions and lighting conditions while keeping the underlying physical states and actions identical.
2. Execution Design
The authors argue that how you run the model is just as important as how you train it. Their deployment pipeline includes:
- Receding-Horizon Control: Instead of executing a full predicted "chunk" of actions, the robot executes only the first $H$ steps before taking a new observation and replanning.
- Task-Specific Gripper Mapping: Some tasks require continuous gripper control, while others benefit from a binary "full closure" threshold. DEED tunes these thresholds ($\theta_g$) per task.
- Inverse Kinematics (IK): Converting end-effector predictions into precise joint increments for the robot.
Datasets
The framework was evaluated on the MetaFine benchmark, which consists of five challenging tasks:
- T1: Part-level grasping (cap vs. body).
- T2: Directional motion (forward/left/right).
- T3: Switch toggling (red vs. blue).
- T4: Object sorting (red/blue/green cubes).
- T5: Letter insertion (C/o/R/L).
The authors used the Xiaomi-Robotics-1 model as their pretrained base and expanded the provided MetaFine demonstrations into a much larger pool (e.g., T5 grew to 7,840 records).
Key Insights and Ideas
- Data Quality > Data Quantity: Simply adding more data isn't enough. The structure of the data—specifically pairing different instructions in the same visual context—is what prevents the model from taking "lazy" visual shortcuts.
- The "H" Factor: The execution horizon ($H$) is a critical hyperparameter. Executing too many steps without re-observing leads to drift, but executing too few can lead to jittery, inefficient motion.
- Visual Robustness is Free (in Sim): By using a simulator to re-render trajectories, you can gain massive robustness to lighting and camera shifts without needing to collect new expert demonstrations.
Key Results
DEED achieved an official Overall score of 85.55%, a massive leap over the 43.01% baseline.
As shown in the leaderboard comparison, DEED achieved near-perfect scores on T2, T3, and T4. The most significant gains were seen in T5 (Letter Insertion), where the baseline scored a mere 4.17% compared to DEED's 64.55%.
The authors also provided a detailed breakdown of the task-specific settings that led to these results: Table showing the records, execution horizon (H), denoising steps (D), and gripper thresholds used for each task.
Limitations and Future Work
While DEED dominates the benchmark, two tasks (T1 and T5) remain significantly more difficult than the others, with scores in the mid-60s. These tasks involve the most complex geometry and tightest tolerances.
The authors also note that their execution parameters (like the horizon $H$ and gripper threshold $\theta_g$) were selected through task-specific evaluation. A promising direction for future work would be developing a policy that can dynamically adjust its own execution strategy based on the perceived difficulty or state of the task, rather than relying on manual tuning. Additionally, while the results in simulation are stellar, the "sim-to-real" gap for such fine-grained tasks remains a perennial challenge for the field.
Summary updated