Abstract
Teaching robots to perform complex manipulation tasks by observing human demonstrations is a significant challenge in robotics. While video provides a rich source of information, it often fails to capture "invisible" parameters such as force, pressure, and precise timing. This paper introduces Chain-of-Modality (CoM), a novel prompting strategy that enables Vision-Language Models (VLMs) to transform multimodal human demonstrations—integrating video with muscle activity (EMG) or audio signals—into executable robot programs. By processing modalities sequentially rather than simultaneously, CoM overcomes the common issue of "modality neglect," achieving a 3x improvement in task plan accuracy over standard multimodal baselines.
Introduction
A long-standing goal in robotics is the ability for machines to learn by "watching" humans. However, visual data alone is frequently insufficient for tasks requiring physical interaction. For instance, when inserting a power plug, the visual motion of the hand does not reveal the specific force required to wiggle the plug into place or the firm push needed for final seating.
While state-of-the-art Vision-Language Models (VLMs) excel at high-level reasoning, they struggle with these fine-grained physical details. Furthermore, when presented with multiple data streams (e.g., video, audio, and sensors) simultaneously, these models often suffer from "modality neglect," where they ignore secondary inputs or fail to correlate them correctly across time. Chain-of-Modality addresses these challenges by structuring the VLM's reasoning process into a sequential refinement pipeline.
The Chain-of-Modality (CoM) Framework
The core innovation of this work is the sequential prompting framework. Instead of feeding all multimodal data into a VLM in a single merged batch, CoM guides the model through a four-stage progressive refinement process:
- Force and Audio Analysis: The VLM first analyzes numerical signals, such as EMG muscle activity or audio volume, to identify specific timestamps where effort is exerted. This creates a "coarse skeleton" of the task based on physical exertion.
- Hand Pose Integration: The model then incorporates fingertip tracking data to infer the specific type of action (e.g., "grasping" vs. "twisting") and the direction of movement.
- Visual Grounding: With the action sequence established, the VLM looks at RGB video frames to identify the specific objects involved, such as a bottle cap or a power strip.
- Code Generation: Finally, the refined analysis is translated into a Python program using high-level robot APIs (e.g.,
Grasp(),Twist(), orInsert()).
This "chaining" approach allows the model to use the most certain data—such as a spike in force—to anchor its interpretation of more ambiguous visual data, such as a hand moving near an object.
Experimental Evaluation
The researchers evaluated the CoM framework on four distinct manipulation tasks designed to test different physical parameters:
- Opening a Bottle: Requires precise twisting and grasping coordination.
- Inserting a Plug: Requires varying force levels for orientation and final insertion.
- Wiping a Board: Involves spatial patterns and consistent pressure.
- Playing a Drum: Relies on audio signals to determine the intensity and timing of hits.
The system was tested using 10 videos per task with varying objects and camera angles. For real-world validation, the generated programs were executed on ViperX bi-manual robots and KUKA arms. The evaluation included testing generalization on unseen objects, such as seven different types of bottles, and randomized object placements.
Key Insights and Results
The experimental results highlight the effectiveness of sequential reasoning in multimodal contexts:
- Accuracy Improvements: CoM achieved 60% accuracy in extracting exact task plans and control parameters. In contrast, vision-only methods achieved 0% accuracy (as they could not infer force or timing), and "merged" multimodal baselines reached only 17%.
- Real-World Success: On physical hardware, the system achieved an average 73% success rate across all tasks, demonstrating robust generalization to new objects and environments.
- Force as a Segmenter: A key finding was that force data (EMG) acts as a natural "clock" for the task, clearly marking sub-task boundaries that might look visually identical, such as simply holding a bottle versus actively applying torque to the cap.
- Cross-Embodiment via Code: By generating high-level Python code rather than low-level motor commands, the system remains embodiment-agnostic. The same VLM-generated plan can be executed by different robot platforms as long as they support the same API.
Limitations and Future Work
While CoM represents a significant step forward, several limitations remain. Currently, the model uses simplified audio processing (volume only) and does not reason about frequency or pitch, which could provide deeper insights into mechanical interactions. Additionally, the robot currently operates in an "open-loop" fashion; it generates and runs a program without real-time feedback to correct errors during execution. Future research will explore incorporating tactile sensors and closed-loop feedback to further enhance the system's reliability and breadth of application.
Summary updated