Introduction
Learning from Demonstration (LfD) is a cornerstone of robotics, yet it often requires extensive training data to capture low-level motion trajectories. Furthermore, traditional LfD systems struggle with real-time corrections; minor errors often necessitate complete retraining. Conversely, teaching agents through natural language alone is hindered by the agent's lack of physical grounding. This paper presents a multimodal interactive framework designed to bridge these gaps, enabling "few-shot" learning of primitive actions through a combination of visual demonstrations and active linguistic dialogue.
Methodology
The proposed framework transitions the agent from a passive observer to an active participant. The process involves several key stages:
- Qualitative Spatial (QS) Representations: Instead of raw coordinates, the system abstracts data into human-relatable concepts such as cardinal directions, qualitative distance (e.g., "close" vs. "far"), and trajectory calculus (e.g., "moving clockwise").
- Pattern Mining and Confidence Scoring: The system identifies logical invariants across demonstrations (e.g., "Object B remains static"). A heuristic confidence score is applied, incorporating human spatial biases to align the agent's assumptions with human intent.
- Active Questioning: The agent generates natural language queries (e.g., "Is the green block always stationary?") to verify high-confidence hypotheses. This dialogue reduces the number of required physical demonstrations.
- Action Reenactment: Confirmed patterns serve as constraints for a search algorithm, allowing the agent to execute the learned action in novel environments.
Experimental Setup
The framework was implemented using the VoxML/VoxSim platform. The setup utilizes a Kinect camera to track human gestures and physical objects, while a 3D avatar interacts with the user in real-time. This environment facilitates the grounding of linguistic concepts in physical space.
Results and Discussion
The authors provide a qualitative proof-of-concept demonstrating the system's ability to learn the action "move the red block around the green block." By extracting spatial constraints—such as the stationary nature of the target block and the constant distance maintained by the moving block—the agent successfully generated context-aware questions to solidify its understanding.
A key insight is that qualitative representations provide a more robust bridge between continuous trajectory data and discrete language than quantitative methods. Furthermore, active learning via dialogue significantly lowers the data barrier for complex action acquisition.
Conclusion and Future Work
While the current system relies on pre-segmented data, future iterations aim to process continuous, real-time human performance data. This will involve developing robust noise removal and automatic keyframe detection. Additionally, the researchers plan to explore hierarchical concept recognition, allowing the agent to understand relationships between actions (e.g., "clockwise" as a subclass of "around") and inherit previously learned constraints.
Summary updated