Introduction
Robots often struggle to follow natural language instructions because their sensor-based perception of the world frequently conflicts with human descriptions. For instance, a robot might misidentify a "green apple" as a "pear" due to lighting or sensor noise. Traditionally, resolving such ambiguities requires the robot to stop and ask for clarification. This research proposes a Bayesian learning framework that leverages the implicit spatial information within a command—such as "the apple next to the mug"—to automatically correct perceptual errors without human intervention.
Methodology: Bridging Perception and Language
The proposed architecture is modular, designed to reconcile raw visual data with linguistic instructions through three primary components:
Visual Object Anchoring
Using a Kinect RGB-D sensor, the system segments the scene into object anchors. A fine-tuned GoogLeNet CNN categorizes these objects into 101 kitchen-related classes, extracting attributes like color and position.
Natural Language Grounding via Neural Module Networks (NMNs)
Instead of a monolithic model, the system parses instructions into a computation graph of functional units. As shown in Figure 1, the instruction "pick up the apple to the right of the black mug" is decomposed into modules like Detect (for nouns/adjectives), Shift (for spatial relations), and Locate (to find the target).
Bayesian Belief Revision
The core innovation is treating the human instruction as "evidence" to update the robot's internal beliefs. If the spatial relations in a command only hold true if a mislabeled "pot" is actually a "mug," the Bayesian update re-labels the object anchor. To maintain computational tractability, the system only considers the top two most likely labels for each object.
Experimental Results
The framework was evaluated using both synthetic grid worlds and a real-world setup featuring a Kinova Jaco robotic arm.
Learning Efficiency and Complexity
Synthetic tests revealed a significant disparity in learning difficulty between different linguistic components. While the model mastered nouns and adjectives within approximately 300,000 instances, prepositional phrases (e.g., "next to") required over 1.3 million instances to reach comparable accuracy. This is attributed to the complexity of the Shift modules, which must learn many-to-many spatial relations across the scene.
Real-World Validation
In a physical demonstration, the robot initially misclassified a black mug as a "pot." Upon receiving the instruction "drop it [the ball] in front of the mug," the system utilized the spatial context to correctly identify the "pot" as the intended "mug," successfully completing the manipulation task.
Conclusion and Future Work
This research demonstrates that implicit information in natural language can serve as a powerful corrective signal for robotic perception. Future work aims to expand the system's vocabulary and explore active dialogue, allowing robots to learn new concepts through a combination of implicit reasoning and explicit human feedback.
Summary updated