Introduction
Interactive Task Learning (ITL) aims to develop robots capable of acquiring new knowledge through real-time human interaction. A core challenge in ITL is symbol grounding—mapping linguistic terms like "magenta" or "behind" to visual features. While traditional methods often require large datasets or simple labels, humans naturally use Complex Referential Expressions (REs), such as "the blue block next to the red one." This paper introduces IGRE (Interactive Symbol Grounding with Referential Expressions), a framework designed to learn from such complex descriptions incrementally and efficiently.
Methodology: The IGRE Framework
IGRE addresses the limitations of purely neural approaches by combining neural perception with symbolic logic. The system's architecture, detailed in Figure 1, bridges the gap between low-level visual features and high-level linguistic constraints.
The framework consists of several key components:
- Neural Few-Shot Grounder: Utilizes Matching Networks to relate "open-class" words (nouns, adjectives, relations) to visual percepts, enabling learning from minimal examples.
- Symbolic Reasoning: Employs Formal Semantics for "closed-class" words (quantifiers like "every" or "both"). These have fixed mathematical definitions that serve as inductive biases.
- Logical Theory Update ($Δ$): Parses human utterances into logical forms, building a cumulative "theory" of the scene.
- Model Counting (MC): Uses the ADDMC model counter to calculate the probability of object attributes based on logical constraints. This allows the system to infer properties of objects not explicitly pointed to by the teacher (oblique information).
Experimental Evaluation
The researchers evaluated IGRE using ShapeWorld, a synthetic environment with various shapes, colors, and spatial relations. The system was compared against two baselines: HEAD (learning only from the head noun) and EXIST (treating all descriptions as simple existential statements).
The evaluation focused on two metrics:
- Intrinsic Performance: Accuracy in predicting the overall domain state (e.g., identifying all red objects).
- Extrinsic Performance: Accuracy in identifying the specific objects the teacher referred to.
Key Results
IGRE demonstrated significant improvements in both data efficiency and accuracy. As shown in Figure 3, IGRE (represented by the green line) consistently outperforms the baselines as the number of referential expressions increases.
By leveraging the logic of quantifiers, IGRE effectively generates "noisy" training examples for background objects, providing the necessary negative exemplars that stabilize neural learning. Notably, IGRE achieved an F1 score of 0.16 for referent prediction, nearly triple that of the HEAD baseline (0.06). It also showed exceptional proficiency in learning spatial relations, reaching perfect recall in intrinsic evaluations.
Conclusion
The IGRE framework proves that symbolic logic can significantly enhance neural symbol grounding by providing structured constraints and enabling "oblique" learning. While computational complexity in model counting remains a challenge for scaling to extremely complex scenes, the hybrid approach offers a promising path toward more data-efficient and natural human-robot interaction. Future work will explore dynamic semantics and integration into physical robotic platforms.
Summary updated