Learning to Connect Language and Perception

Raymond J. MooneyAAAI 2008

Abstract

This paper explores the challenge of "symbol grounding"—bridging the gap between abstract linguistic symbols and physical perception. By training a system to sportscast simulated soccer games through observation of human commentary and game states, the research demonstrates that machines can learn to map language to world events even in noisy, ambiguous environments.

Introduction: The Symbol Grounding Problem

Most Natural Language Processing (NLP) and Computer Vision (CV) systems operate in silos, lacking "symbol grounding." While NLP models process text and CV models label images, they often fail to understand the physical reality behind words. A primary obstacle is referential uncertainty: in a dynamic environment, multiple events occur simultaneously, making it difficult for an AI to determine which specific world event corresponds to a given phrase—a "blooming, buzzing confusion" that must be resolved for effective learning.

Grounded Language Learning Framework

The research utilizes two primary systems to address grounded language learning:

  • KRISPER: Designed for weakly-supervised, ambiguous training data, KRISPER uses an iterative, Expectation-Maximization (EM)-like approach. It initially considers all possible event-sentence pairings and gradually filters out noise by scoring pairings based on its current best "guess," eventually converging on correct mappings.
  • WASP: This system treats language grounding as a Statistical Machine Translation (SMT) problem, translating natural language (NL) into a formal Meaning Representation (MR). Its use of a synchronous context-free grammar allows the model to be symmetric, enabling both parsing (understanding) and generation (speaking).
  • Strategic Generation: Beyond linguistic mapping, the system must decide what to say. It learns to identify "noteworthy" game events that warrant commentary by analyzing patterns in the training data.

Experimental Setup: Robocup Simulation League

The primary testbed for this framework is the Robocup Simulation League. Researchers utilized human-provided "play-by-play" commentaries from Robocup finals (2001-2004). The system was trained on three games and evaluated on a fourth. Using a simulator allowed the team to bypass the complexities of raw pixel processing while retaining the dynamic, multi-agent, and noisy nature of a real-world environment.

Figure 1: A snapshot of the Robocup Simulation League environment showing players and the corresponding human-provided commentary.

Key Results and Insights

Human judges evaluated the machine-generated sportscasts against human commentary on a 5-point scale. The AI's performance was remarkably close, receiving scores only 0.7 points lower than human counterparts. This demonstrates the feasibility of grounded language learning from limited, realistic training data.

The core insight is that ambiguity is a feature, not a bug. By using iterative algorithms like KRISPER, the system can "bootstrap" its way to understanding by observing the same words used across different ambiguous contexts.

Limitations and Future Directions

While successful, the current research uses symbolic descriptions of the game state (e.g., player coordinates) rather than raw video pixels. Future work must integrate computer vision to extract these symbols automatically. Additionally, the problem of mapping sentences to one of many possible meanings represents a form of "Multiple Instance Learning (MIL) for structured output," a promising area for further exploration.

Conclusion

This work advocates for a "developmental" approach to AI, where systems learn from rich perceptual environments rather than static, hand-labeled datasets. Grounded language learning serves as a vital venue for re-integrating the disparate sub-fields of AI—NLP, CV, and Robotics—into a cohesive framework for true machine understanding.

Summary updated

BibTeX
@inproceedings{mooney2008learning,
  author = {Raymond J. Mooney},
  title = {{Learning to Connect Language and Perception}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure