Reasoning about RoboCup Soccer Narratives

Hannaneh Hajishirzi, Julia Hockenmaier, Erik T. Mueller, Eyal Amir

Abstract

This paper presents an approach for learning to translate simple narratives, i.e., texts (sequences of sentences) describing dynamic systems, into coherent sequences of events without the need for labeled training data. Our approach incorporates domain knowledge in the form of preconditions and effects of events, and we show that it outperforms state-of-the-art supervised learning systems on the task of reconstructing RoboCup soccer games from their commentaries.

Introduction

Understanding a narrative involves more than just translating individual sentences; it requires building a coherent discourse model of the world. In a soccer game, for example, a commentary like "Pink goalie kicks off to Pink2" implies a change in ball possession that must be reflected in the model. Mapping these natural language descriptions to formal event sequences (e.g., pass, kick, or steal) is a significant challenge in AI, particularly because traditional supervised methods require expensive, manually annotated datasets.

This research addresses the "cold start" problem: learning to interpret narratives using only the text itself and a basic understanding of the domain's physical rules. The authors introduce ITEM (Iterative Event Mapping), a framework that leverages domain knowledge to guide an unsupervised learning process.

Methods

The ITEM algorithm is built upon two primary pillars: logical domain knowledge and iterative self-training.

Domain Knowledge as a Logical Bias

Instead of human-provided labels, the system uses a small set of STRIPS-like axioms that define the preconditions and effects of 16 soccer event types. For instance, a player cannot kick the ball unless they are currently holding it. This requirement for "physical coherence" acts as a filter, allowing the system to discard interpretations that are logically impossible within the game state.

Iterative Learning (IterTrain)

ITEM employs a "self-training" approach similar to Hard Expectation-Maximization (EM). The process begins by making initial guesses for labels based on simple heuristics, such as the edit distance between words and event names, while ensuring physical possibility. A binary logistic regression classifier is then trained on these guesses. The system iteratively re-scores and updates the labels using the classifier's output, repeating the loop until the model parameters converge.

The ITEM framework architecture, illustrating the flow from sentences and prior knowledge to iterative training and final inference.

Inference via Dynamic Programming

During the test phase, the system uses a Viterbi-like algorithm to determine the event sequence that maximizes the classifier's utility score while strictly adhering to physical constraints. This ensures that the final output is not only linguistically plausible but also physically consistent.

Results and Evaluation

The system was evaluated using human-written English commentaries from four RoboCup simulation league championship games (2001–2004), totaling 1,872 comments. The world model included 24 player constants, 10 state predicates, and 16 event types.

ITEM achieved a micro-averaged accuracy of 77.9% across the four games. Remarkably, this unsupervised approach outperformed the previous state-of-the-art supervised system, WGIM, which achieved only 70.3% despite having access to annotated training data.

Performance comparison of ITEM against various baselines and the supervised WGIM system across four RoboCup datasets.

As shown in the results table, even when the authors' physical consistency inference logic was applied to the supervised WGIM model (WGIM-Inference), it only reached 73.4%. This suggests that ITEM’s iterative learning process is fundamentally superior at capturing the relationship between natural language and world states.

Conclusion and Future Work

The core insight of this work is that the "logic of the world" can effectively replace human annotators in narrative understanding tasks. By enforcing temporal and physical consistency, the ITEM framework provides a robust method for learning from unlabeled text.

Future research will focus on moving beyond linear sequences to handle complex rhetorical structures like flashbacks, incorporating deeper syntactic parsing, and scaling the domain knowledge using massive semantic lexicons like VerbNet.

Summary updated

Abstract

This paper presents an approach for learning to translate simple narratives, i.e., texts (sequences of sentences) describing dynamic systems, into coherent sequences of events without the need for labeled training data. Our approach incorporates domain knowledge in the form of preconditions and effects of events, and we show that it outperforms state-of-the-art supervised learning systems on the task of reconstructing RoboCup soccer games from their commentaries.
BibTeX
@inproceedings{hajishirzi2011reasoning,
  author = {Hannaneh Hajishirzi and Julia Hockenmaier and Erik T. Mueller and Eyal Amir},
  title = {{Reasoning about RoboCup Soccer Narratives}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=YmyB2RdRsX},
}
Figure