Reasoning about RoboCup Soccer Narratives

Hannaneh Hajishirzi, Julia Hockenmaier, Erik T. Mueller, Eyal Amir

TL;DR

This paper introduces ITEM (Iterative Event Mapping), a framework that learns to translate natural language narratives—specifically soccer commentaries—into structured event sequences without requiring any labeled training data. By leveraging a small amount of domain knowledge (preconditions and effects of actions), the system uses an iterative, EM-like learning process to outperform state-of-the-art supervised models.

What Problem Does This Paper Address?

Understanding a narrative isn't just about translating individual sentences; it’s about building a coherent "discourse model" of the world. In a soccer game, if a commentator says, "Pink goalie kicks off to Pink2," a human listener automatically infers that Pink2 now has possession of the ball.

The challenge in AI is mapping these messy, human-written sentences to a formal sequence of events (like pass, kick, or steal). Most existing approaches rely on supervised learning, which requires expensive, manually annotated datasets where every sentence is paired with a ground-truth event. The authors tackle the "cold start" problem: How can we learn to interpret these narratives using only the text itself and a basic understanding of how the world (or the game) works?

Methods

The authors propose the ITEM algorithm, which rests on two main pillars:

  1. Domain Knowledge as a Logical Bias: Instead of labels, the authors provide the system with a small set of STRIPS-like axioms. These define the preconditions and effects of 16 soccer event types. For example, a player cannot kick the ball unless they are holding it. This "physical coherence" acts as a filter to discard impossible interpretations.
  2. Iterative Learning (IterTrain): This is a "self-training" approach reminiscent of Hard Expectation-Maximization (EM).
  3. Initialization: The system makes a "best guess" for labels based on simple heuristics (like the edit distance between words in a sentence and the name of an event) and whether the event is physically possible in the current state.
  4. The Loop: It trains a binary logistic regression classifier on these guesses, uses the classifier to re-score and update the labels, and repeats until the model parameters converge.
  5. Inference via Dynamic Programming: At test time, the system doesn't just look at sentences in isolation. It uses a Viterbi-like algorithm to find the sequence of events that maximizes "utility" (classifier score) while penalizing sequences that violate physical constraints (preconditions).

Datasets

The study uses the RoboCup simulation league commentaries dataset.

  • Content: Human-written English commentaries from four championship games (2001–2004).
  • Scale: 1,872 total comments.
  • Evaluation: While the model is trained without labels, the authors use a "gold standard" event log from the original dataset to measure accuracy.
  • Domain Details: The world model includes 24 player constants, 10 state predicates (like holding(player, ball)), and 16 event types.

Key Insights and Ideas

  • Coherence is a Strong Supervisor: The most striking insight is that the "logic of the world" can replace human annotators. Because narratives describe dynamic systems where actions have consequences, the requirement for temporal and physical consistency provides enough signal to guide a learning algorithm.
  • Handling "Noise" is Critical: Real-world commentary includes sentences like "Today we have a nice match!" which don't map to any game event. The authors included a "Nothing" event with no preconditions or effects, allowing the model to gracefully ignore irrelevant text.
  • State-Aware Classification: Unlike standard NLP models that only look at the text, the ITEM classifier takes the current state of the world as an input feature. This allows the model to learn that the same sentence might mean different things depending on who has the ball.

Key Results

  • Superior Accuracy: ITEM achieved a micro-averaged accuracy of 77.9% across the four games.
  • Beating Supervision: Remarkably, this unsupervised approach outperformed the previous state-of-the-art supervised system (WGIM), which achieved 70.3% despite having access to annotated training data.
  • The Power of Inference: Even when the authors added their "physical consistency" inference logic to the old supervised WGIM model, it only reached 73.4%, suggesting that ITEM’s iterative learning process is fundamentally better at capturing the relationship between language and world states.

Limitations and Future Work

  • Linearity Assumption: The model assumes a strictly linear sequence of events. It cannot currently handle complex rhetorical structures like flashbacks ("He scored because he stole the ball earlier") or elaborations.
  • Simple Linguistics: The system uses a basic bag-of-words model. The authors suggest that incorporating syntactic parsing (understanding sentence structure) could improve results for more complex narratives.
  • Scaling Up: The current domain is small (16 events). The authors plan to scale this to broader narratives using VerbNet, a massive lexicon containing semantic information for approximately 5,000 verbs.
  • Missing Events: Sometimes commentators skip events. The authors propose using partial-order planning to "fill in the blanks" for events that must have happened but weren't mentioned.

Summary updated

Abstract

This paper presents an approach for learning to translate simple narratives, i.e., texts (sequences of sentences) describing dynamic systems, into coherent sequences of events without the need for labeled training data. Our approach incorporates domain knowledge in the form of preconditions and effects of events, and we show that it outperforms state-of-the-art supervised learning systems on the task of reconstructing RoboCup soccer games from their commentaries.
BibTeX
@inproceedings{hajishirzi2011reasoning,
  author = {Hannaneh Hajishirzi and Julia Hockenmaier and Erik T. Mueller and Eyal Amir},
  title = {{Reasoning about RoboCup Soccer Narratives}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=YmyB2RdRsX},
}

Loading related content...

Figure