SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain

Ruochen Zhang, Carsten Eickhoff

Introduction: The Challenge of Sparse State Tracking

In Natural Language Processing (NLP), State Tracking involves maintaining an internal representation of a situation as a narrative unfolds. Traditionally, this has been studied in information-dense domains like dialogue systems (e.g., booking a hotel) or procedural texts (e.g., following a recipe), where nearly every sentence corresponds to a state change.

However, real-world discourse is often "information-sparse." In a sports broadcast, a commentator might engage in minutes of "chatter"—discussing a player's history or tactical nuances—before a significant event like a goal or substitution occurs. Researchers at Brown University have introduced SOCCER, a new dataset designed to benchmark how well AI can navigate these low signal-to-noise environments.

The SOCCER Dataset

The SOCCER dataset is curated from over 2,200 professional soccer matches (including the Premier League and Serie A) between 2016 and 2020. It comprises over 135,000 commentary paragraphs and 31,000 discrete in-game events.

Figure 1: An illustration of the SOCCER task, mapping live commentary to discrete game states across a match timeline.

The defining characteristic of SOCCER is its Information Density (ID). While traditional state-tracking datasets like MultiWOZ have an ID of approximately 1.05, SOCCER’s ID is just 0.19. This means events are rare, and models must learn to ignore irrelevant "chatter" to identify the few sentences that actually signal a state change.

Methodology: Tracking at Scale

The authors frame the problem as a Discourse State Tracking task with two levels of granularity:

  1. Team Level: A binary classification task determining if a specific event (e.g., a goal) happened for a team in a given minute.
  2. Player Level: A more complex task requiring the system to identify the specific entity involved in the event.

To establish baselines, the study evaluated two distinct architectures:

  • GRU Classifier + BERT: A discriminative model that uses BERT for text embedding and a Gated Recurrent Unit (GRU) to track temporal dependencies, outputting a vector of event types.
  • GPT-2 Based Variant: A generative approach where the model is prompted with commentary to generate a sequence of events and player names.
Figure 6: Comparison of the two baseline architectures: the discriminative BERT+GRU model (left) and the generative GPT-2 model (right).

Key Results and the "Chatter" Effect

The experiments revealed that while models can achieve high raw accuracy, they struggle significantly with Positive Recall—the ability to actually find the rare events.

  • The Sparsity Trap: Because 97.6% of timestamps contain no events, a model can reach 97% accuracy by simply predicting that nothing happened. However, the GPT-2 variant’s recall dropped from 48.5% at the team level to a mere 7.7% at the player level.
  • Noise Sensitivity: As the amount of "chatter" (non-event text) increases, model performance drops steadily. Current architectures are easily confused by descriptions of "near misses" or general tactical discussion that mimic the language of actual events.
  • Rare Event Blindness: Generative models failed entirely on infrequent events like red cards, highlighting a need for better handling of long-tail data in sparse streams.

Conclusion and Future Directions

The SOCCER benchmark demonstrates that current state-of-the-art NLP models are not yet robust enough for real-world, information-sparse discourse. Future work will need to focus on improved temporal reasoning and better entity-linking to bridge the gap between sensing an event and identifying its participants. Such advancements could eventually power automated sports journalism and more sophisticated live-blogging assistants.

Reference: Zhang, R., & Eickhoff, C. (2021). SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain. Proceedings of NAACL-HLT 2021.

Summary updated

Abstract

In the pursuit of natural language understanding, there has been a long standing interest in tracking state changes throughout narratives. Impressive progress has been made in modeling the state of transaction-centric dialogues and procedural texts. However, this problem has been less intensively studied in the realm of general discourse where ground truth descriptions of states may be loosely defined and state changes are less densely distributed over utterances. This paper proposes to turn to simplified, fully observable systems that show some of these properties: Sports events. We curated 2,263 soccer matches including time-stamped natural language commentary accompanied by discrete events such as a team scoring goals, switching players or being penalized with cards. We propose a new task formulation where, given paragraphs of commentary of a game at different timestamps, the system is asked to recognize the occurrence of in-game events. This domain allows for rich descriptions of state while avoiding the complexities of many other real-world settings. As an initial point of performance measurement, we include two baseline methods from the perspectives of sentence classification with temporal dependence and current state-of-the-art generative model, respectively, and demonstrate that even sophisticated existing methods struggle on the state tracking task when the definition of state broadens or non-event chatter becomes prevalent.
BibTeX
@inproceedings{zhang2021soccer,
  author = {Ruochen Zhang and Carsten Eickhoff},
  title = {{SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.342},
}
Figure