SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain

Ruochen Zhang, Carsten Eickhoff

Introduction

In Natural Language Processing (NLP), State Tracking is the fundamental task of maintaining an internal representation of a situation as a narrative unfolds. While traditional benchmarks often rely on information-dense data—such as instruction manuals or goal-oriented dialogues where nearly every sentence advances the state—real-world discourse is frequently "messy" and information-sparse.

Researchers from Brown University have introduced SOCCER, a novel dataset designed to evaluate models in these low signal-to-noise environments. Using real-world soccer match commentaries, the dataset highlights a significant challenge: distinguishing critical event-driven updates from the surrounding "chatter."

Overview of the SOCCER dataset task, showing the alignment between commentary "chatter" and discrete game state changes.

The SOCCER Dataset

The SOCCER dataset, compiled from goal.com, comprises 2,263 matches from elite leagues (e.g., Premier League, Serie A) between 2016 and 2020. It contains over 135,000 commentary paragraphs mapped to approximately 31,000 discrete in-game events, including goals, assists, yellow/red cards, and substitutions.

A defining characteristic of SOCCER is its Information Density (ID). While common datasets like MultiWOZ have an ID of 1.05 (roughly one state change per turn), SOCCER’s ID is a mere 0.19. This means the match state only updates roughly every five commentary segments, forcing models to ignore vast amounts of irrelevant context.

Methodology

The authors framed state tracking as both a classification and a generation task across two levels of granularity:

  1. Team Level: A binary task identifying if a specific event occurred for the Home or Guest team.
  2. Player Level: A more complex task requiring the identification of the specific player involved (e.g., "Who provided the assist?").

To establish baselines, two primary architectures were evaluated:

  • BERT + GRU: A discriminative approach using BERT for sentence embeddings and a Gated Recurrent Unit (GRU) to model temporal dependencies.
  • GPT-2: A generative approach that treats state tracking as a sequence generation problem, producing structured state strings (e.g., goal_home: Joao Mario) directly from the commentary.
Comparison of the baseline architectures: the discriminative BERT+GRU model (left) and the generative GPT-2 model (right).

Key Results and Insights

The study revealed that standard accuracy metrics are misleading in sparse domains. Because events are rare, a model predicting "no change" achieves >97% accuracy. Consequently, the researchers focused on Positive Recall—the ability to detect events when they actually occur.

  • The Performance Gap: At the team level, GPT-2 achieved a Positive Recall of 48.5%, while the GRU reached 39.9%. Both models missed more than half of the actual events.
  • The Player-Level Challenge: Performance plummeted at the player level, with GPT-2’s recall dropping to 7.7%. The "open-vocabulary" nature of player names—often referred to by nicknames or pronouns—makes mapping to specific entities extremely difficult.
  • Sparsity Sensitivity: Controlled experiments showed that as "chatter" increases, a model's ability to recognize true state updates drops sharply, even as its overall accuracy appears to improve.

Conclusion

The SOCCER dataset exposes the limitations of current state-of-the-art models in information-sparse environments. Future progress will likely require moving beyond simple sequence modeling toward entity-centric reasoning and better handling of long-range dependencies. SOCCER provides a rigorous playground for developing AI that can truly "understand" the flow of real-world discourse.

Summary updated

BibTeX
@inproceedings{zhang2021soccer,
  author = {Ruochen Zhang and Carsten Eickhoff},
  title = {{SOCCER: An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.342},
}
Figure