TL;DR
StreamSoccer introduces an event-driven memory system designed for real-time, streaming soccer commentary across multiple temporal scales (current, recent, and historical). By organizing video history into discrete, bounded "event lifecycles" rather than raw frames or token caches, it achieves state-of-the-art commentary quality while maintaining a constant, faster-than-real-time computational footprint over a full 90-minute match.
What Problem Does This Paper Address?
Generating live sports commentary requires an AI system to process a continuous video stream causally—meaning it can only use information available up to the current moment—while simultaneously handling diverse temporal contexts. A human commentator naturally switches between describing a foul that just happened (current event), summarizing the flow of play over the last few minutes (recent window), or recalling a specific corner kick from early in the first half (historical memory).
Standard streaming Video-Language Models (VLMs) compress history using task-agnostic frame or token caches. As the video progresses, these caches either overflow their context windows or suffer from massive computational slowdowns, making them impractical for long broadcasts. StreamSoccer addresses this gap by structuring video history around semantic event units that evolve, close upon completion, and persist as retrievable records.
Methods
StreamSoccer utilizes a multi-layered, event-driven architecture to manage the lifecycle of match events:
- Streaming Event Memory Encoder: Maps incoming video clips to a fixed budget of clip-level event tokens. An Operational Transition Head compares new clips against the active memory to predict event boundaries (e.g., identifying when play stops or a set piece occurs). A 24-second maximum duration safeguard ensures the active state remains bounded.
- Memory-to-Record Consolidation: When an event closes, its compact latent state is stored in a local buffer for immediate use. Concurrently, a projector and language model asynchronously convert the closed event into a textual Historical Event Record containing a generated caption and structured metadata.
- Multi-Context Commentary Generator: A deterministic, rule-assisted scheduler decides whether the system should speak or remain silent based on cooldowns and event proximity. If it speaks, a unified generator combines the active event memory, recent latent buffers, and retrieved text records into a single language model prompt using LoRA adaptation.
Datasets
The authors constructed a new three-track streaming soccer commentary dataset by aligning human-annotated SoccerNet action timestamps with MatchTime commentary descriptions. The dataset contains 27,639 samples (15,189 current-event, 7,127 recent-window, and 5,323 historical-memory examples) across 422 matches. To ensure rigorous evaluation, the dataset uses strict match-disjoint splits (70% train, 15% validation, 15% test) and automated checks to prevent future data leakage.
Key Insights and Ideas
- Events over Frames: Shifting the core memory unit from raw video frames to semantic "event lifecycles" allows the model to naturally compress long histories without losing critical context.
- Hybrid Latent-Text Memory: Keeping recent history as rich, compact latent vectors while converting older history into retrievable text records provides the best of both worlds: high-fidelity local context and an infinite, computationally cheap long-term memory.
- Constant Workload Scaling: Because historical events are stored as text records and retrieved via a top-$k$ query, the system completely avoids the computational "history tax" that bogs down traditional streaming VLMs over long sequences.
Key Results
- Commentary Quality: StreamSoccer achieved the highest reported CIDEr scores on the current-event (38.62) and historical-memory (17.39) tracks, outperforming proprietary MLLMs (like GPT-4) and general streaming VLMs. It ranked second on the recent-window track (23.96), just behind UniSoccer.
- Computational Efficiency: As shown in Figure 3, StreamSoccer maintains a 95th-percentile Real-Time Factor (RTF) of approximately 0.10–0.22 throughout a 90-minute match. This means it processes video 5x to 10x faster than real-time with zero computational growth over time, whereas baseline models like VideoLLM-online slow down drastically as match history accumulates.
Limitations and Future Work
- Recent-Window Aggregation: The system's second-place finish on the recent-window track indicates a minor weakness in synthesizing multiple rapid-fire or overlapping recent events into a single summary.
- Rule-Based Eventization: The event boundaries rely on deterministic football rules and a fixed 24-second safeguard rather than fully learned, unconstrained semantic discovery.
- Generalization: The model was developed and evaluated entirely within a single soccer data ecosystem; its performance on other sports or varying broadcast styles (e.g., different languages or leagues) remains an open question for future research.
Summary updated