Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki

Abstract

Real-time video commentary generation provides textual descriptions of ongoing events in videos, supporting accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone.

Introduction

Generating live commentary for video games or sports is a dual-decision task: the system must determine the content (what to say) and the timing (when to say it). While MLLMs like GPT-4o or Qwen2.5-VL are excellent at describing video frames, they typically struggle with the temporal "rhythm" of commentary. Most existing approaches either assume fixed-length video clips or rely on pre-annotated timestamps to trigger speech. In real-world streaming, this leads to overlapping commentary—where the next trigger occurs before the previous sentence finishes—and excessive verbosity, where models speak even when no significant events occur.

This paper introduces "pause-aware" decoding strategies that allow MLLMs to manage both content and timing through clever prompting and decoding strategies rather than complex, multi-model supervised pipelines.

Methods: Pause-Aware Decoding

The researchers compared two primary prompting-based strategies:

  1. Fixed-Interval Decoding: A baseline where the model is queried every $N$ seconds. This was tested in "Stateless" (no memory) and "Feedback" (includes previous utterances in the prompt) versions.
  2. Dynamic Interval-based Decoding (Realtime): This novel approach introduces two key mechanisms:
  3. The <WAIT> Token: The model is explicitly instructed to output a <WAIT> token if no significant event has occurred, allowing the model to "choose" silence.
  4. Duration-Aware Scheduling: After generating an utterance, the system estimates its speaking duration ($d$) based on word count and a fixed speech rate. The next query is delayed by $d$ seconds, preventing overlap and ensuring a fresh window of video context.
The proposed dynamic interval-based decoding framework, incorporating a feedback loop and duration-aware scheduling.

Crucially, these methods are LLM-agnostic and require zero fine-tuning, relying entirely on in-context learning (ICL) and system prompting. Passing previous commentary back into the prompt acts as a "narrative memory," preventing repetition and helping the model distinguish between new developments and ongoing states.

Experimental Setup and Datasets

The study utilizes a multilingual benchmark across two distinct gaming genres:

  • Car Racing (English & Japanese): Featuring professional-style commentary.
  • Fighting Games (Japanese): Utilizing the SmashCorpus (Super Smash Bros. Ultimate), which features a much faster narrative pace and more frequent "key events."

Results and Discussion

The dynamic "Realtime" approach significantly outperformed fixed-interval methods in human evaluations. On the Japanese racing dataset, the human score for "Pause-awareness" jumped from 2.22 to 3.50 (on a 5-point scale). The dynamic method effectively eliminated overlapping commentary, leading to a more "breathable" viewing experience.

The Verbosity Gap

A striking finding was that MLLMs are naturally much more "chatty" than humans. Even with timing improvements, models generated significantly more words than human commentators.

Comparison of generated commentary length (avg, min, max) across models and datasets, highlighting the verbosity gap compared to human references.

As shown in the table above, models like Qwen 7b and GPT-4.1 consistently produced longer average utterances than the human reference across all datasets. This suggests that "conciseness" remains a major hurdle for AI broadcasters.

Conclusion

This research demonstrates that pause-aware decoding can significantly improve the naturalness of AI-generated commentary without expensive fine-tuning. While the dynamic approach successfully manages timing and reduces overlap, future work must address the "verbosity gap" to match the punchy, brief style of professional human commentators. Additionally, domain-specific technical nuances in fast-paced games may still benefit from targeted fine-tuning.

Summary updated

Abstract

Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
BibTeX
@inproceedings{afzal2026real,
  author = {Anum Afzal and Yuki Saito and Hiroya Takamura and Katsuhito Sudoh and Shinnosuke Takamichi and Graham Neubig and Florian Matthes and Tatsuya Ishigaki},
  title = {{Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches}},
  booktitle = {LREC},
  year = {2026},
  doi = {10.63317/5m3djogm95q9},
}
Figure