Abstract
Real-time video commentary generation provides textual descriptions of ongoing events in videos, supporting accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone.
Introduction
Generating live commentary for video games or sports is a dual-decision task: the system must determine the content (what to say) and the timing (when to say it). While MLLMs like GPT-4o or Qwen2.5-VL are excellent at describing video frames, they typically struggle with the temporal "rhythm" of commentary. Most existing approaches either assume fixed-length video clips or rely on pre-annotated timestamps to trigger speech. In real-world streaming, this leads to overlapping commentary—where the next trigger occurs before the previous sentence finishes—and excessive verbosity, where models speak even when no significant events occur.
This paper introduces "pause-aware" decoding strategies that allow MLLMs to manage both content and timing through clever prompting and decoding strategies rather than complex, multi-model supervised pipelines.
Methods: Pause-Aware Decoding
The researchers compared two primary prompting-based strategies:
- Fixed-Interval Decoding: A baseline where the model is queried every $N$ seconds. This was tested in "Stateless" (no memory) and "Feedback" (includes previous utterances in the prompt) versions.
- Dynamic Interval-based Decoding (Realtime): This novel approach introduces two key mechanisms:
- The
<WAIT>Token: The model is explicitly instructed to output a<WAIT>token if no significant event has occurred, allowing the model to "choose" silence. - Duration-Aware Scheduling: After generating an utterance, the system estimates its speaking duration ($d$) based on word count and a fixed speech rate. The next query is delayed by $d$ seconds, preventing overlap and ensuring a fresh window of video context.
Crucially, these methods are LLM-agnostic and require zero fine-tuning, relying entirely on in-context learning (ICL) and system prompting. Passing previous commentary back into the prompt acts as a "narrative memory," preventing repetition and helping the model distinguish between new developments and ongoing states.
Experimental Setup and Datasets
The study utilizes a multilingual benchmark across two distinct gaming genres:
- Car Racing (English & Japanese): Featuring professional-style commentary.
- Fighting Games (Japanese): Utilizing the SmashCorpus (Super Smash Bros. Ultimate), which features a much faster narrative pace and more frequent "key events."
Results and Discussion
The dynamic "Realtime" approach significantly outperformed fixed-interval methods in human evaluations. On the Japanese racing dataset, the human score for "Pause-awareness" jumped from 2.22 to 3.50 (on a 5-point scale). The dynamic method effectively eliminated overlapping commentary, leading to a more "breathable" viewing experience.
The Verbosity Gap
A striking finding was that MLLMs are naturally much more "chatty" than humans. Even with timing improvements, models generated significantly more words than human commentators.
As shown in the table above, models like Qwen 7b and GPT-4.1 consistently produced longer average utterances than the human reference across all datasets. This suggests that "conciseness" remains a major hurdle for AI broadcasters.
Conclusion
This research demonstrates that pause-aware decoding can significantly improve the naturalness of AI-generated commentary without expensive fine-tuning. While the dynamic approach successfully manages timing and reduces overlap, future work must address the "verbosity gap" to match the punchy, brief style of professional human commentators. Additionally, domain-specific technical nuances in fast-paced games may still benefit from targeted fine-tuning.
Summary updated