Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation

Ryota Kawamatsu, Anum Afzal, Yuki Saito, Shinnosuke Takamichi, Graham Neubig, Katsuhito Sudoh, Hiroya Takamura, Tatsuya IshigakiarXiv 2026 · arXiv

Here is a concise, engaging summary of the paper, tailored for a technically literate audience.

TL;DR

This paper introduces an end-to-end, low-latency real-time audio game commentary system that generates spoken commentary directly from live gameplay video. By decoupling text generation from speech playback and running them in parallel, the system slashes the average silence between commentary lines from 9.6 seconds to just 0.3 seconds, dramatically improving the natural rhythm of the broadcast.

What Problem Does This Paper Address?

Generating real-time automated game commentary from raw video requires a tight pipeline: capturing video frames, generating descriptive text using a multimodal Large Language Model (LLM), and converting that text to speech via a Text-to-Speech (TTS) module.

Traditional architectures process these steps sequentially. Crucially, they wait for the current speech audio to finish playing before requesting the next text generation from the LLM. In fast-paced gaming environments, this strict sequentiality creates a massive latency bottleneck. The accumulated waiting times lead to long, unnatural silences (often nearly 10 seconds) between commentary lines, breaking viewer immersion and failing to keep up with the on-screen action.

Methods

To eliminate this idle waiting time, the authors propose a low-latency system design built on two core ideas:

  1. Parallel Text Generation & Buffering: Instead of waiting for speech playback to finish, the system continuously captures new video segments and generates candidate commentary text in the background. These candidates are stored in a buffer. As soon as the current speech playback ends, the system immediately pulls a candidate from the buffer and synthesizes it, ensuring near-instantaneous continuity. The authors test three lightweight buffer selection policies to avoid adding decision latency: Latest (most recent segment), Oldest (earliest buffered), and Random.
  2. Video Delay Control: Because LLM inference and TTS synthesis still introduce an unavoidable baseline delay, the system intentionally buffers and delays the outgoing live video stream. By matching the video delay to the initial end-to-end generation latency, the system ensures that the generated audio aligns perfectly with the visuals seen by the viewer.
Figure 1: System overview: sequential baseline vs. our parallel gen- eration with buffering and video delay control.

The system uses GPT-4.1-mini for text generation and an expressive TTS model optimized for fighting games.

Datasets

The system was evaluated using the Smash Corpus (Saito et al., 2020), a spontaneous speech corpus containing gameplay videos of Super Smash Bros. Ultimate. This game was specifically chosen because it is exceptionally fast-paced, making any commentary latency highly noticeable. The corpus also includes professional human commentaries, providing a gold-standard baseline for timing and content quality.

Key Insights and Ideas

  • Pipeline Architecture Trumps Raw Model Speed: The primary bottleneck in real-time commentary isn't just the inference speed of the LLM, but the sequential design of the pipeline. Parallelizing the workflow yields massive latency rewards without needing a smaller, less capable language model.
  • Policy Simplicity is Sufficient: Surprisingly, the choice of buffer selection policy (Latest vs. Oldest vs. Random) had a relatively minor impact on silence metrics. The heavy lifting is done entirely by the parallel architecture and buffering mechanism itself.
  • Tuning Output Length Alters Behavior: Changing the max_new_tokens parameter creates distinct trade-offs. A limit of 40 tokens best matches human speaking duration, while 60 tokens yields text lengths closest to actual human commentary.

Key Results

  • Dramatic Silence Reduction: The parallel system reduced the mean inter-utterance silence from 9.6 seconds (sequential baseline) down to 0.3 seconds, closely approaching the human baseline of 1.7 seconds.
  • Human-like Timing: The mean Intersection over Union (mIoU) metric—which measures how closely the system's speaking/silence patterns match professional human commentators—leaped from a dismal ~0.01 in sequential baselines to 0.60 with the parallel approach.
  • User Study Validation: A human evaluation involving 120 experienced game players showed that the parallel system significantly outperformed sequential baselines across all metrics, including commentary rhythm naturalness, video-audio alignment, and overall quality.
User study results showing Mean Opinion Scores (MOS) across different methods and token lengths.

Limitations and Future Work

While the lightweight buffer selection policies (Latest, Oldest, Random) prevent decision latency, they are fundamentally simplistic. They do not evaluate the semantic importance or excitement level of the buffered text. The authors note that future work should explore smarter, context-aware selection policies that can prioritize high-impact game events without introducing significant computational overhead. Additionally, evaluating the system across other fast-paced genres (like MOBAs or traditional sports) remains an open direction.

Summary updated

Abstract

We present a low-latency real-time audio game commentary system that generates spoken commentary directly from live gameplay video. In this end-to-end setting, a key bottleneck is accumulated waiting time; conventional pipelines capture frames, generate text, and synthesize speech sequentially for each utterance, and do not request the next generation until speech playback has completed. This strict sequentiality causes long and unnatural silence between utterances. To address this latency bottleneck, our system runs text generation in parallel with speech playback and buffers multiple candidate utterances ahead of time, enabling immediate synthesis at playback boundaries. Experiments on fast-paced game videos show that our parallel design reduces the mean inter-utterance silence from 9.6 seconds to 0.3 seconds compared to sequential baselines. It also improves similarity to professional speaking--silence timing patterns by over 40 %, and a user study with 120 experienced game players confirms significantly improved perceived speaking rhythm. Our demo video is available at: https://youtu.be/pmrRUlvav8M.
BibTeX
@misc{kawamatsu2026low,
  author = {Ryota Kawamatsu and Anum Afzal and Yuki Saito and Shinnosuke Takamichi and Graham Neubig and Katsuhito Sudoh and Hiroya Takamura and Tatsuya Ishigaki},
  title = {{Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation}},
  year = {2026},
  eprint = {2606.13322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.13322},
}

Loading related content...

Figure