StreamDuo: A Live Commentator and a Research Agent for Unseen Games

Zixian Gao, Penghao Luo, Xiangcheng Cao, Lixin Xiu, Ziqi Cai, Zhen Li, Erwin Wu, Kaipeng ZhangarXiv 2026 · arXiv

TL;DR

StreamDuo is a novel two-agent framework for live game commentary that generalizes to games never seen during training by pairing a trainable commentator with a "research agent" that looks up game-specific info in real-time wikis. This approach significantly boosts entity recognition and slashes "knowledge leakage"—where models hallucinate names from their training data—from 9.9% down to just 1.0%.

What Problem Does This Paper Address?

While streaming video Large Language Models (LLMs) have become surprisingly good at commentating on games they were trained on, they fall apart when faced with a new title. The authors identify a critical "knowledge gap": while an AI can learn the timing of commentary (e.g., knowing to shout when a fight starts) across different games, it cannot transfer world knowledge.

When a model trained on Baldur’s Gate 3 tries to commentate on The Legend of Zelda: Tears of the Kingdom, it often stays vague or, worse, hallucinates names from its training set—calling a Zelda character "Karlach," for instance. This phenomenon is termed source-game leakage.

Figure 1: StreamDuo correctly identifies a character in an unseen game by asking its research agent, whereas the baseline model hallucinates.

Methods

The authors propose StreamDuo, a system split into two agents with complementary strengths:

  1. The Commentator Agent (Trainable): This agent watches the live stream and decides when to speak. Crucially, it has a second "head" that decides when it needs help. Instead of trying to describe what it doesn't know in text, it simply sends the current video frames to its partner.
  2. The Research Agent (Frozen): A vision-language model (VLM) with access to the game's wiki. It receives frames, identifies entities, searches the wiki, and returns a few relevant entries (name, type, description) about two seconds later.

Training Innovation

The commentator is trained on a single game (Baldur’s Gate 3) in two specialized stages:

  • Stage 1: Learning When to Ask. Using information-gain labels, the model learns to ask for help only when an answer would actually help it reproduce what a human commentator said. If it already knows the character, it stays silent to save "compute."
  • Stage 2: Learning How to Use Answers. Through on-policy self-distillation, a "teacher" version of the model (which sees the perfect wiki entry) guides a "student" version (which sees noisy, real-world research results) on how to correctly integrate that information into natural commentary.
Figure 2: Overview of the StreamDuo architecture and its two-stage training process.

Datasets

  • Training: 10,286 clips from Baldur’s Gate 3.
  • Evaluation: 2,470 clips across ten unseen games, ranging from open-world adventures like Elden Ring and Minecraft to competitive esports like League of Legends and StarCraft II.
  • Benchmark: The study utilizes the Proact-VL game commentary benchmark, modified to ensure game knowledge can only come from the provided wikis (190,722 pages total).

Key Insights and Ideas

  • Frames as Queries: A model that doesn't recognize a character can't name them in a text search. By sending raw frames, the research agent can "see" what the commentator is confused about, bypassing the linguistic bottleneck.
  • Asynchronous Knowledge: The system handles a 2-second delay for research. The commentator keeps talking while waiting, then seamlessly integrates the new knowledge once it arrives.
  • Selective Curiosity: Figure 3 shows that the model's entity naming rate spikes exactly when the research agent's answer arrives, proving it effectively uses the external knowledge it requested.
Figure 3: Entity naming rate spikes significantly after the research agent's answer arrives (at +2s).

Key Results

StreamDuo outperforms existing streaming video LLMs across the board on unseen games:

  • Entity Recall: Improved from 0.082 (Proact-VL) to 0.116.
  • Source-Game Leakage: Reduced from 9.9% to 1.0%, effectively stopping the AI from "forgetting" which game it is watching.
  • Quality Metrics: Achieved the highest scores for liveliness (LiveU) and overall quality (FinalQ) as judged by a strong LLM (Qwen3-30B).
  • Efficiency: The research agent typically responds in ~0.4s, well within the 2s buffer, and the commentator runs 4x faster than real-time on a single H100 GPU.
Table 1: Main quantitative results showing StreamDuo's superiority in timing, knowledge, and overall quality.

Limitations and Future Work

  • Noisy Filtering: While the model is good at using wiki entries, it occasionally repeats irrelevant information if the research agent returns a "near-miss" search result.
  • Wiki Dependency: The system relies on the existence of a comprehensive game wiki. For brand-new or obscure games without documentation, the research agent would be less effective.
  • Complex Interactions: Future work could explore multi-turn "dialogue" between the agents, allowing the commentator to ask follow-up questions for deeper analysis.

Summary updated

Abstract

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.
BibTeX
@misc{gao2026streamduo,
  author = {Zixian Gao and Penghao Luo and Xiangcheng Cao and Lixin Xiu and Ziqi Cai and Zhen Li and Erwin Wu and Kaipeng Zhang},
  title = {{StreamDuo: A Live Commentator and a Research Agent for Unseen Games}},
  year = {2026},
  eprint = {2611.01120},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01120},
}
Figure