TL;DR
StreamDuo is a novel two-agent framework for live game commentary that generalizes to games never seen during training by pairing a trainable commentator with a "research agent" that looks up game-specific info in real-time wikis. This approach significantly boosts entity recognition and slashes "knowledge leakage"—where models hallucinate names from their training data—from 9.9% down to just 1.0%.
What Problem Does This Paper Address?
While streaming video Large Language Models (LLMs) have become surprisingly good at commentating on games they were trained on, they fall apart when faced with a new title. The authors identify a critical "knowledge gap": while an AI can learn the timing of commentary (e.g., knowing to shout when a fight starts) across different games, it cannot transfer world knowledge.
When a model trained on Baldur’s Gate 3 tries to commentate on The Legend of Zelda: Tears of the Kingdom, it often stays vague or, worse, hallucinates names from its training set—calling a Zelda character "Karlach," for instance. This phenomenon is termed source-game leakage.
Methods
The authors propose StreamDuo, a system split into two agents with complementary strengths:
- The Commentator Agent (Trainable): This agent watches the live stream and decides when to speak. Crucially, it has a second "head" that decides when it needs help. Instead of trying to describe what it doesn't know in text, it simply sends the current video frames to its partner.
- The Research Agent (Frozen): A vision-language model (VLM) with access to the game's wiki. It receives frames, identifies entities, searches the wiki, and returns a few relevant entries (name, type, description) about two seconds later.
Training Innovation
The commentator is trained on a single game (Baldur’s Gate 3) in two specialized stages:
- Stage 1: Learning When to Ask. Using information-gain labels, the model learns to ask for help only when an answer would actually help it reproduce what a human commentator said. If it already knows the character, it stays silent to save "compute."
- Stage 2: Learning How to Use Answers. Through on-policy self-distillation, a "teacher" version of the model (which sees the perfect wiki entry) guides a "student" version (which sees noisy, real-world research results) on how to correctly integrate that information into natural commentary.
Datasets
- Training: 10,286 clips from Baldur’s Gate 3.
- Evaluation: 2,470 clips across ten unseen games, ranging from open-world adventures like Elden Ring and Minecraft to competitive esports like League of Legends and StarCraft II.
- Benchmark: The study utilizes the Proact-VL game commentary benchmark, modified to ensure game knowledge can only come from the provided wikis (190,722 pages total).
Key Insights and Ideas
- Frames as Queries: A model that doesn't recognize a character can't name them in a text search. By sending raw frames, the research agent can "see" what the commentator is confused about, bypassing the linguistic bottleneck.
- Asynchronous Knowledge: The system handles a 2-second delay for research. The commentator keeps talking while waiting, then seamlessly integrates the new knowledge once it arrives.
- Selective Curiosity: Figure 3 shows that the model's entity naming rate spikes exactly when the research agent's answer arrives, proving it effectively uses the external knowledge it requested.
Key Results
StreamDuo outperforms existing streaming video LLMs across the board on unseen games:
- Entity Recall: Improved from 0.082 (Proact-VL) to 0.116.
- Source-Game Leakage: Reduced from 9.9% to 1.0%, effectively stopping the AI from "forgetting" which game it is watching.
- Quality Metrics: Achieved the highest scores for liveliness (LiveU) and overall quality (FinalQ) as judged by a strong LLM (Qwen3-30B).
- Efficiency: The research agent typically responds in ~0.4s, well within the 2s buffer, and the commentator runs 4x faster than real-time on a single H100 GPU.
Limitations and Future Work
- Noisy Filtering: While the model is good at using wiki entries, it occasionally repeats irrelevant information if the research agent returns a "near-miss" search result.
- Wiki Dependency: The system relies on the existence of a comprehensive game wiki. For brand-new or obscure games without documentation, the research agent would be less effective.
- Complex Interactions: Future work could explore multi-turn "dialogue" between the agents, allowing the commentator to ask follow-up questions for deeper analysis.
Summary updated