TL;DR
DirCast is the first single streaming video-language model capable of performing "live broadcast production"—simultaneously acting as both a director and a commentator for esports. By processing ten raw player feeds in real-time, it decides which camera to put on air and provides live narration, outperforming zero-shot models like GPT-5 in directing and rivaling top-tier streaming commentators.
What Problem Does This Paper Address?
In a professional esports broadcast, two roles work in tight coordination: the director, who watches a wall of player cameras to choose the most exciting shot, and the caster, who narrates the action on that chosen screen.
Current AI models for esports are mostly "backseat casters"—they can commentate, but only if a human has already done the hard work of directing the feed. If you give these models ten raw player cameras, they fail because they can't handle the massive influx of visual data or the complex decision-making required to "find the fun." This paper introduces the task of live broadcast production, where a model must receive all raw feeds and produce a finished program (video + audio) using only past observations.
Methods
The authors present DirCast, a single causal video-language model (fine-tuned from Qwen3-VL-8B) that handles both tasks in a unified stream. Key technical innovations include:
- Coarse-to-Fine Processing: To keep computation manageable, DirCast scans all ten cameras at low resolution (128px) to make directing decisions. Once a camera is chosen, it reads only that feed at high resolution (288px) to generate commentary.
- Hold-or-Switch Director Head: Instead of a simple classifier, DirCast uses a "gate" to decide whether to stay on the current shot or cut away. This mimics human behavior, where shots are typically held for several seconds to avoid "flickering" cuts.
- Program-as-Memory: To prevent the model's context from exploding, it only remembers the "program"—the high-res frames that actually went on air and the words spoken. The nine unselected low-res "scouting" views expire after 8 seconds.
- On-Policy Rollouts: During training, the model is exposed to its own potential mistakes (drifting away from human choices) to learn how to recover and continue commentating effectively even from "bad" shots.
Datasets
The researchers built a new dataset from 27 professional Dota 2 matches (including EPL Masters and BLAST Slam).
- Multi-Camera Data: They rendered ten synchronized cameras (one per hero) from match replays.
- Ground Truth: They aligned these renders with official broadcasts to label which hero the human director chose and what the professional casters said at every second.
- Scale: The test set alone includes 5 hours of gameplay across 496 clips.
Key Insights and Ideas
- Directing is the Foundation: Commentary is only as good as the director's choice. If the director misses a kill, the commentator has nothing to talk about. By coupling these tasks, the model learns that "action is worth showing."
- Context Management is Key: You don't need to remember what every camera showed a minute ago. By letting non-aired views expire, DirCast maintains a constant computational cost regardless of match length, allowing it to run in real-time on a single GPU.
- Human Rhythm via Dynamic Thresholds: To avoid the "silence trap" (where a model becomes increasingly likely to stay quiet), DirCast uses dynamic thresholds that adjust based on its recent activity, ensuring the broadcast remains lively.
Key Results
DirCast sets a new state-of-the-art for automated esports production:
- Directing: It achieved a 27.0% Top-1 agreement with human directors, significantly outperforming rule-based systems and zero-shot VLMs like GPT-5 (13.5%). Crucially, it covered 89.1% of all kills in the matches.
- Commentary: In head-to-head comparisons judged by GPT-5, DirCast's commentary won 29.2% of comparisons against GPT-4o. This is particularly impressive because DirCast had to direct its own shots, while the baselines were given the "perfect" human-directed feed.
- Efficiency: The model runs at approximately 0.3 seconds of compute per video second, making it fully capable of live deployment.
Limitations and Future Work
- Camera Variety: Currently, the model is limited to the ten hero-locked cameras. Professional broadcasts often use "free cameras" or cinematic sweeps, which DirCast cannot yet produce.
- Game Scope: The dataset and evaluation are currently specific to Dota 2. Extending this to other genres (like FPS or Battle Royales) would test the model's generalizability.
- Dataset Scaling: The authors note that the model shows no signs of saturation; training on hundreds of matches (which can be collected automatically from replays) would likely lead to even better performance.
Summary updated