Turns Matter: Advancing Streaming Video RL via Turn-Level Credit

Shuai Dong, Kanghui Tian, Linghan Chen, Wenqi Shao, Hongrui Wu, Haowen Hou, Yongfu Zhu, Xuehui Wang, Ziyue Wang, Wenpu Liu, Weichu Xie, Yuqi Xu, Congcong Wang, Kaiwen Tuo, Caoyuan Ma, Taiqiang Wu, Ji Zhao, Wenzheng Chang, Dianyi Wang, Siyuan Wang, Zhongyu Wei, Yu Qiao, Jiaqi WangarXiv 2026 · arXiv

TL;DR

TurnWise is a new reinforcement learning (RL) framework that significantly improves how AI models interact with streaming video. By shifting from "trajectory-level" rewards to "turn-level" credit assignment and balancing loss across turns, it prevents models from becoming redundant and helps them decide exactly when to speak or stay silent.

What Problem Does This Paper Address?

In streaming video understanding, an AI doesn't get the whole video at once. It watches a live feed and must repeatedly decide: Should I answer the user now, or wait for more evidence?

Current state-of-the-art methods use Group Relative Policy Optimization (GRPO), but they suffer from a "granularity mismatch." They typically assign a single reward score to an entire interaction (a trajectory). This creates a "shared credit" problem: if a model gives a great answer at the start but then repeats itself three times, the high initial reward might accidentally reinforce those redundant later turns. Conversely, a model that fails early but provides a brilliant follow-up might be unfairly penalized because the overall trajectory score is low.

Shared advantages in standard GRPO can reinforce redundant replies, whereas TurnWise provides turn-specific signals.

As shown in Figure 1, standard GRPO broadcasts a single positive signal across every decision, while TurnWise isolates the value of each specific turn, penalizing irrelevant or repetitive responses even within a "successful" overall trajectory.

Methods

The authors introduce two primary mechanisms to align RL training with the actual decision-making process of a streaming agent:

  1. Turn-Level Credit Assignment (Silence-Completed Prefix Scoring): To figure out the value of a specific turn without needing a complex "critic" model, TurnWise uses a clever trick. It scores a trajectory "prefix"—the conversation up to turn t—and assumes the model remains silent for the rest of the video. By subtracting this "accrued reward" from the final total reward, the system isolates the "remaining return" generated by the decisions made from that turn onward.
  2. Turn-Mean Policy Optimization: Standard RL calculates loss based on the number of tokens. This means a long, rambling response has a much larger impact on the model's learning than a short, decisive "Wait" action. TurnWise averages the loss within each turn first, then across turns. This ensures that the decision to stay silent (which might be just one token) is weighted equally with a multi-sentence response.
The TurnWise workflow: from streaming input to turn-specific credit assignment and balanced policy optimization.

As illustrated in Figure 2, these components work together to ensure the model learns the value of timing and information density rather than just maximizing token-level probabilities.

Datasets

The models (Qwen2.5-VL-3B and Qwen3-VL-4B) were first fine-tuned on a massive corpus of over 100,000 instruction examples from datasets like EgoExo4D, EgoExoLearn, and LLaVA-Video.

For the RL phase and final evaluation, the authors used three specialized benchmarks:

  • ProactiveVQA: Measures answer quality and timeliness.
  • OVO-Timing: Evaluates event-triggered responses (precision in timing).
  • StreamingBench: Tests real-time visual understanding and proactive output.

Key Insights and Ideas

  • The "Wait" Action is First-Class: By using turn-mean aggregation, the model treats the decision to not speak as a critical action. This leads to much more selective and useful behavior.
  • Critic-Free Efficiency: Most RL methods require a "critic" (a second model that predicts rewards). TurnWise achieves turn-level granularity by simply re-running the reward function on partial trajectories, requiring no extra learned models or additional rollouts.
  • Breaking the Redundancy Loop: The "accrued reward correction" is the "aha moment" here. By subtracting what the model already earned, the RL signal for later turns focuses purely on new information gain. This naturally suppresses the tendency of LLMs to repeat themselves after a successful turn.

Key Results

TurnWise achieved the best aggregate scores across all three benchmarks for both the 3B and 4B model backbones.

Quantitative results showing TurnWise outperforming SFT and other RL baselines across multiple benchmarks.

Key highlights from Table 1 (labeled as Figure 3 in the provided artifacts) include:

  • ProactiveVQA: On the Qwen3-VL-4B backbone, TurnWise reached 42.44 PAUC, a +6.26 point improvement over the strongest non-TurnWise baseline.
  • Repetition Reduction: On the 3B model, TurnWise reduced repetition by 61.5% compared to credit-only optimization.
  • Timing Accuracy: It significantly improved Timing F1 scores (e.g., from 17.72 to 21.59 on the 3B model), proving that the model isn't just smarter, but also more punctual.

Limitations and Future Work

The authors note that while TurnWise is efficient, it currently relies on a fixed trajectory scorer (in this case, a GPT-based judge). If the reward function itself has biases, the turn-level credit assignment will inherit them.

Future work could explore:

  • Scaling to longer videos: The current experiments focused on 20-second segments; longer horizons might introduce new challenges in credit assignment.
  • Dynamic Observation Plans: Currently, the model checks the video at fixed intervals. Allowing the model to decide when to look at the next frame (active sensing) would be a natural extension of this turn-based logic.

Summary updated

Abstract

Streaming video understanding requires models to repeatedly decide whether to respond or continue observing. Recently, Group Relative Policy Optimization (GRPO) has been applied to this setting, with existing approaches emphasizing reward design while retaining a single trajectory-level advantage across turns. Such shared credit can reinforce redundant replies after earlier successes and discourage useful follow-ups after earlier failures. We propose TurnWise, a turn-level GRPO framework that aligns learning with individual streaming decisions. TurnWise addresses the mismatch between trajectory-level feedback and turn-level interaction, promoting informative responses and reducing redundancy. Specifically, turn-level credit assignment uses silence-completed prefix scoring to construct group-relative advantages from each decision's remaining return, separating its learning signal from earlier rewards. Turn-mean policy optimization averages token losses within each turn and then across turns, giving response and waiting decisions equal weight regardless of output length. These complementary designs improve streaming policy learning without a learned critic or additional rollouts. Extensive experiments show our method outperforms prior RL approaches for streaming video understanding.
BibTeX
@misc{dong2026turns,
  author = {Shuai Dong and Kanghui Tian and Linghan Chen and Wenqi Shao and Hongrui Wu and Haowen Hou and Yongfu Zhu and Xuehui Wang and Ziyue Wang and Wenpu Liu and Weichu Xie and Yuqi Xu and Congcong Wang and Kaiwen Tuo and Caoyuan Ma and Taiqiang Wu and Ji Zhao and Wenzheng Chang and Dianyi Wang and Siyuan Wang and Zhongyu Wei and Yu Qiao and Jiaqi Wang},
  title = {{Turns Matter: Advancing Streaming Video RL via Turn-Level Credit}},
  year = {2026},
  eprint = {2611.06728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06728},
}
Figure