TL;DR
TurnWise is a new reinforcement learning (RL) framework that significantly improves how AI models interact with streaming video. By shifting from "trajectory-level" rewards to "turn-level" credit assignment and balancing loss across turns, it prevents models from becoming redundant and helps them decide exactly when to speak or stay silent.
What Problem Does This Paper Address?
In streaming video understanding, an AI doesn't get the whole video at once. It watches a live feed and must repeatedly decide: Should I answer the user now, or wait for more evidence?
Current state-of-the-art methods use Group Relative Policy Optimization (GRPO), but they suffer from a "granularity mismatch." They typically assign a single reward score to an entire interaction (a trajectory). This creates a "shared credit" problem: if a model gives a great answer at the start but then repeats itself three times, the high initial reward might accidentally reinforce those redundant later turns. Conversely, a model that fails early but provides a brilliant follow-up might be unfairly penalized because the overall trajectory score is low.
As shown in Figure 1, standard GRPO broadcasts a single positive signal across every decision, while TurnWise isolates the value of each specific turn, penalizing irrelevant or repetitive responses even within a "successful" overall trajectory.
Methods
The authors introduce two primary mechanisms to align RL training with the actual decision-making process of a streaming agent:
- Turn-Level Credit Assignment (Silence-Completed Prefix Scoring): To figure out the value of a specific turn without needing a complex "critic" model, TurnWise uses a clever trick. It scores a trajectory "prefix"—the conversation up to turn t—and assumes the model remains silent for the rest of the video. By subtracting this "accrued reward" from the final total reward, the system isolates the "remaining return" generated by the decisions made from that turn onward.
- Turn-Mean Policy Optimization: Standard RL calculates loss based on the number of tokens. This means a long, rambling response has a much larger impact on the model's learning than a short, decisive "Wait" action. TurnWise averages the loss within each turn first, then across turns. This ensures that the decision to stay silent (which might be just one token) is weighted equally with a multi-sentence response.
As illustrated in Figure 2, these components work together to ensure the model learns the value of timing and information density rather than just maximizing token-level probabilities.
Datasets
The models (Qwen2.5-VL-3B and Qwen3-VL-4B) were first fine-tuned on a massive corpus of over 100,000 instruction examples from datasets like EgoExo4D, EgoExoLearn, and LLaVA-Video.
For the RL phase and final evaluation, the authors used three specialized benchmarks:
- ProactiveVQA: Measures answer quality and timeliness.
- OVO-Timing: Evaluates event-triggered responses (precision in timing).
- StreamingBench: Tests real-time visual understanding and proactive output.
Key Insights and Ideas
- The "Wait" Action is First-Class: By using turn-mean aggregation, the model treats the decision to not speak as a critical action. This leads to much more selective and useful behavior.
- Critic-Free Efficiency: Most RL methods require a "critic" (a second model that predicts rewards). TurnWise achieves turn-level granularity by simply re-running the reward function on partial trajectories, requiring no extra learned models or additional rollouts.
- Breaking the Redundancy Loop: The "accrued reward correction" is the "aha moment" here. By subtracting what the model already earned, the RL signal for later turns focuses purely on new information gain. This naturally suppresses the tendency of LLMs to repeat themselves after a successful turn.
Key Results
TurnWise achieved the best aggregate scores across all three benchmarks for both the 3B and 4B model backbones.
Key highlights from Table 1 (labeled as Figure 3 in the provided artifacts) include:
- ProactiveVQA: On the Qwen3-VL-4B backbone, TurnWise reached 42.44 PAUC, a +6.26 point improvement over the strongest non-TurnWise baseline.
- Repetition Reduction: On the 3B model, TurnWise reduced repetition by 61.5% compared to credit-only optimization.
- Timing Accuracy: It significantly improved Timing F1 scores (e.g., from 17.72 to 21.59 on the 3B model), proving that the model isn't just smarter, but also more punctual.
Limitations and Future Work
The authors note that while TurnWise is efficient, it currently relies on a fixed trajectory scorer (in this case, a GPT-based judge). If the reward function itself has biases, the turn-level credit assignment will inherit them.
Future work could explore:
- Scaling to longer videos: The current experiments focused on 20-second segments; longer horizons might introduce new challenges in credit assignment.
- Dynamic Observation Plans: Currently, the model checks the video at fixed intervals. Allowing the model to decide when to look at the next frame (active sensing) would be a natural extension of this turn-based logic.
Summary updated