A Reason-then-Describe Instruction Interpreter for Controllable Video Generation

Shengqiong Wu, Weicai Ye, Yuanxing Zhang, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Kun Gai, Hao Fei, Tat-Seng Chua

Abstract

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent.

Introduction

Modern video generation models like Diffusion Transformers (DiTs) achieve high fidelity but suffer from an "intent-output mismatch." While these models require dense, detailed captions for optimal performance, real-world user prompts are often sparse or ambiguous. Existing prompt optimizers typically lack the reasoning capabilities to handle complex, multi-modal instructions—such as combining a specific person's identity with a depth map and complex camera motion. This often leads to generated videos that ignore user intent or exhibit physical incoherence.

Methodology: The ReaDe Framework

ReaDe addresses these challenges by emulating human-like reasoning to parse instructions. It functions as a model-agnostic "instruction interpreter" that translates short user prompts into the detailed captions required by high-end generators. The framework is optimized through a two-stage process:

  1. Stage 1: CoT-Guided Reasoning Initialization (SFT): The model is trained on "instruction-thinking-answer" triples. It learns to perform four internal reasoning steps: interpreting textual intent, understanding non-textual inputs (like poses or camera trajectories), aligning multiple modalities to resolve conflicts, and completing supplementary details (e.g., lighting or style).
  2. Stage 2: Incentive Reinforcement Learning (GRPO): To move beyond simple pattern matching, ReaDe employs Group Relative Policy Optimization (GRPO). A multi-dimensional reward assigner evaluates model outputs based on user intent coverage, detail accuracy, imagination quality, and internal consistency.
The two-stage optimization pipeline of ReaDe, illustrating the CoT-guided reasoning and reinforcement learning stages.

Key Insights and Results

A primary finding of this work is that explicit reasoning is the "secret sauce" for controllability. The <think> phase allows the model to reconcile conflicting constraints before committing to a final description. For example, in reasoning-intensive tasks like "an animal growing up," ReaDe correctly identifies the need for a biological transition (e.g., tadpole to frog) and generates a structured caption to guide the video generator.

ReaDe consistently outperforms baselines like Any2Caption and Prompt-A-Video. On the VBench benchmark using CogVideoX-2B, ReaDe improved the "Multiple Objects" score from 40.17 to 71.89.

Furthermore, the model demonstrates remarkable cross-domain generalization. As shown in the intention accuracy heatmap, training on one type of condition (e.g., depth maps) significantly improves performance on others (e.g., camera moves). This suggests that ReaDe learns a generalized spatial and semantic "world logic" rather than just memorizing specific input-output pairs.

Heatmap showing the intention accuracy of ReaDe across different training and evaluation conditions, highlighting its cross-domain generalization capabilities.

Conclusion

ReaDe provides a robust, plug-and-play solution for bridging the gap between human intent and controllable video generation. By incorporating a reasoning-based interpreter, it enables high-fidelity generation that faithfully respects complex, multi-modal user instructions. Future work will explore expanding the range of supported modalities and extending these reasoning capabilities to long-horizon narrative video generation.

Summary updated

Abstract

Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.
BibTeX
@misc{wu2025reason,
  author = {Shengqiong Wu and Weicai Ye and Yuanxing Zhang and Jiahao Wang and Quande Liu and Xintao Wang and Pengfei Wan and Kun Gai and Hao Fei and Tat-Seng Chua},
  title = {{A Reason-then-Describe Instruction Interpreter for Controllable Video Generation}},
  year = {2025},
  eprint = {2511.20563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.20563},
}
Figure