TL;DR
Current video generation models require dense, highly descriptive prompts to produce high-quality outputs, but human users typically provide short, ambiguous, or multimodal instructions. ReaDe introduces a model-agnostic, "reason-then-describe" instruction interpreter that uses Chain-of-Thought reasoning to resolve cross-modal ambiguities before synthesizing structured prompts for downstream video diffusion models. Trained via a two-stage pipeline combining supervised reasoning initialization with multi-dimensional RL (GRPO), ReaDe significantly enhances controllability and video fidelity across diverse single- and multi-condition generation tasks using only ~16K training samples.
What Problem Does This Paper Address?
Modern Diffusion Transformer (DiT) models excel at synthesizing high-fidelity video, but their performance heavily relies on detailed, structured training captions specifying subject appearance, camera trajectory, lighting, and temporal actions. At inference time, user inputs are typically brief, vague, or accompanied by disparate non-textual conditions (e.g., identity reference images, depth maps, camera poses, or audio).
This mismatch causes downstream generators to misinterpret user intent or produce visually incoherent videos. While prior prompt-enhancement frameworks exist, most are text-only or rely on brute-force Supervised Fine-Tuning (SFT) over massive datasets (e.g., Any2Caption). These SFT-based methods frequently fail to resolve conflicting multimodal constraints, generalize poorly to unseen controls, and lack the reasoning ability needed to interpret complex, reasoning-intensive user requests.
Methods
ReaDe addresses this by acting as an upstream, generator-agnostic interpreter that translates arbitrary multimodal instructions into dense, structured prompts through a two-stage "reason-then-describe" framework:
[ Multimodal User Input ]
│
▼
┌──────────────────────────────────────────────┐
│ Stage 1: CoT-Guided Reasoning SFT │
│ - Step 1: Interpret Textual Intent │
│ - Step 2: Parse Non-Textual Conditions │
│ - Step 3: Cross-Modal Alignment │
│ - Step 4: Contextual Detail Completion │
└──────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ Stage 2: Incentive Reinforcement Learning │
│ - Group Relative Policy Optimization (GRPO) │
│ - Multi-Dimensional Reward Assigner (LLM) │
│ (User Intent + Visual Details + Harmony) │
└──────────────────────────────────────────────┘
│
▼
[ Dense, Structured Prompt (<think> ... <answer>) ]
│
▼
[ Downstream Video Generator (DiT / CogVideoX / Kling) ]
- Architecture: Initialized from the multimodal model
Qwen2.5-Omni(handling text, image, video, and audio) augmented with a lightweight 3D camera encoder. - Stage 1: CoT-Guided Reasoning Initialization: The model is fine-tuned on instruction–thinking–answer triplets following a structured 4-step Chain-of-Thought process:
- Textual intent extraction (identifying whether the goal is creation, modification, or deletion).
- Non-textual condition parsing (converting poses, depth, or camera inputs into semantic features).
- Multimodal alignment (resolving conflicting signals by prioritizing explicit user instructions).
- Supplementary completion (grounding missing scene context like lighting or style).
- Stage 2: Incentive Reinforcement Learning via GRPO: Generating free-form descriptions lacks deterministic, rule-based verification. ReaDe applies Group Relative Policy Optimization (GRPO) using a multi-dimensional reward assigner evaluated by a critic LLM (Qwen3-30B): $$R = \alpha R_{\text{user}} + \rho R_{\text{detail}} + \gamma R_{\text{supp}} - \lambda R_{\text{contra}}$$ This explicitly rewards user-intent coverage ($R_{\text{user}}$), condition fidelity ($R_{\text{detail}}$), and plausible scene completion ($R_{\text{supp}}$), while penalizing contradictions ($R_{\text{contra}}$) and improper structured formatting ($R_{\text{format}}$).
Datasets
- Training Data: Curated high-precision datasets totaling ~16.7K examples (8,452 for Stage 1 SFT; 8,309 for Stage 2 RL), covering identity images (1–5 IDs), depth sequences, camera trajectories, and human poses. This is substantially smaller than prior frameworks like Any2Caption (~337K instances).
- Evaluation Benchmarks:
- FullDiT Benchmark: Tested across single-condition and compositional multi-condition controls (Camera + Depth + Multi-IDs).
- VBench: Comprehensive text-to-video quality benchmark evaluated with CogVideoX-2B.
- Dream-1K: Public benchmark evaluating temporal and event-centric video descriptions.
Key Insights and Ideas
- Explicit Reasoning Prevents Cross-Modal Hallucinations: Simply teaching an LLM to output dense captions (direct SFT) causes it to memorize patterns or hallucinate details that contradict non-textual signals. Explicitly separating "deliberation" (
<think>) from "description" (<answer>) allows the model to reconcile cross-modal conflicts before writing the prompt. - Data Quality and Deliberation Trump Scale: ReaDe outperforms models trained on over 20× more data (337K vs. 16K) by focusing on structured reasoning traces rather than raw caption memorization.
- Cross-Condition Transferability: Training on geometric conditions like depth yields the highest zero-shot transferability to other conditions (e.g., camera and identity), as depth maps inherently encode dense spatial layouts and structural priors.
Key Results
- Superior Controllability: Across single-condition benchmarks (FullDiT), ReaDe outperforms Any2Caption and specialized baselines:
- Multi-Identity Control: CLIP-T increases from 17.15 to 18.64, and identity similarity (DINO-I) jumps from 39.42 to 45.28.
- Depth Control: Mean Absolute Error (MAE) drops from 21.87 to 18.79 (lower is better).
- Camera Control: Rotation and translation errors drop to 1.30 and 3.46 (from 1.45 and 4.37).
- Multi-Condition Composition: Under heavy multi-modal constraints (Camera + Depth + Multi-IDs), ReaDe improves prompt fidelity (CLIP-T: 21.24 vs. 19.52) and visual consistency (DINO-I: 39.46 vs. 38.74; MAE: 17.03 vs. 17.41).
- Downstream Generation Quality (VBench on CogVideoX-2B): When ReaDe incorporates generator-in-the-loop feedback, it achieves top scores across dynamic degree (65.92 vs. 60.00 baseline) and background consistency (98.44 vs. 95.90 baseline).
Limitations and Future Work
- Inference Latency: Generating intermediate reasoning traces (
<think>) alongside dense structured captions increases token length and inference latency prior to video generation. - Reliance on LLM Evaluators during RL: Stage 2 depends on an external LLM judge (Qwen3-30B) to score semantic coverage, which introduces computational overhead during training.
- Future Directions: The authors plan to expand ReaDe to broader modalities (e.g., tactile or fine-grained audio conditioning), support longer-horizon multi-shot reasoning, and explore safety-aligned, human-in-the-loop optimization for real-world creative deployment.
Summary updated