Building high-quality video generation models is no longer just about rendering pretty pixels; it’s about semantic control. As the field moves from short clips to coherent audiovisual sequences, the bottleneck has shifted toward how well a model "understands" the world it is trying to create.
In this technical report, the Kling Team pulls back the curtain on the "understanding stack" behind their Kling models. They argue that video quality is a direct projection of multimodal understanding across the entire pipeline: from how data is captioned to how prompts are enhanced and how rewards are modeled.
TL;DR
The Kling Team presents a comprehensive framework for video generation where quality is driven by a unified multimodal foundation model (Kling-MLLM). Their core thesis is that generation is a "projection of understanding" across a four-stage chain—Data, Conditioning, Alignment, and Evaluation—prioritizing dense semantic descriptions and perceptual training over raw language ability.
What Problem Does This Paper Address?
Current video generation systems often struggle with "semantic drift"—hallucinating actions, losing subject identity across shots, or failing to synchronize audio and video. The authors identify that standard multimodal benchmarks (like VQA) don't capture the specific requirements of video generation, such as camera language, spatial relations, and dense temporal changes.
Furthermore, Diffusion Transformers (DiTs) are trained on far fewer samples than Large Language Models (LLMs). This creates a "data efficiency" gap: every training video must carry as much supervision as possible to teach the model complex controls.
Methods
The authors propose a unified architecture and a modular pipeline to solve these issues:
- Kling-MLLM: A shared foundation model (ranging from 27B to a 397B Mixture-of-Experts) that unifies images, videos, and audio into a single autoregressive interface. It uses a specialized audio tower and a visual path (ViT + Vision Aligner) to map all inputs into a common sequence.
- The Semantic Contract: Instead of vague, fluent captions, they use "Generation-Oriented Captioning." This treats captions as a dense data structure specifying shot boundaries, timestamps, OCR text, and synchronized audio events.
- Reference-Grounded Captioning (RGC): A novel approach for controllable generation where captions explicitly bind reference assets (like a character portrait or a voice track) to specific entities and time intervals in the target video.
- Prompt Enhancement (PE): A two-tier system. Agentic PE refines the user's story and retrieves compelling examples, while Basic PE (a trained rewriter) translates that intent into the specific "caption dialect" the generator understands.
- Dual Reward Modeling: They split reward modeling into a General Reward Model (GRM) for explainable, localized critiques (using bounding boxes and timestamps) and a Decision Reward Model (DRM) for fast, multidimensional scoring during reinforcement learning.
Datasets
The team utilized a massive mixture of data, summarized in their high-level supervision pipeline:
- Public Data: Ingested via an automated "AutoResearch" agent that monitors Hugging Face for new datasets.
- Kling-Specific Task Data: Internal data targeting specific perception gaps like camera behavior, multi-shot structure, and bilingual OCR.
- Multimodal Agent Data: Data from tool-use loops and 3D Blender-based tasks to teach spatial reasoning.
Key Insights and Ideas
- Description Density is King: Because DiTs can't be scaled with the same data volume as LLMs, the quality of text per second of video is the primary lever for data efficiency.
- Perception Before Judgment: A reward model cannot penalize a defect it cannot "see." The authors found that training the vision tower on low-level artifact perception (like VAE reconstruction errors) is more important than deep language reasoning for detecting video glitches.
- Decide in the Prefix: For their Decision Reward Model, they found that fixing answer slots in the prompt prefix (rather than decoding them autoregressively) reduces latency and prevents the model from learning "ordering shortcuts" that inflate scores.
- The "Know but Cannot Tell" Gap: Strong VQA performance doesn't guarantee a model can write good captions. Captioning requires autonomous decision-making about what is important to describe, which is a harder task than answering a specific question.
Key Results
Kling-MLLM variants achieved state-of-the-art or highly competitive results across a battery of benchmarks:
- Audio: Kling-MLLM-397B-A17B achieved the lowest Word Error Rate (WER) on LibriSpeech (clean) and led the Chinese LATBench results.
- Video Understanding: The model led in five categories on video benchmarks, including DSI-Bench (spatial intelligence) and VidCapBench (captioning coverage).
- Human Preference: Refinements to the Text Encoder (multimodal adaptation and multi-layer fusion) resulted in a 1.27x gain in human preference scores over a standard Qwen baseline.
Limitations and Future Work
The authors acknowledge several remaining hurdles:
- Context vs. Dynamics: Balancing the need for high-resolution spatial detail with the long-context requirements of high-frame-rate video.
- Long-form Consistency: Maintaining referential consistency for subjects over very long durations remains a challenge.
- Inference Costs: The "Agentic PE" approach, while powerful, adds significant latency and compute costs due to tool calls and retrieval steps.
- Ambition vs. Safety: Reinforcement learning tends to make models "safer" (avoiding complex motion to reduce artifacts), which can lower the "peak vividness" of generated videos. Finding rewards that credit ambitious but executable content is an open problem.
Summary updated