Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

Kling Team, Dong Chen, Jialu Chen, Xinlong Chen, Yifan Dai, Haotian Dong, Binzong Geng, Yushuo Guan, Minghui Guo, Guanlin He, Guanxiao He, Jingyun Hua, Yanwei Huang, Yiyan Ji, Xueping Jia, Yizhuo Jia, Bozhou Li, Jiaze Li, Min Li, Shihao Li, Shujie Li, Shengyi Liao, Weihong Lin, Shiyuan Liu, Xinyu Liu, Siyang Mao, Xiaoru Qu, Dianmo Sheng, Yang Shi, Jiafu Tang, Longxiang Tang, Yuqi Tang, Chengzhuo Tong, Kuo Wang, Qixun Wang, Rui Wang, Tianduo Wang, Yuran Wang, Zekun Wang, Zihao Wang, Boyang Xia, Yifan Xu, Liu Yang, Zhenlong Yuan, Bohan Zeng, Mujin Zhang, Yuanxing Zhang, Zhicheng Zhang, Zhuoran Zhang, Yuanhong Zheng, Kun Gai, Chengru Song, Pengfei WanarXiv 2026 · arXiv

Building high-quality video generation models is no longer just about rendering pretty pixels; it’s about semantic control. As the field moves from short clips to coherent audiovisual sequences, the bottleneck has shifted toward how well a model "understands" the world it is trying to create.

In this technical report, the Kling Team pulls back the curtain on the "understanding stack" behind their Kling models. They argue that video quality is a direct projection of multimodal understanding across the entire pipeline: from how data is captioned to how prompts are enhanced and how rewards are modeled.

TL;DR

The Kling Team presents a comprehensive framework for video generation where quality is driven by a unified multimodal foundation model (Kling-MLLM). Their core thesis is that generation is a "projection of understanding" across a four-stage chain—Data, Conditioning, Alignment, and Evaluation—prioritizing dense semantic descriptions and perceptual training over raw language ability.

Multimodal understanding along the data–conditioning–alignment–evaluation chain behind Kling.

What Problem Does This Paper Address?

Current video generation systems often struggle with "semantic drift"—hallucinating actions, losing subject identity across shots, or failing to synchronize audio and video. The authors identify that standard multimodal benchmarks (like VQA) don't capture the specific requirements of video generation, such as camera language, spatial relations, and dense temporal changes.

Furthermore, Diffusion Transformers (DiTs) are trained on far fewer samples than Large Language Models (LLMs). This creates a "data efficiency" gap: every training video must carry as much supervision as possible to teach the model complex controls.

Methods

The authors propose a unified architecture and a modular pipeline to solve these issues:

  1. Kling-MLLM: A shared foundation model (ranging from 27B to a 397B Mixture-of-Experts) that unifies images, videos, and audio into a single autoregressive interface. It uses a specialized audio tower and a visual path (ViT + Vision Aligner) to map all inputs into a common sequence.
  2. The Semantic Contract: Instead of vague, fluent captions, they use "Generation-Oriented Captioning." This treats captions as a dense data structure specifying shot boundaries, timestamps, OCR text, and synchronized audio events.
  3. Reference-Grounded Captioning (RGC): A novel approach for controllable generation where captions explicitly bind reference assets (like a character portrait or a voice track) to specific entities and time intervals in the target video.
  4. Prompt Enhancement (PE): A two-tier system. Agentic PE refines the user's story and retrieves compelling examples, while Basic PE (a trained rewriter) translates that intent into the specific "caption dialect" the generator understands.
  5. Dual Reward Modeling: They split reward modeling into a General Reward Model (GRM) for explainable, localized critiques (using bounding boxes and timestamps) and a Decision Reward Model (DRM) for fast, multidimensional scoring during reinforcement learning.
Kling-MLLM architecture showing the visual and audio encoding paths.

Datasets

The team utilized a massive mixture of data, summarized in their high-level supervision pipeline:

  • Public Data: Ingested via an automated "AutoResearch" agent that monitors Hugging Face for new datasets.
  • Kling-Specific Task Data: Internal data targeting specific perception gaps like camera behavior, multi-shot structure, and bilingual OCR.
  • Multimodal Agent Data: Data from tool-use loops and 3D Blender-based tasks to teach spatial reasoning.
High-level normalization and supervision pipelines by modality.

Key Insights and Ideas

  • Description Density is King: Because DiTs can't be scaled with the same data volume as LLMs, the quality of text per second of video is the primary lever for data efficiency.
  • Perception Before Judgment: A reward model cannot penalize a defect it cannot "see." The authors found that training the vision tower on low-level artifact perception (like VAE reconstruction errors) is more important than deep language reasoning for detecting video glitches.
  • Decide in the Prefix: For their Decision Reward Model, they found that fixing answer slots in the prompt prefix (rather than decoding them autoregressively) reduces latency and prevents the model from learning "ordering shortcuts" that inflate scores.
  • The "Know but Cannot Tell" Gap: Strong VQA performance doesn't guarantee a model can write good captions. Captioning requires autonomous decision-making about what is important to describe, which is a harder task than answering a specific question.

Key Results

Kling-MLLM variants achieved state-of-the-art or highly competitive results across a battery of benchmarks:

  • Audio: Kling-MLLM-397B-A17B achieved the lowest Word Error Rate (WER) on LibriSpeech (clean) and led the Chinese LATBench results.
  • Video Understanding: The model led in five categories on video benchmarks, including DSI-Bench (spatial intelligence) and VidCapBench (captioning coverage).
  • Human Preference: Refinements to the Text Encoder (multimodal adaptation and multi-layer fusion) resulted in a 1.27x gain in human preference scores over a standard Qwen baseline.
Audio benchmarks comparing Kling-MLLM variants against other frontier models.

Limitations and Future Work

The authors acknowledge several remaining hurdles:

  • Context vs. Dynamics: Balancing the need for high-resolution spatial detail with the long-context requirements of high-frame-rate video.
  • Long-form Consistency: Maintaining referential consistency for subjects over very long durations remains a challenge.
  • Inference Costs: The "Agentic PE" approach, while powerful, adds significant latency and compute costs due to tool calls and retrieval steps.
  • Ambition vs. Safety: Reinforcement learning tends to make models "safer" (avoiding complex motion to reduce artifacts), which can lower the "peak vividness" of generated videos. Finding rewards that credit ambitious but executable content is an open problem.

Summary updated

Abstract

Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.
BibTeX
@misc{team2026behind,
  author = {Kling Team and Dong Chen and Jialu Chen and Xinlong Chen and Yifan Dai and Haotian Dong and Binzong Geng and Yushuo Guan and Minghui Guo and Guanlin He and Guanxiao He and Jingyun Hua and Yanwei Huang and Yiyan Ji and Xueping Jia and Yizhuo Jia and Bozhou Li and Jiaze Li and Min Li and Shihao Li and Shujie Li and Shengyi Liao and Weihong Lin and Shiyuan Liu and Xinyu Liu and Siyang Mao and Xiaoru Qu and Dianmo Sheng and Yang Shi and Jiafu Tang and Longxiang Tang and Yuqi Tang and Chengzhuo Tong and Kuo Wang and Qixun Wang and Rui Wang and Tianduo Wang and Yuran Wang and Zekun Wang and Zihao Wang and Boyang Xia and Yifan Xu and Liu Yang and Zhenlong Yuan and Bohan Zeng and Mujin Zhang and Yuanxing Zhang and Zhicheng Zhang and Zhuoran Zhang and Yuanhong Zheng and Kun Gai and Chengru Song and Pengfei Wan},
  title = {{Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling}},
  year = {2026},
  eprint = {2611.04916},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04916},
}
Figure