Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration

Haoran Bai, Xiaoxu Chen, Canqian Yang, Zongyao He, Sibin Deng, Ying Chen

Abstract

Restoring low-quality video into high-fidelity, photorealistic content remains a significant challenge in computer vision. While generative Text-to-Video (T2V) models have demonstrated remarkable synthesis capabilities, adapting them for restoration often leads to "distribution drift," where models lose their generative priors when fine-tuned on degraded data. We introduce Vivid-VR, a framework that utilizes a novel concept distillation strategy and a redesigned ControlNet architecture to bridge this gap. By aligning training samples with the model's internal generative concepts, Vivid-VR restores videos with superior texture realism and temporal stability, outperforming current state-of-the-art methods on both real-world and AI-generated benchmarks.

Introduction

Video restoration is an inherently ill-posed problem characterized by the need to reconstruct missing high-frequency details from degraded inputs. Traditional methods often produce over-smoothed or "plastic" textures to avoid artifacts. Recent attempts to leverage powerful generative models for this task have encountered the "alignment problem": the gap between the model's high-quality training distribution and the messy reality of low-quality video. This often results in flickering artifacts or a loss of the model's original generative "imagination."

Methodology

Vivid-VR addresses these challenges through three key architectural and procedural innovations designed to preserve the "vividness" of the original generative model while maintaining strict fidelity to the input video.

1. Concept Distillation Training

To combat distribution drift, Vivid-VR employs a "concept distillation" strategy. Instead of training solely on standard low-quality/high-quality pairs, the authors use a pre-trained T2V model (CogVideoX) to generate "distilled" versions of training videos. By adding noise to a source video and then denoising it using the T2V model guided by text prompts, they create targets that are perfectly aligned with the model's internal conceptual space. Blending these distilled samples into the training process ensures the model retains its ability to generate realistic textures during restoration.

2. Architectural Enhancements

The framework introduces a Control Feature Projector, a lightweight CNN-based module that filters degradation artifacts from input video latents before they enter the main pipeline. This prevents the model from inadvertently "restoring" noise. Furthermore, a Dual-branch ControlNet Connector is utilized. This module uses an MLP branch for basic feature mapping and a Cross-Attention branch to dynamically retrieve structural details from the control signal, ensuring the output remains faithful to the original video's geometry.

The Vivid-VR architecture, featuring a Control Feature Projector and a dual-branch ControlNet connector integrated with a Diffusion Transformer.

Experiments and Results

The model was trained on a curated dataset of 500,000 high-quality text-video pairs derived from a massive pool of 3 million HD videos. Evaluation was conducted across synthetic (SPMCS, YouHQ40), real-world (VideoLQ, UGC50), and AI-generated (AIGC50) benchmarks.

Quantitative Performance

Vivid-VR consistently achieves top rankings in no-reference quality metrics such as NIQE and MUSIQ. On the UGC50 real-world dataset, it achieved a NIQE score of 4.361, a significant improvement over competing methods like MGLD (4.633). These scores reflect the model's ability to produce natural-looking images that lack the artificial "hallucinations" common in other generative restorers.

Qualitative Comparison

Visually, Vivid-VR excels in rendering fine-grained textures such as human hair, architectural details, and complex landscapes. Unlike image-based restorers that often suffer from inter-frame flickering, Vivid-VR maintains high temporal consistency, producing stable and fluid video outputs.

Comparison of Vivid-VR with existing methods on real-world (left) and AIGC (right) video restoration, demonstrating superior detail and texture recovery.

Conclusion

Vivid-VR represents a significant step forward in generative video restoration by successfully decoupling the "cleaning" of noise from the "creation" of detail. By distilling concepts from large-scale Diffusion Transformers, it achieves a level of photorealism previously unattainable in the field. While the current 5-billion parameter architecture requires substantial computational resources, future work will focus on efficiency through "one-step" diffusion techniques to bring these high-quality restoration capabilities to consumer-grade hardware.

Summary updated

BibTeX
@inproceedings{bai2026vivid,
  author = {Haoran Bai and Xiaoxu Chen and Canqian Yang and Zongyao He and Sibin Deng and Ying Chen},
  title = {{Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=YV5Zgv8pdg},
}
Figure