TL;DR
This paper introduces Reg-DPO, a novel framework for improving video generation quality by addressing key challenges in data construction, training stability, and memory efficiency. It proposes a GT-Pair strategy for high-quality preference data generation and an SFT-regularized DPO loss to stabilize training, leading to superior video generation compared to existing methods.
What Problem Does This Paper Address?
Direct Preference Optimization (DPO) has emerged as an efficient method for aligning generative models with human preferences. However, its application to video generation faces significant hurdles. Existing DPO methods, often adapted from image domains, struggle with the complex spatiotemporal nature of videos, leading to issues like:
- Costly Data Construction: Creating high-quality preference pairs for videos is labor-intensive and expensive, requiring extensive human annotation or sophisticated automatic evaluators.
- Training Instability: DPO can suffer from rapid convergence and distribution shifts, leading to model collapse and poor generation quality, especially with large models.
- High Memory Consumption: Video generation models, particularly those exceeding 10 billion parameters, combined with the multi-frame nature of video data and DPO's requirements, demand substantial GPU memory, limiting scalability.
This paper aims to overcome these limitations by proposing a comprehensive framework that tackles data construction, algorithmic stability, and memory optimization simultaneously.
Methods
The paper introduces three core innovations:
- GT-Pair for Data Construction: Instead of relying on human annotations or solely model-generated pairs, GT-Pair leverages ground-truth (GT) videos as positive samples and their GT-conditioned generated counterparts as negatives. This automatically generates high-quality preference pairs without external labeling, improving discriminability and reducing costs.
- Reg-DPO for Training Stability: To address DPO's inherent instability, the authors propose Reg-DPO, which incorporates a Supervised Fine-Tuning (SFT)-based regularization term into the DPO loss. This regularization term, weighted by a dynamic parameter
r, acts as a constraint on positive samples, enhancing training stability and preventing distribution shift. A dynamic weighting scheme forr(set toDGR) is proposed for optimal performance. - Memory Optimization Scheme: To handle the memory demands of large video models, the authors combine the Fully Sharded Data Parallel (FSDP) framework with several techniques: Flash Attention, Context Parallelism, fully parallelized pair computation, prompt pre-encoding, model offloading for frozen modules, and refined graph and memory management. This multi-pronged approach significantly reduces memory usage, enabling training of larger models.
Datasets
The paper introduces two new high-quality datasets and utilizes existing ones:
- Human-I2V: Focuses on human activity videos, comprising training subsets (Human-I2V-GT-Pair and Human-I2V-Annotated_Pair) and a test set (Human-I2V-Test).
- OpenVid-12K: Derived from OpenVid-1M, it covers diverse categories and includes training subsets for Image-to-Video (I2V) and Text-to-Video (T2V) tasks (OpenVid-12K-I2V, OpenVid-12K-T2V) and a test set (OpenVid-12K-Test).
Key Insights and Ideas
- Real videos are superior preference anchors: Using ground-truth videos as positive samples in preference learning is more effective than relying solely on model-generated data, even with manual annotations. This is because real videos inherently possess higher quality, consistency, and semantic completeness.
- SFT regularization is key to DPO stability: While DPO optimizes the difference between positive and negative sample scores, it lacks direct supervision on the samples themselves. Adding an SFT-like regularization term, even if targeting only positive samples, implicitly constrains negative samples and prevents the model from diverging too far from the reference, thus stabilizing training.
- Dynamic regularization is crucial: A fixed regularization strength might be suboptimal. The paper proposes a dynamic
r = DGRto adapt the regularization to the training dynamics, ensuring effectiveness across different stages and scenarios. - Memory optimization is a system-level challenge: Addressing memory constraints for large video models requires a combination of techniques, not just one solution. The paper demonstrates that integrating multiple strategies (Flash Attention, CP, etc.) with FSDP is essential for achieving significant memory savings and scalability.
Key Results
- GT-Pair outperforms manual annotation: Experiments show that GT-Pair data leads to significantly better video quality and prompt adherence compared to manually annotated pairs, even when the latter is used in larger quantities.
- Reg-DPO achieves state-of-the-art performance: Reg-DPO consistently outperforms baseline, SFT, and vanilla DPO methods on both I2V and T2V tasks across multiple datasets and evaluation metrics (GSB and VBench). For instance, on Human-I2V-GT-Pair, Reg-DPO achieved a VQ Score of 2.05 and PF Score of 1.73, surpassing DPO and SFT.
- Significant memory savings: The proposed memory optimization scheme, when combined with FSDP, triples the effective training capacity, enabling training of ultra-large models.
- Stabilized training: Reg-DPO significantly mitigates the instability and collapse issues observed in vanilla DPO, resulting in smoother training curves and consistently clear video outputs.
Limitations and Future Work
- Limitations:
- The paper notes that while Reg-DPO improves stability, the optimal value of
rmight be task-dependent.
- The paper notes that while Reg-DPO improves stability, the optimal value of
- The memory optimization strategy, while effective, involves some I/O overhead (e.g., module offloading).
- Future Work:
- The authors suggest that their unified and memory-optimized framework could be extended to support more complex video generation tasks and larger models.
- The paper mentions releasing the data and code, which will enable further research and development in this area.
Summary updated