Abstract
Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.
Introduction
While modern text-to-video (T2V) models like Sora or CogVideoX produce stunning visuals, they often struggle with precise spatial and motion instructions. For example, a request for "a cat running from left to right behind a sofa" might result in incorrect directions, misplaced objects, or multiple subjects.
Existing solutions to this alignment problem generally fall into two categories: computationally expensive fine-tuning on layout data or memory-intensive attention map manipulation during inference. The latter often doubles or triples memory requirements, making it difficult to run state-of-the-art 5B+ parameter models on consumer-grade hardware. Video-MSG offers a "third way": a training-free framework that provides precise control without modifying model weights or exceeding standard memory budgets.
Methodology
The core of Video-MSG is the creation of a "Video Sketch"—a low-fidelity draft that guides the final high-quality generation. This process occurs in three main stages:
- Background Planning: A Multimodal Large Language Model (MLLM) generates a detailed background description. A text-to-image (T2I) model creates the scene, and an image-to-video (I2V) model adds natural environmental motion.
- Foreground & Trajectory Planning: To ensure physical plausibility, the system detects existing objects in the background. The MLLM then plans trajectories for new foreground objects using bounding boxes. These objects are generated and composited into the background to create the Video Sketch.
- Structured Noise Initialization: This is the framework's key innovation. Instead of starting with pure random noise, the Video Sketch undergoes noise inversion. This turns the sketch into a specific noise pattern that "hints" at the desired layout. The T2V model then denoises this structured starting point into the final video.
Evaluation and Results
Video-MSG was evaluated using T2VCompBench (for compositional skills like attribute binding and numeracy) and VBench (for general video quality and temporal consistency). Testing on backbones like VideoCrafter2 and CogVideoX-5B revealed significant improvements:
- Motion Binding: On the CogVideoX-5B backbone, Video-MSG improved motion binding scores by 0.1544.
- Spatial & Numeracy: The framework showed gains in spatial relationships (+0.06) and numeracy (+0.10), indicating better object placement and counting accuracy.
- Efficiency: Unlike attention-based methods that often fail on 80GB A100 GPUs when paired with large models, Video-MSG runs comfortably on 48GB A6000 hardware.
Conclusion
Video-MSG demonstrates that T2V models do not necessarily need to be retrained for layout control if provided with a structured starting point. By grounding MLLM planning in visual context and utilizing noise inversion, the framework achieves superior alignment with text prompts. While current limitations include handling complex physical interactions or dynamic attribute changes (e.g., color shifts), Video-MSG provides a scalable, training-free path for enhancing the controllability of large-scale video generation models.
Summary updated