Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization

Jialu Li, Shoubin Yu, Han Lin, Jaemin Cho, Jaehong Yoon, Mohit BansalSubmission to ICLR 2026 · Rejected

TL;DR

VIDEO-MSG (Multimodal Sketch Guidance) is a training-free framework that allows text-to-video (T2V) diffusion models to follow complex spatial, trajectory, and numerical prompts. By orchestrating multimodal LLMs, object detectors, and segmentation models to build a rough draft video ("VIDEO SKETCH"), it guides downstream video diffusion models via structured noise inversion—drastically improving text alignment without requiring retraining or memory-heavy attention map manipulation.

What Problem Does This Paper Address?

While modern text-to-video diffusion models can generate visually stunning footage, they notoriously struggle with compositional text alignment:

  • Motion direction and trajectory: Getting an object to consistently move left, descend, or follow a designated path.
  • Numeracy and layout: Correctly generating precise counts (e.g., "six penguins") and relative positions (e.g., "to the right of the umbrella").

Existing remedies rely on layout guidance (e.g., bounding boxes or optical flow), but they typically require fine-tuning the video model or running gradient-based optimizations on attention maps during inference (such as LVD). These approaches demand immense GPU memory, making them nearly impossible to scale to modern multi-billion-parameter foundation models (e.g., CogVideoX-5B cannot run LVD even on an 80GB A100).

Methods

VIDEO-MSG achieves training-free layout and motion control using a three-stage pipeline that consumes only around 16 GB of VRAM:

[Prompt] ──> 1. Background Planning (MLLM + T2I + I2V)
                   │
                   ▼
             2. Foreground Planning (RAM + Grounding-DINO + MLLM + SAM)
                   │
                   ▼ (Draft "VIDEO SKETCH")
             3. Structured Noise Inversion (3D VAE + DPM-Solver++ on T2V Backbone)
                   │
                   ▼
             [Final Coherent Video]
  1. Background Planning: An MLLM (GPT-4o) extracts and enriches only the static scene/environment description, strictly removing foreground objects. A text-to-image (T2I) model (FLUX.1-dev or SDXL) renders the scene, which is animated using an image-to-video (I2V) model (CogVideoX-5B) to provide natural environmental dynamics (e.g., swaying leaves, moving water) under a static camera constraint.
  2. Foreground Object Layout & Trajectory Planning:
  3. Open-vocabulary detectors (RAM + Grounding-DINO) detect existing elements in the background to provide spatial context.
  4. The MLLM generates frame-by-frame bounding box coordinates and motion reasoning for foreground objects over 25 frames.
  5. Foreground objects are synthesized via T2I, segmented using SAM to strip isolated backgrounds, and pasted onto the background frames according to the bounding boxes. This creates a rough draft video called VIDEO SKETCH.
  6. Structured Noise Initialization: Rather than denoising from pure Gaussian noise at timestep $T$, the VIDEO SKETCH is encoded using a 3D VAE. Forward diffusion injects noise up to an intermediate timestep $t_{\text{inv}} = \alpha T$ (where $\alpha \in (0, 1)$). The downstream T2V model denoises from $t_{\text{inv}}$ onward using DPM-Solver++, naturally refining coarse draft boundaries into realistic, temporally coherent video. The inversion ratio $\alpha$ is dynamically predicted by an LLM based on prompt difficulty.

Datasets

The authors evaluate VIDEO-MSG without additional training on standard open benchmarks:

  • T2V-CompBench: A benchmark evaluating compositional video generation across categories such as motion binding, numeracy, spatial relationships, attribute binding, and interactions.
  • VBench: Used primarily to quantify motion smoothness and overall visual quality.

Key Insights and Ideas

  • Background detection grounds the MLLM: Prompting an MLLM with only raw background images often leads to physical hallucinations (e.g., placing a dog floating in mid-air). Providing explicit bounding box annotations of background regions (e.g., {"label": "path", "box": [...]}) drastically anchors the MLLM's spatial reasoning.
  • T2I + I2V outperforms direct T2V for background synthesis: Direct T2V generators often ignore "static camera" prompts, introducing erratic camera sweeps that confuse foreground insertion. A strong T2I model animated via I2V reliably enforces static cameras while retaining high visual fidelity.
  • Inversion circumvents attention hijacking: Instead of performing backpropagation through attention maps at every denoising step, starting reverse diffusion from an intermediate noisy latent $z_{t_{\text{inv}}}$ guides spatial layout and motion naturally while letting diffusion priors handle object consistency and texture blending.
  • Adaptive noise scheduling ($\alpha$): Prompts requiring strict trajectories (e.g., moving left-to-right) benefit from lower $\alpha$ (less noise, stricter adherence to the sketch). In contrast, complex actions or dynamic state changes benefit from higher $\alpha$ (allowing the diffusion backbone's natural motion priors to dominate).

Key Results

Experiments on VideoCrafter2 and CogVideoX-5B show substantial gains over vanilla models and prior layout-guided baselines:

  • T2V-CompBench Improvements (CogVideoX-5B):
  • Motion Binding: Increases from 0.2943 to 0.4487 (+52.46% relative gain), outperforming closed-source models such as Gen-3 (0.3111).
  • Numeracy: Increases from 0.2603 to 0.3647 (+40.11% relative gain).
  • Spatial Relationships: Increases from 0.5461 to 0.6070 (+11.15% relative gain).
  • Comparison to LVD: On VideoCrafter2, VIDEO-MSG outperforms the attention-manipulation baseline LVD across almost all compositional categories, beating it in motion binding by 0.1554 while requiring dramatically less memory (VIDEO-MSG runs on a 48GB GPU; LVD runs out of memory even on an 80GB A100).
  • VBench Smoothness: Reaches 99.01 with LLM-controlled noise inversion, confirming that noise initialization does not induce temporal jitter.

Limitations and Future Work

  • Planner Dependency: Performance depends heavily on the reasoning strength of the planner. Replacing GPT-4o with an open-source model like Qwen2-VL-7B yields smaller improvements (Spatial score of 0.5166 vs. 0.6447).
  • Bounding-Box Representational Limits: The method does not improve dynamic attribute changes or complex multi-agent interactions (e.g., handshakes, singing), as bounding box trajectories cannot express intricate, deformation-heavy physical interactions.
  • Full Semantic Adherence: While spatial layout and counts improve noticeably, the pipeline does not guarantee pixel-perfect adherence to fine-grained textual attributes.

Summary updated

Abstract

Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.
BibTeX
@misc{li2025training,
  author = {Jialu Li and Shoubin Yu and Han Lin and Jaemin Cho and Jaehong Yoon and Mohit Bansal},
  title = {{Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization}},
  year = {2025},
  eprint = {2504.08641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.08641},
}

Loading related content...

Figure