Compositional Video Generation as Flow Equalization

Xingyi Yang, Xinchao WangSubmission to ICLR 2025 · Rejected

TL;DR

Existing text-to-video (T2V) diffusion models often drop concepts or swap actions when handling complex prompts. Vico introduces a training-free test-time optimization framework that models information propagation across both spatial and temporal attention layers as a network flow graph. By equalizing the "attention flow" across input tokens via a fast, differentiable min-max path-flow approximation, Vico ensures all concepts and actions are faithfully rendered.

What Problem Does This Paper Address?

While modern T2V models generate photorealistic scenes, they frequently fail on compositional prompts. Common failure modes include:

  • Missing subjects or spatial confusion: Dominant tokens suppress others (e.g., generating only a cat when asked for "a bird and a cat").
  • Semantic leakage & motion mixing: Attributes or actions bleed across subjects (e.g., a prompt describing "a whale flying a hot air balloon" results in a flying whale instead).

In text-to-image models, techniques like Attend-and-Excite fix this by steering spatial cross-attention maps. However, in video models, cross-attention only operates per-frame; subsequent temporal self-attention layers mix tokens across time. Relying purely on spatial cross-attention causes severe temporal flickering and fails to track motion dynamics.

Methods

Vico reframes compositional video generation as Flow Equalization:

[Text Tokens (Sources)] ---> [Spatial Cross-Attn + Temporal Self-Attn Graph] ---> [Video Tokens (Sink)]
                                        │
                                        ▼
                      Differentiable ST-Flow Estimation
                                        │
                                        ▼
             Test-Time Latent Update: max_x_t (min_i Attribution(v_i))
  1. Spatial-Temporal Attention Flow (ST-Flow): The entire model’s attention layers (both cross-attention and spatiotemporal self-attention) are structured as a layered directed capacity graph $G=(V, E)$. The influence of a text token $v_i$ on the final video output is formulated as the maximum network flow from source $v_i$ to an auxiliary sink node $v_t$ connected to all output tokens.
  2. Differentiable Subgraph Approximation: Standard max-flow algorithms (like Dinic's) are non-differentiable and take ~8 seconds per token. The authors prove that the max flow over any subgraph lower-bounds the graph's true max flow. They approximate ST-Flow by computing the maximum path flow (the bottleneck edge capacity along the strongest path).
  3. Vectorized Min-Max Matrix Multiplication: Path bottlenecks are computed efficiently by redefining matrix multiplication where addition is replaced by maximization and multiplication by minimization: $$C_{i,j} = \max_r \left( \min(A_{i,r}, B_{r,j}) \right)$$ A smooth version uses $\text{softmin}$/$\text{softmax}$ via the log-sum-exp trick to maintain differentiability.

  4. Fairness Optimization: During the first half of reverse diffusion, Vico updates the latent $x_t$ via gradient ascent on a max-min objective: $$\max_{x_t} \min_{v_i} \{A_1, \dots, A_K\}$$ boosting the influence of whichever key concept is currently least represented.

Datasets & Benchmarks

  • VBench: Evaluated particularly on Multiple Object Composition and Spatial Relations.
  • T2V-CompBench: Evaluated across all 5 benchmark dimensions (Consistent-Attribute Binding, Spatial Relations, Motion Binding, Action Binding, and Object Interactions).
  • Motion Composition (New): A newly curated evaluation set of 70 prompts of the form "obj1 is motion1 and obj2 is motion2". Captions generated by GRiT on the output video automatically verify whether both (object, motion) bindings appear.
  • Ref-DAVIS2017: Used for evaluating zero-shot video object segmentation to test whether ST-Flow provides better spatial-temporal attribution than standard cross-attention.

Key Insights and Ideas

  • Why cross-attention fails in video: Temporal self-attention layers act as information sinks that reshuffle semantics across frames. An attribution metric must account for the full chain of attention layers to capture object persistence and action dynamics.
  • Min-max flow replaces expensive graph algorithms: Formulating path bottlenecks as matrix operations enables GPU parallelization, achieving a ~100× speedup over exact flow solvers (0.037s vs. 8.0s per inference step).
  • Fairness via min-loss vs. variance loss: Penalizing the variance across all tokens over-constrains the diffusion trajectory and degrades visual fidelity. Maximizing only the minimum token's attribution selectively rescues omitted concepts without distorting dominant, coherent regions.

Key Results

  • Significant Compositional Gains: On VideoCrafterv2, Vico increases the Multiple Object Composition score from 40.66% to 73.55% (soft-flow), decisively outperforming Attend-and-Excite (66.62%) and Compel token re-weighting (49.16%).
  • Competitive with Supervised Models: On T2V-CompBench, training-free Vico on top of VideoCrafterv2 matches or outperforms retraining-based methods like LVD and VideoTetris in Action Binding (0.6020 vs. 0.5280) and Object Interaction (0.7800 vs. 0.7600).
  • Broad Model Generalization: Successfully adapts to 3D-UNets (VideoCrafterv2, AnimateDiff, ZeroScopev2) as well as modern Diffusion Transformers like Open-Sora and CogVideoX-2B (raising CogVideoX-2B VBench composition score from 53.70 to 63.21).
  • Practical Runtime: Adds minimal overhead (~45s per video on VideoCrafterv2 vs. 48s for Attend-and-Excite and 23s for baseline generation).

Limitations and Future Work

  • Lack of Explicit Attribute-Subject Binding: While Vico ensures all tokens (nouns and verbs) participate in the generation, it does not explicitly enforce which adjective binds to which noun.
  • Sensitivity of Latent Updates: Excessive gradient ascent steps on the latent code can distort realism and introduce artifacts; optimization must be carefully stopped (e.g., during the latter half of sampling).
  • Extension to Other Modalities: The authors suggest the underlying formulation of differentiable attention-graph equalization could be extended beyond diffusion models to long-context language modeling and concept erasure tasks.

Summary updated

Abstract

Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated unprecedented capability to transform natural language descriptions into stunning and photorealistic videos. Despite these promising results, a significant challenge remains: these models struggle to fully grasp complex compositional interactions between multiple concepts and actions. This issue arises when some words dominantly influence the final video, overshadowing other concepts. To tackle this problem, we introduce \textbf{Vico}, a generic framework for compositional video generation that explicitly ensures all concepts are represented properly. At its core, Vico analyzes how input tokens influence the generated video, and adjusts the model to prevent any single concept from dominating. Specifically, Vico extracts attention weights from all layers to build a spatial-temporal attention graph, and then estimates the influence as the \emph{max-flow} from the source text token to the video target token. Although the direct computation of attention flow in diffusion models is typically infeasible, we devise an efficient approximation based on subgraph flows and employ a fast and vectorized implementation, which in turn makes the flow computation manageable and differentiable. By updating the noisy latent to balance these flows, Vico captures complex interactions and consequently produces videos that closely adhere to textual descriptions. We apply our method to multiple diffusion-based video models for compositional T2V and video editing. Empirical results demonstrate that our framework significantly enhances the compositional richness and accuracy of the generated videos.
BibTeX
@misc{yang2024compositional,
  author = {Xingyi Yang and Xinchao Wang},
  title = {{Compositional Video Generation as Flow Equalization}},
  year = {2024},
  eprint = {2407.06182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.06182},
}

Loading related content...

Figure