Abstract
Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated unprecedented capability to transform natural language descriptions into stunning and photorealistic videos. Despite the promising results, a significant challenge remains: these models struggle to fully grasp complex compositional interactions between multiple concepts and actions. This issue arises when some words dominantly influence the final video, overshadowing other concepts. To tackle this problem, we introduce Vico, a generic framework for compositional video generation that explicitly ensures all concepts are represented properly. At its core, Vico analyzes how input tokens influence the generated video, and adjusts the model to prevent any single concept from dominating. Specifically, Vico extracts attention weights from all layers to build a spatial-temporal attention graph, and then estimates the influence as the max-flow from the source text token to the video target token. Although the direct computation of attention flow in diffusion models is typically infeasible, we devise an efficient approximation based on subgraph flows and employ a fast and vectorized implementation, which in turn makes the flow computation manageable and differentiable. By updating the noisy latent to balance these flows, Vico captures complex interactions and consequently produces videos that closely adhere to textual descriptions. We apply our method to multiple diffusion-based video models for compositional T2V and video editing. Empirical results demonstrate that our framework significantly enhances the compositional richness and accuracy of the generated videos.
Addressing Compositional Failure in Video Generation
While current Text-to-Video (T2V) models like VideoCrafter or AnimateDiff produce stunning visuals, they are often "compositionally challenged." When presented with complex prompts involving multiple subjects or specific actions, these models frequently fail to represent all elements accurately.
The authors identify four specific failure modes that Vico aims to resolve:
- Missing Subject: An object specified in the prompt simply does not appear in the generated video.
- Spatial Confusion: Objects are placed in incorrect locations relative to each other or their environment.
- Semantic Leakage: Attributes of one object "bleed" into another (e.g., a bird appearing with cat-like features).
- Motion Mixing: An action intended for one subject is incorrectly attributed to another.
The root cause of these issues is an imbalance in attention. In standard diffusion models, certain "dominant" tokens (words) hijack the model's attention layers, overshadowing other concepts during the generation process.
Methodology: Flow Equalization
Vico introduces the principle of Flow Equalization to ensure every concept in a prompt is represented fairly.
Spatial-Temporal Attention Flow (ST-flow)
Rather than relying on simple cross-attention maps—which often ignore temporal consistency and only look at individual layers—Vico treats the diffusion model as a directed graph. In this graph, nodes represent text and video tokens, and edges are weighted by attention scores. The influence of a specific word is calculated as the "max-flow" from the text token to the video pixels.
Differentiable Approximation
Calculating exact max-flow is computationally expensive and not naturally differentiable. Vico overcomes this by using a "Min-Max Path Flow" approximation. By implementing a specialized "min-max matrix multiplication," the framework can estimate this flow in a fully vectorized, differentiable manner that is approximately 100x faster than exact methods.
Test-Time Optimization
During generation, Vico performs a "fairness" check at each step. It identifies tokens with the least influence and performs a rapid optimization to update the latent code, "boosting" underrepresented concepts before the next frame is generated.
Evaluation and Key Results
The researchers evaluated Vico across several benchmarks, including VBench (focusing on spatial relations and multi-object composition) and Ref-DAVIS2017 (for zero-shot video segmentation). They also introduced a new Motion Composition dataset consisting of 70 custom prompts designed to test object-motion pairs.
Key findings include:
- Quantitative Performance: On VideoCrafterv2, Vico improved the "Multiple Object" score from 40.66% to 73.55%, representing a significant leap in compositional accuracy.
- Temporal Consistency: Vico's ST-flow demonstrated superior accuracy and consistency in object tracking compared to standard attention-based methods, resulting in less flickering.
- Efficiency: Despite the additional graph calculations, Vico remains efficient. The "soft" version generates video in about 45 seconds, comparable to other optimization-based methods like Attend-and-Excite (48s) and roughly twice the time of the base model (23s).
Discussion and Future Directions
The core philosophy of Vico is that every word in a prompt deserves an "equal opportunity" to influence the final output. By framing video generation as a resource allocation problem, the authors successfully apply graph theory to solve deep learning challenges.
However, some limitations remain. Vico currently struggles with explicit attribute binding (e.g., ensuring a "red ball" and "blue cube" don't swap colors). Furthermore, there is a delicate balance between optimizing for composition and maintaining overall semantic coherence; overly aggressive optimization can lead to nonsensical results. Future work may explore applying "flow equalization" to other domains, such as language modeling or concept erasure in generative models.
Summary updated