Introduction
Generating high-quality video from text prompts is a significant milestone in generative AI, yet accurately incorporating multiple specific subjects remains a formidable challenge. Current Diffusion Transformer (DiT) models often suffer from two primary immersion-breaking issues: scale inconsistency and permutation sensitivity. Scale inconsistency occurs when the model fails to maintain realistic proportions between subjects (e.g., generating a ladybug the size of a lion), often because it is misled by the zoom levels of reference images. Permutation sensitivity refers to the model's tendency to change the output significantly based on the order in which reference images are provided.
To address these limitations, we introduce MoFu (Scale-Aware Modulation and Fourier Fusion), a unified framework designed to ensure subjects appear at natural relative scales and remain unaffected by the sequence of reference inputs. By leveraging Large Language Models (LLMs) for spatial reasoning and Fourier-based math for feature fusion, MoFu establishes a new state-of-the-art in multi-subject video synthesis.
The Core Challenges in Multi-Subject Generation
1. Scale Inconsistency
In multi-subject generation, reference images are often sourced from disparate datasets or captured at different focal lengths. If a reference photo of a bowl is a close-up and a photo of a person is a wide shot, a naive model may struggle to reconcile these perspectives, resulting in physically implausible size ratios. Existing architectures lack an explicit mechanism to interpret the "natural" scale of objects relative to one another based on the textual context.
2. Permutation Sensitivity
Most current models process reference images sequentially, typically through concatenation. This introduces a spatial bias where the first image or the central image receives disproportionate attention. Consequently, swapping the order of input images can lead to subject distortion, disappearance, or inconsistent behavior, undermining the reliability of the generation process.
The MoFu Framework
MoFu integrates three innovative components into a DiT backbone to solve these issues:
Scale-Aware Modulation (SMO)
MoFu employs a frozen LLM to act as a "Scale Architect." The LLM analyzes the text prompt to extract implicit semantic scale relationships—for instance, understanding that a sparrow is inherently smaller than an elephant. This semantic information is converted into modulation parameters that dynamically adjust the DiT’s internal features, forcing the model to respect realistic proportions regardless of the reference image zoom levels.
Fourier Fusion
To achieve true order independence, MoFu moves reference image features into the frequency domain using a Fast Fourier Transform (FFT). Because the addition of frequencies is a commutative operation ($A + B = B + A$), the resulting fused representation is mathematically guaranteed to be permutation-invariant. This prevents the model from favoring one reference image over another based on its position in the input sequence.
Scale-Permutation Stability Loss (SPSL)
MoFu is trained using a dual-purpose loss function:
- Scale Loss: Re-weights the training process to focus on subjects with larger relative areas, ensuring the model prioritizes the most prominent subjects.
- Permutation Loss: Penalizes the model if its output changes when reference images are shuffled, enforcing consistency across different input orders.
Data and Benchmarking
Recognizing the lack of robust datasets for this task, the authors developed two new resources:
- MoFu-1M: A high-quality training dataset featuring 1 million unique video clips. Each clip is paired with multiple reference images stored as RGBA to preserve subject masks and identity fidelity.
- MoFu-Bench: A specialized evaluation benchmark consisting of 1,000 subject-text pairs. It includes "controlled variations," such as zoomed-in references and shuffled input orders, specifically designed to stress-test scale and permutation stability.
Key Insights and Results
The "Near-Orthogonality" Trick
MoFu utilizes a theorem from high-dimensional probability: random vectors in high-dimensional space tend to be nearly orthogonal. By aggregating features in the frequency domain via simple summation, the framework creates a unified representation that preserves individual subject identities without the computational overhead or bias inherent in sequential processing.
Performance Metrics
MoFu was evaluated against several state-of-the-art (SOTA) models, including Phantom, SkyReels-A2, and VACE.
- Quantitative Success: MoFu achieved the highest scores across nearly all metrics on MoFu-Bench. Notably, it led in Aesthetics (0.401) and ScaleScore (0.585). The ScaleScore, calculated via GPT-4o, specifically measures the alignment between generated sizes and the textual prompt.
- Qualitative Superiority: In head-to-head comparisons, MoFu consistently maintained correct orientations and realistic proportions. For example, it successfully kept a sparrow small relative to an elephant even when the sparrow's reference photo was a close-up, a feat other models failed to achieve.
Limitations and Future Work
While MoFu represents a significant leap forward, some challenges remain:
- Fine-Grained Dynamics: The model can occasionally struggle with extremely complex or high-speed temporal interactions between multiple subjects.
- Reference Dependency: The framework relies on high-quality, segmented reference inputs; performance may diminish if reference images are noisy or poorly cropped.
- Domain Specifics: Future iterations could benefit from domain-specific priors, such as specialized human motion models, to further enhance performance in specific categories like athletics or dance.
Conclusion
MoFu addresses the critical issues of scale and order in multi-subject video generation by combining LLM-driven spatial reasoning with frequency-domain mathematics. By ensuring permutation invariance and scale consistency, MoFu provides a more robust and physically plausible framework for complex video synthesis, outperforming existing SOTA models in both identity fidelity and visual quality.
Summary updated