Abstract
Controlling video synthesis remains a significant challenge due to the inherent complexity of temporal dynamics and the limitations of unimodal conditioning. We introduce MMVID, a multimodal framework designed to bridge the flexibility gap in video generation. By allowing users to specify content ("what") through images or masks and motion ("how") via natural language, MMVID provides a highly controllable synthesis pipeline. Our architecture utilizes a bidirectional transformer coupled with a novel Video Consistency Estimation task to ensure temporal stability. Experimental results across four benchmarks, including a newly introduced Multimodal VoxCeleb dataset, demonstrate that MMVID achieves state-of-the-art performance in visual quality, instruction following, and long-term consistency.
Introduction
Traditional video generation models are often restricted by a "unimodal" bottleneck, typically relying on either a text prompt or a single starting image. This creates a significant usability gap: while images are excellent for defining specific characters or backgrounds, they struggle to convey complex trajectories. Conversely, text prompts can describe motion but often lack the precision required to define fine-grained visual details. Furthermore, many autoregressive models suffer from "drift" or "exposure bias," where errors accumulate over time, leading to blurry or inconsistent frames.
MMVID addresses these issues by treating video synthesis as a multimodal completion task. By leveraging a bidirectional transformer, the model can "plan" an entire sequence simultaneously, ensuring that the generated motion is physically plausible and visually consistent from start to finish.
Methodology
MMVID employs a two-stage framework to transform video synthesis into a discrete sequence prediction problem.
Stage 1: Discrete Representation
The model utilizes a VQGAN (Vector Quantized Generative Adversarial Network) to compress high-dimensional video frames into a grid of discrete visual tokens. This compression allows the model to operate in a lower-dimensional latent space while preserving the essential textures and structures of the original video.
Stage 2: Bidirectional Transformer and Multimodal Conditioning
Unlike standard autoregressive models that predict tokens sequentially, MMVID uses a BERT-like bidirectional transformer. This architecture allows the model to consider both preceding and succeeding frames when filling in masked regions, significantly reducing the drift common in one-way prediction.
Key components of the conditioning process include:
- Multimodal Inputs: The transformer accepts tokens from diverse sources, including text (processed via a pretrained RoBERTa model), segmentation masks, and partially occluded images.
- The VID Token: To solve temporal jitter, the authors introduced a "Video Consistency Estimation" task. The model is trained to distinguish between coherent video sequences and "broken" ones (e.g., shuffled frames) using a specialized
VIDtoken. This self-supervised objective forces the model to learn the underlying physics of motion. - Inference via Mask-Predict: During generation, the model starts with a fully masked sequence and iteratively unmasks tokens. This process is refined using noise annealing to prevent local minima and beam search to optimize the final sequence.
Experimental Setup and Datasets
The framework was evaluated on four distinct benchmarks to test its versatility:
- Shapes: A synthetic dataset of geometric objects moving in specified paths (e.g., "a green square moving northeast").
- MUG: A facial expression dataset featuring 52 different actors.
- iPER: A dataset focused on human body motion and fashion synthesis.
- Multimodal VoxCeleb: A new benchmark introduced by the authors, consisting of 19,522 talking-head videos manually labeled with 36 facial attributes. This dataset provides a complex environment for testing nuanced text-to-video synthesis.
Results and Discussion
MMVID achieved state-of-the-art (SOTA) results across all evaluated metrics:
- Instruction Accuracy: On the Shapes dataset, MMVID achieved 98.12% average accuracy in following text instructions, outperforming the previous SOTA, TFGAN (92.88%).
- Visual Fidelity: On the MUG dataset, the model reached an Inception Score (IS) of 5.94, the highest reported for this benchmark.
- Human Preference: In head-to-head evaluations on the VoxCeleb dataset, human raters preferred MMVID’s outputs over the ART-V baseline 61.2% of the time, citing superior diversity and fewer temporal artifacts.
- Long-Sequence Stability: By utilizing "Preservation Control," the model successfully extrapolated videos beyond its training window, maintaining sharp details where autoregressive models typically fail.
Conclusion and Future Work
MMVID demonstrates that bidirectional transformers, when combined with multimodal conditioning and consistency-focused training, can significantly advance the state of video synthesis. While the model currently faces challenges scaling to 256x256 resolutions due to sequence length, it provides a robust foundation for future research. The authors suggest that incorporating additional modalities, such as synchronized audio, could further enhance the realism of synthesized talking heads and environmental scenes.
Summary updated