Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning

Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris N. Metaxas, Sergey Tulyakov

Abstract

Controlling video synthesis remains a significant challenge due to the inherent complexity of temporal dynamics and the limitations of unimodal conditioning. We introduce MMVID, a multimodal framework designed to bridge the flexibility gap in video generation. By allowing users to specify content ("what") through images or masks and motion ("how") via natural language, MMVID provides a highly controllable synthesis pipeline. Our architecture utilizes a bidirectional transformer coupled with a novel Video Consistency Estimation task to ensure temporal stability. Experimental results across four benchmarks, including a newly introduced Multimodal VoxCeleb dataset, demonstrate that MMVID achieves state-of-the-art performance in visual quality, instruction following, and long-term consistency.

Introduction

Traditional video generation models are often restricted by a "unimodal" bottleneck, typically relying on either a text prompt or a single starting image. This creates a significant usability gap: while images are excellent for defining specific characters or backgrounds, they struggle to convey complex trajectories. Conversely, text prompts can describe motion but often lack the precision required to define fine-grained visual details. Furthermore, many autoregressive models suffer from "drift" or "exposure bias," where errors accumulate over time, leading to blurry or inconsistent frames.

MMVID addresses these issues by treating video synthesis as a multimodal completion task. By leveraging a bidirectional transformer, the model can "plan" an entire sequence simultaneously, ensuring that the generated motion is physically plausible and visually consistent from start to finish.

Methodology

MMVID employs a two-stage framework to transform video synthesis into a discrete sequence prediction problem.

Stage 1: Discrete Representation

The model utilizes a VQGAN (Vector Quantized Generative Adversarial Network) to compress high-dimensional video frames into a grid of discrete visual tokens. This compression allows the model to operate in a lower-dimensional latent space while preserving the essential textures and structures of the original video.

Stage 2: Bidirectional Transformer and Multimodal Conditioning

Unlike standard autoregressive models that predict tokens sequentially, MMVID uses a BERT-like bidirectional transformer. This architecture allows the model to consider both preceding and succeeding frames when filling in masked regions, significantly reducing the drift common in one-way prediction.

Key components of the conditioning process include:

  • Multimodal Inputs: The transformer accepts tokens from diverse sources, including text (processed via a pretrained RoBERTa model), segmentation masks, and partially occluded images.
  • The VID Token: To solve temporal jitter, the authors introduced a "Video Consistency Estimation" task. The model is trained to distinguish between coherent video sequences and "broken" ones (e.g., shuffled frames) using a specialized VID token. This self-supervised objective forces the model to learn the underlying physics of motion.
  • Inference via Mask-Predict: During generation, the model starts with a fully masked sequence and iteratively unmasks tokens. This process is refined using noise annealing to prevent local minima and beam search to optimize the final sequence.

Experimental Setup and Datasets

The framework was evaluated on four distinct benchmarks to test its versatility:

  1. Shapes: A synthetic dataset of geometric objects moving in specified paths (e.g., "a green square moving northeast").
  2. MUG: A facial expression dataset featuring 52 different actors.
  3. iPER: A dataset focused on human body motion and fashion synthesis.
  4. Multimodal VoxCeleb: A new benchmark introduced by the authors, consisting of 19,522 talking-head videos manually labeled with 36 facial attributes. This dataset provides a complex environment for testing nuanced text-to-video synthesis.

Results and Discussion

MMVID achieved state-of-the-art (SOTA) results across all evaluated metrics:

  • Instruction Accuracy: On the Shapes dataset, MMVID achieved 98.12% average accuracy in following text instructions, outperforming the previous SOTA, TFGAN (92.88%).
  • Visual Fidelity: On the MUG dataset, the model reached an Inception Score (IS) of 5.94, the highest reported for this benchmark.
  • Human Preference: In head-to-head evaluations on the VoxCeleb dataset, human raters preferred MMVID’s outputs over the ART-V baseline 61.2% of the time, citing superior diversity and fewer temporal artifacts.
  • Long-Sequence Stability: By utilizing "Preservation Control," the model successfully extrapolated videos beyond its training window, maintaining sharp details where autoregressive models typically fail.

Conclusion and Future Work

MMVID demonstrates that bidirectional transformers, when combined with multimodal conditioning and consistency-focused training, can significantly advance the state of video synthesis. While the model currently faces challenges scaling to 256x256 resolutions due to sequence length, it provides a robust foundation for future research. The authors suggest that incorporating additional modalities, such as synchronized audio, could further enhance the realism of synthesized talking heads and environmental scenes.

Summary updated

Abstract

Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized representations for videos and apply a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. To improve video quality and consistency, we propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. We introduce text augmentation to improve the robustness of the textual representation and diversity of generated videos. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images. It can generate much longer sequences than the one used for training. In addition, our model can extract visual information as suggested by the text prompt, e.g., "an object in image one is moving northeast", and generate corresponding videos. We run evaluations on three public datasets and a newly collected dataset labeled with facial attributes, achieving state-of-the-art generation results on all four.
BibTeX
@inproceedings{han2022show,
  author = {Ligong Han and Jian Ren and Hsin-Ying Lee and Francesco Barbieri and Kyle Olszewski and Shervin Minaee and Dimitris N. Metaxas and Sergey Tulyakov},
  title = {{Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure