SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding

Tanzila Rahman, Renjie Liao, Leonid SigalarXiv 2026 · arXiv

TL;DR

SynMulti introduces a unified synthetic data generation pipeline that transforms a single static image into a fully annotated, multi-modal video training sample containing text, consistent video sequences, segmentation masks, and audio. By fine-tuning multimodal large language models (MLLMs) on this synthetic data using a structured Visual Question Answering (VQA) strategy, the framework achieves strong generalization and often outperforms models trained on traditional real-world datasets.

What Problem Does This Paper Address?

Training MLLMs for complex video understanding tasks—like object counting, video object segmentation, and visual reasoning—requires massive amounts of annotated data. However, collecting and labeling real-world video data is a notorious bottleneck: it is slow, prohibitively expensive, and requires dense frame-by-frame spatial-temporal tracking. Furthermore, real-world datasets often suffer from human biases and limited diversity in reasoning patterns.

While synthetic data generation has emerged as a promising alternative, existing methods typically generate different modalities (like text, video, or audio) independently. This fragmentation leads to weak cross-modal alignment and data inconsistency. SynMulti tackles this by creating a unified, image-anchored generation chain that ensures all modalities remain tightly coupled and semantically consistent.

Fig. 1: Comparison of real and synthetic data for training video foundation models. Real data (left) requires manual annotation, making it costly and labor- intensive, while synthetic data (right) is automatically generated, offering a scalable and efficient alternative for creating high-quality training datasets.

Methods

The core innovation of SynMulti is its sequential conditional generation pipeline. Rather than synthesizing modalities in isolation, the pipeline models the joint distribution by anchoring everything to a primary conditioning signal: a single annotated input image.

The generation flow operates as follows:

  1. LLM-Guided Semantic Annotation: Using a static seed image (and its base labels), ChatGPT generates a "future-aware" caption that describes a plausible temporal progression of the scene, alongside structured object counts and complex VQA pairs.
  2. Text- and Image-Conditioned Video Generation: A large-scale video generative model (Wan2.2, 14B variant) takes the initial image and the future-aware caption to synthesize a temporally coherent video sequence.
  3. Mask Generation for Video Segmentation: The Segment Anything Model 2 (SAM2) generates precise masks on the initial image, which are then propagated across the generated video frames using MUG-VOS to maintain temporal consistency.
  4. Audio Generation (Optional): A video-to-audio model (VTA-LDM) synthesizes a synchronized auditory track based on the visual motion and text context.

Additionally, the authors introduce a VQA-based fine-tuning strategy. Instead of training MLLMs on passive video captions or simple instructions, they condition the model on targeted visual questions. This forces the model to actively attend to specific visual evidence, attributes, and temporal relationships, leading to deeper visual grounding.

Fig. 2: Overview of the proposed multimodal synthetic data generation pipeline. The pipeline takes a static Input Image from which an LLM generates a plausible future caption to establish temporal intent. The caption and image are used to condition a video generation model to produce a coherent visual sequence. Subse- quently, an audio generation model, conditioned on both the text and the generated video, synthesizes a synchronized auditory track (optional). The final output is a high- fidelity generated video w/wo audio suitable for training MLLMs on complex temporal and cross-modal reasoning tasks.

Datasets

The synthetic pipeline used images from the MSCOCO dataset as seeds to generate a new synthetic corpus consisting of approximately 5K training videos and 1K validation videos.

To test real-world generalization, the models fine-tuned on SynMulti were evaluated across three distinct benchmarks:

  • MSCOCO validation set (for counting and segmentation)
  • LV-VIS (Large Vocabulary Video Instance Segmentation)
  • YouTube-VIS (YouTube Video Instance Segmentation)

Key Insights and Ideas

  • Sequential Conditioning Beats Independent Generation: By making each step of the pipeline conditional on the previous output (Image $\rightarrow$ Text $\rightarrow$ Video $\rightarrow$ Masks/Audio), the framework drastically reduces common generative artifacts like identity drift and unrealistic object transformations.
  • VQA is Superior to Captioning for Fine-Tuning: Training models to answer structured questions about visual content enforces a much tighter alignment between visual perception and language than standard caption matching. It prevents the model from relying on surface-level textual descriptions.
  • Imperfect Data is Still Highly Valuable: Even though synthetic videos may contain minor physical or visual artifacts, their dense, error-free automated annotations and vast diversity allow models to learn generalizable reasoning principles rather than overfitting to dataset-specific biases.

Key Results

The authors evaluated their approach on three downstream tasks, demonstrating that purely synthetic fine-tuning yields substantial gains over baseline models:

  • Video Object Counting: On the MSCOCO validation set, fine-tuning the InternVideo2.5 baseline with SynMulti VQA data reduced the Mean Absolute Error (MAE) from 5.46 to 3.18 and Mean Squared Error (MSE) from 75.38 to 27.21. Similar improvements were observed on the LV-VIS dataset.
  • Video Question Answering: Incorporating synthetic VQA data caused the Word Understanding Precision (WUP) score on MSCOCO to skyrocket from 0.18 to 0.81, showing a massive leap in answer correctness.
  • Video Object Segmentation: Fine-tuning SAM on the synthetic video masks improved the overall mean Intersection over Union (mIoU) on MSCOCO by +5.28 points (moving from 0.4711 to 0.5239).
  • Data Scalability: Increasing the synthetic dataset size from 2K videos to 5K videos consistently boosted the mIoU from 0.4694 to 0.5239, proving that the pipeline scales effectively.

Limitations and Future Work

Despite the strong transfer performance, the authors note that the synthetic videos are not entirely flawless. They occasionally exhibit minor visual artifacts, unrealistic object physics, or mask propagation errors over long sequences.

For future work, the team plans to expand their experimental validation to comprehensive audio-visual tasks by fully leveraging the pipeline's synthetic audio generation capabilities, moving towards holistic, multi-sensory video understanding.

Summary updated

Abstract

Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, and segmentation. However, collecting and annotating multimodal video data in real-world is costly, slow, and inherently limited in diversity and coverage. To address this challenge, we propose~\textbf{SynMulti} dataset, along with a unified synthetic data generation pipeline capable of automatically producing unlimited multimodal video data with rich and diverse supervision. Our framework supports multiple task formats within a single pipeline, enabling scalable and consistent data creation across tasks. To further enhance reasoning ability, we introduce a VQA-based fine-tuning strategy that trains models to answer structured questions about visual content rather than relying solely on captions or simple instructions. This formulation encourages deeper visual grounding and reasoning. We evaluate our approach in three challenging tasks: video object counting, video-based visual question answering, and video object segmentation. Experimental results demonstrate that models trained predominantly on synthetic data generalize effectively to real-world datasets, often outperforming traditionally trained counterparts. Our findings highlight the potential of unified synthetic data pipelines as a scalable alternative to expensive real-world annotation for multimodal video understanding.
BibTeX
@misc{rahman2026synmulti,
  author = {Tanzila Rahman and Renjie Liao and Leonid Sigal},
  title = {{SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding}},
  year = {2026},
  eprint = {2604.12335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.12335},
}

Loading related content...

Figure