TL;DR
SynMulti introduces a unified synthetic data generation pipeline that transforms a single static image into a fully annotated, multi-modal video training sample containing text, consistent video sequences, segmentation masks, and audio. By fine-tuning multimodal large language models (MLLMs) on this synthetic data using a structured Visual Question Answering (VQA) strategy, the framework achieves strong generalization and often outperforms models trained on traditional real-world datasets.
What Problem Does This Paper Address?
Training MLLMs for complex video understanding tasks—like object counting, video object segmentation, and visual reasoning—requires massive amounts of annotated data. However, collecting and labeling real-world video data is a notorious bottleneck: it is slow, prohibitively expensive, and requires dense frame-by-frame spatial-temporal tracking. Furthermore, real-world datasets often suffer from human biases and limited diversity in reasoning patterns.
While synthetic data generation has emerged as a promising alternative, existing methods typically generate different modalities (like text, video, or audio) independently. This fragmentation leads to weak cross-modal alignment and data inconsistency. SynMulti tackles this by creating a unified, image-anchored generation chain that ensures all modalities remain tightly coupled and semantically consistent.
Methods
The core innovation of SynMulti is its sequential conditional generation pipeline. Rather than synthesizing modalities in isolation, the pipeline models the joint distribution by anchoring everything to a primary conditioning signal: a single annotated input image.
The generation flow operates as follows:
- LLM-Guided Semantic Annotation: Using a static seed image (and its base labels), ChatGPT generates a "future-aware" caption that describes a plausible temporal progression of the scene, alongside structured object counts and complex VQA pairs.
- Text- and Image-Conditioned Video Generation: A large-scale video generative model (Wan2.2, 14B variant) takes the initial image and the future-aware caption to synthesize a temporally coherent video sequence.
- Mask Generation for Video Segmentation: The Segment Anything Model 2 (SAM2) generates precise masks on the initial image, which are then propagated across the generated video frames using MUG-VOS to maintain temporal consistency.
- Audio Generation (Optional): A video-to-audio model (VTA-LDM) synthesizes a synchronized auditory track based on the visual motion and text context.
Additionally, the authors introduce a VQA-based fine-tuning strategy. Instead of training MLLMs on passive video captions or simple instructions, they condition the model on targeted visual questions. This forces the model to actively attend to specific visual evidence, attributes, and temporal relationships, leading to deeper visual grounding.
Datasets
The synthetic pipeline used images from the MSCOCO dataset as seeds to generate a new synthetic corpus consisting of approximately 5K training videos and 1K validation videos.
To test real-world generalization, the models fine-tuned on SynMulti were evaluated across three distinct benchmarks:
- MSCOCO validation set (for counting and segmentation)
- LV-VIS (Large Vocabulary Video Instance Segmentation)
- YouTube-VIS (YouTube Video Instance Segmentation)
Key Insights and Ideas
- Sequential Conditioning Beats Independent Generation: By making each step of the pipeline conditional on the previous output (Image $\rightarrow$ Text $\rightarrow$ Video $\rightarrow$ Masks/Audio), the framework drastically reduces common generative artifacts like identity drift and unrealistic object transformations.
- VQA is Superior to Captioning for Fine-Tuning: Training models to answer structured questions about visual content enforces a much tighter alignment between visual perception and language than standard caption matching. It prevents the model from relying on surface-level textual descriptions.
- Imperfect Data is Still Highly Valuable: Even though synthetic videos may contain minor physical or visual artifacts, their dense, error-free automated annotations and vast diversity allow models to learn generalizable reasoning principles rather than overfitting to dataset-specific biases.
Key Results
The authors evaluated their approach on three downstream tasks, demonstrating that purely synthetic fine-tuning yields substantial gains over baseline models:
- Video Object Counting: On the MSCOCO validation set, fine-tuning the InternVideo2.5 baseline with SynMulti VQA data reduced the Mean Absolute Error (MAE) from 5.46 to 3.18 and Mean Squared Error (MSE) from 75.38 to 27.21. Similar improvements were observed on the LV-VIS dataset.
- Video Question Answering: Incorporating synthetic VQA data caused the Word Understanding Precision (WUP) score on MSCOCO to skyrocket from 0.18 to 0.81, showing a massive leap in answer correctness.
- Video Object Segmentation: Fine-tuning SAM on the synthetic video masks improved the overall mean Intersection over Union (mIoU) on MSCOCO by +5.28 points (moving from 0.4711 to 0.5239).
- Data Scalability: Increasing the synthetic dataset size from 2K videos to 5K videos consistently boosted the mIoU from 0.4694 to 0.5239, proving that the pipeline scales effectively.
Limitations and Future Work
Despite the strong transfer performance, the authors note that the synthetic videos are not entirely flawless. They occasionally exhibit minor visual artifacts, unrealistic object physics, or mask propagation errors over long sequences.
For future work, the team plans to expand their experimental validation to comprehensive audio-visual tasks by fully leveraging the pipeline's synthetic audio generation capabilities, moving towards holistic, multi-sensory video understanding.
Summary updated