TL;DR
DaQ-MSA addresses the chronic data scarcity in Multimodal Sentiment Analysis (MSA) by using diffusion models to generate synthetic video and audio augmentations. To prevent "hallucinated" or low-quality synthetic data from ruining model training, it introduces an automated Quality-Aware Scorer that assigns adaptive weights to samples, ensuring the model learns from high-fidelity data while ignoring generative noise.
What Problem Does This Paper Address?
Multimodal Sentiment Analysis (MSA)—the task of inferring human emotion from text, video, and audio—is notoriously difficult because high-quality, human-annotated data is scarce. While Multimodal Large Language Models (MLLMs) have the "brainpower" to solve this, they are often hamstrung by small, noisy datasets like MOSI or MOSEI, which suffer from temporal misalignment and subjective labels.
Simply scaling up models doesn't help if the data is bad. Traditional data augmentation (like cropping or adding static noise) only goes so far. Generative AI (diffusion models) offers a way to create diverse new samples, but there’s a catch: diffusion models often produce "artifacts"—think weird facial glitches or audio that doesn't match the speaker's mouth. If you train a model on this "hallucinated" data, performance actually drops.
Methods
The authors propose a three-stage framework called DaQ-MSA (Denoising and Qualifying Diffusion Augmentations):
- Semantics-Preserving Augmentation: They use FateZero for video (applying styles like "watercolor" while keeping the motion/expression intact) and Seed-VC for audio (changing the speaker's timbre while keeping the emotional prosody). Crucially, they leave the text unchanged to ensure the core sentiment remains stable.
- Quality-Aware Scoring (The "Denoising" Part): This is the core innovation. They train a decoupled module to distinguish between "good" and "bad" augmented samples. To do this without human labels, they create "negative samples" by intentionally breaking data (e.g., swapping audio from a different clip or masking features). The scorer learns to output a value between 0 and 1, representing the reliability of the sample.
- Weighted Fine-tuning: During the final training of the MLLM (using the HumanOmni backbone), the quality scores act as sample-wise weights. High-quality synthetic samples get a "vote" similar to real data, while glitchy, low-quality samples are effectively ignored by the loss function.
Datasets
The researchers evaluated their framework on three major benchmarks:
- CH-SIMS: A Chinese dataset with fine-grained (5-point) sentiment labels.
- CMU-MOSI: The gold standard English dataset for multimodal sentiment.
- MUStARD: A specialized dataset for sarcasm detection (a high-bar test for multimodal alignment).
- New Contribution: The authors are releasing their diffusion-augmented versions of these datasets to the community.
Key Insights and Ideas
- The "Generate-then-Filter" Paradigm: The paper proves that more data is only better if you have a gatekeeper. By using an automated scorer, they achieve the benefits of a "human-in-the-loop" workflow without the massive cost of manual data cleaning.
- Cross-Modal Consistency is Key: The QA module doesn't just look at image quality; it looks for alignment. If the video looks like a "watercolor" but the sentiment expression is lost, the scorer catches it.
- Data Efficiency: One of the most striking insights is that the model trained on just 10% of labeled data plus the DaQ-MSA augmentations outperformed a baseline trained on 100% of the original data. This suggests that high-quality synthetic diversity is more valuable than raw volume of "in-the-wild" data.
Key Results
DaQ-MSA achieved state-of-the-art (SOTA) results across the board:
- CH-SIMS: Achieved 90.15% binary accuracy, outperforming the previous best (HumanOmni) by over 5 points. Even more impressive was the 18% relative improvement in 5-class fine-grained classification.
- CMU-MOSI: Reached 92.37% binary accuracy, setting a new benchmark for the field.
- MUStARD: Improved sarcasm detection by +3.59 points, proving the model's ability to pick up on subtle, incongruent cues between modalities.
- Qualitative Success: t-SNE visualizations show that while naive data mixing creates "chaotic" clusters, DaQ-MSA produces well-separated, clean sentiment representations.
Limitations and Future Work
Despite the strong results, the authors note a few hurdles:
- Computational Overhead: Running diffusion models (FateZero/Seed-VC) is expensive and slow. This is currently an offline process and cannot be done in real-time.
- The "Lip-Sync" Problem: Diffusion models still struggle with perfect lip synchronization during extreme facial movements, which can occasionally confuse the sentiment model if the QA scorer isn't aggressive enough.
- Linguistic Breadth: While they tested on English and Chinese, the nuances of sentiment in low-resource languages or different cultural contexts remain an open question for future validation.
Summary updated