TL;DR
WeMM-Video is a new family of open-source omni-modal foundation models (ranging from 2B to 35B parameters) that unify text, image, audio, and video understanding. By introducing a "hybrid-thinking" paradigm that supports explicit step-by-step reasoning, it sets a new state-of-the-art for open-source models, even outperforming several prominent proprietary systems in complex audio-visual tasks.
What Problem Does This Paper Address?
While multimodal AI has advanced rapidly, true "omni-modal" video understanding—the ability to simultaneously comprehend continuous visual scenes and acoustic signals—remains a major hurdle. Most existing open-source models are "brittle": they often fail to align audio cues with visual events, overlook fine-grained temporal dynamics, and suffer from cross-modal hallucinations (e.g., describing a sound that isn't supported by the video evidence).
The authors of WeMM-Video aim to move beyond simple object recognition toward a model that can capture the nuanced interplay between what is seen and what is heard over long durations.
Methods
The architecture of WeMM-Video (illustrated in Figure 2) builds upon the video-SALMONN 2 framework. It employs native visual encoders for images and video frames, alongside a dedicated Whisper-based audio encoder. A key component is the window-level Q-Former, which compresses acoustic features into the LLM’s input space at a rate of 10 audio tokens per second.
The most significant methodological contribution is the hybrid-thinking paradigm. The model supports two modes:
- Instruct Mode: For fast, direct responses.
- Thinking Mode: For explicit, grounded chain-of-thought reasoning. This mode allows the model to "deliberate" over visual and acoustic evidence before providing a final answer, which significantly reduces hallucinations.
The training process follows a rigorous three-stage pre-training curriculum (Acoustic Alignment, Omni-Modal Knowledge Injection, and Long-Context Scaling) followed by a sophisticated post-training pipeline involving multi-teacher distillation and reinforcement learning (RL).
Datasets
The models were trained on a massive multimodal corpus of 1.2 trillion tokens. As shown in Figure 3, the data is heavily weighted toward joint audio-visual content (45.4%) to ensure native omni-modal capabilities.
For evaluation, the authors utilized over 120 benchmarks covering:
- Omni-modal Video: Video-MME (with audio), Daily-Omni, WorldSense.
- Silent Video: MVBench, EgoSchema, LongVideoBench.
- Audio: MMAU, LibriSpeech, VoxPopuli.
- Image/Text: MMMU, MathVista, MMLU-Pro.
Key Insights and Ideas
- Thinking Mitigates Hallucinations: The authors found that forcing the model to generate a reasoning trace effectively grounds its perception. By "thinking" through the temporal alignment of audio and video, the model is less likely to invent details.
- Native Omni-modal Training vs. Modality Bolting: Unlike models that add audio as an afterthought, WeMM-Video’s training distribution (Figure 3) treats audio-visual data as the primary signal. This leads to better cross-modal reasoning rather than just standalone modality performance.
- Efficiency of Scale: One of the most surprising findings was that the WeMM-Video-9B model consistently outperformed the much larger Qwen3-Omni-30B on several benchmarks, suggesting that their multi-stage post-training pipeline (specifically the "Specialist RL") is highly effective at squeezing performance out of smaller parameter counts.
Key Results
The flagship WeMM-Video-35B-A3B established a new open-source state-of-the-art (SOTA).
- Omni-modal Performance: On the Video-MME benchmark (with audio), the 35B model achieved a score of 86.4 in thinking mode, significantly outperforming Qwen3.5-Omni-Flash (79.3) and Seed2.0-Mini-0428 (83.6).
- Competitive with Proprietary Models: While it still trails slightly behind Gemini-3.0-Flash, it has narrowed the gap considerably, particularly in specialized tasks like audio-visual interaction and hallucination checks (see Table 1).
- Versatility: Despite the focus on video, the models remain highly competitive in pure text (MMLU-Pro) and pure audio (ASR) tasks, proving that omni-modal training does not necessarily degrade unimodal performance.
Limitations and Future Work
The authors acknowledge that while "Thinking Mode" improves accuracy, it increases latency, making it less ideal for real-time applications without further optimization. Additionally, while the model narrows the gap with top-tier proprietary models like Gemini-3.0-Flash, it hasn't yet surpassed them across all long-context reasoning metrics. Future work will likely focus on further scaling the context window and improving the efficiency of the reasoning traces.
Summary updated