WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

Changli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Jinchang Xu, Fengyun Rao, Chao Zhang, Jing LyuarXiv 2026 · arXiv

TL;DR

WeMM-Video is a new family of open-source omni-modal foundation models (ranging from 2B to 35B parameters) that unify text, image, audio, and video understanding. By introducing a "hybrid-thinking" paradigm that supports explicit step-by-step reasoning, it sets a new state-of-the-art for open-source models, even outperforming several prominent proprietary systems in complex audio-visual tasks.

What Problem Does This Paper Address?

While multimodal AI has advanced rapidly, true "omni-modal" video understanding—the ability to simultaneously comprehend continuous visual scenes and acoustic signals—remains a major hurdle. Most existing open-source models are "brittle": they often fail to align audio cues with visual events, overlook fine-grained temporal dynamics, and suffer from cross-modal hallucinations (e.g., describing a sound that isn't supported by the video evidence).

The authors of WeMM-Video aim to move beyond simple object recognition toward a model that can capture the nuanced interplay between what is seen and what is heard over long durations.

Methods

The architecture of WeMM-Video (illustrated in Figure 2) builds upon the video-SALMONN 2 framework. It employs native visual encoders for images and video frames, alongside a dedicated Whisper-based audio encoder. A key component is the window-level Q-Former, which compresses acoustic features into the LLM’s input space at a rate of 10 audio tokens per second.

The model architecture unifies vision and audio encoders into a single Large Language Model backbone.

The most significant methodological contribution is the hybrid-thinking paradigm. The model supports two modes:

  1. Instruct Mode: For fast, direct responses.
  2. Thinking Mode: For explicit, grounded chain-of-thought reasoning. This mode allows the model to "deliberate" over visual and acoustic evidence before providing a final answer, which significantly reduces hallucinations.

The training process follows a rigorous three-stage pre-training curriculum (Acoustic Alignment, Omni-Modal Knowledge Injection, and Long-Context Scaling) followed by a sophisticated post-training pipeline involving multi-teacher distillation and reinforcement learning (RL).

Datasets

The models were trained on a massive multimodal corpus of 1.2 trillion tokens. As shown in Figure 3, the data is heavily weighted toward joint audio-visual content (45.4%) to ensure native omni-modal capabilities.

The training data is dominated by joint audio-visual and visual modalities, with captioning being the primary task.

For evaluation, the authors utilized over 120 benchmarks covering:

  • Omni-modal Video: Video-MME (with audio), Daily-Omni, WorldSense.
  • Silent Video: MVBench, EgoSchema, LongVideoBench.
  • Audio: MMAU, LibriSpeech, VoxPopuli.
  • Image/Text: MMMU, MathVista, MMLU-Pro.

Key Insights and Ideas

  • Thinking Mitigates Hallucinations: The authors found that forcing the model to generate a reasoning trace effectively grounds its perception. By "thinking" through the temporal alignment of audio and video, the model is less likely to invent details.
  • Native Omni-modal Training vs. Modality Bolting: Unlike models that add audio as an afterthought, WeMM-Video’s training distribution (Figure 3) treats audio-visual data as the primary signal. This leads to better cross-modal reasoning rather than just standalone modality performance.
  • Efficiency of Scale: One of the most surprising findings was that the WeMM-Video-9B model consistently outperformed the much larger Qwen3-Omni-30B on several benchmarks, suggesting that their multi-stage post-training pipeline (specifically the "Specialist RL") is highly effective at squeezing performance out of smaller parameter counts.

Key Results

The flagship WeMM-Video-35B-A3B established a new open-source state-of-the-art (SOTA).

  • Omni-modal Performance: On the Video-MME benchmark (with audio), the 35B model achieved a score of 86.4 in thinking mode, significantly outperforming Qwen3.5-Omni-Flash (79.3) and Seed2.0-Mini-0428 (83.6).
  • Competitive with Proprietary Models: While it still trails slightly behind Gemini-3.0-Flash, it has narrowed the gap considerably, particularly in specialized tasks like audio-visual interaction and hallucination checks (see Table 1).
  • Versatility: Despite the focus on video, the models remain highly competitive in pure text (MMLU-Pro) and pure audio (ASR) tasks, proving that omni-modal training does not necessarily degrade unimodal performance.
Quantitative comparison showing WeMM-Video-35B-A3B leading across various audio-visual benchmarks.

Limitations and Future Work

The authors acknowledge that while "Thinking Mode" improves accuracy, it increases latency, making it less ideal for real-time applications without further optimization. Additionally, while the model narrows the gap with top-tier proprietary models like Gemini-3.0-Flash, it hasn't yet surpassed them across all long-context reasoning metrics. Future work will likely focus on further scaling the context window and improving the efficiency of the reasoning traces.

Summary updated

Abstract

Video understanding, particularly with accompanying audio, remains a central challenge in multimodal intelligence. In this report, we introduce \textbf{WeMM-Video}, a family of Chinese-English bilingual omni-modal foundation models spanning 2B, 4B, 9B, and 35B-A3B variants. Centering on joint audio-visual perception and reasoning, WeMM-Video unifies text, image, audio, and video processing within a hybrid-thinking paradigm, seamlessly supporting both a non-thinking instruct mode for fast, direct responses and a thinking mode for explicit step-by-step reasoning. The models are trained on a large-scale multimodal corpus of approximately 1.2 trillion tokens through progressive pre-training and a multi-stage post-training pipeline incorporating multi-teacher OPD, anti-repetition MrDPO, and specialist RL. Evaluated across more than 120 benchmarks, WeMM-Video demonstrates superior performance in omni-modal video understanding and reasoning, while maintaining highly competitive capabilities across general audio, silent video, image, and text tasks. Specifically, in omni-modal video evaluations, our flagship WeMM-Video-35B-A3B establishes a new open-source state of the art, substantially outperforming proprietary models such as Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 while narrowing the gap with the strong Gemini-3.0-Flash, and our compact WeMM-Video-9B surpasses the larger open-source Qwen3-Omni-30B-A3B-Instruct. We publicly release the WeMM-Video family along with our evaluation and reproduction pipeline to foster further research and development in the open-source community.
BibTeX
@misc{tang2026wemm,
  author = {Changli Tang and Jie Yang and Tianyi Wang and Lei Li and Daoze Zhang and Zijie Xin and Zhan Liu and Li Li and Jinchang Xu and Fengyun Rao and Chao Zhang and Jing Lyu},
  title = {{WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding}},
  year = {2026},
  eprint = {2611.26409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26409},
}
Figure