Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

Inclusion Ai, Bowen Ma, Cheng Zou, ChengKun Du, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Chengxiang Fan, Dandan Zheng, Fudong Wang, Furong Xu, Guangming Yao, Haohao Liu, Han Peng, Jun Zhou, Junluan Xia, Jingdong Chen, Jianing Li, Jianxin Sun, Jianjiang Zhu, Jianping Jiang, Jinpeng Ou, Jun Peng, Jin Peng, Kaixiang Ji, Li Tang, Libin Wang, Lixiang Ru, Longhua Tan, Lu Ma, Lan Wang, Mochen Bai, Minghong Cai, Mingxue Yang, Ning Gao, Qingpei Guo, Qinglong Zhang, Qiang Xu, Qin Zhao, Rui Liu, Ruijie Xiong, Ruobing Zheng, Sirui Gao, Shaoxiong Lin, Tao Zhang, Tianqi Li, Tinghao Liu, Tongli Wang, Taoye Huang, Weilong Chai, Xiaomei Wang, Xiaolong Wang, Xiaojian Liu, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Xuezhi Wang, Yi Yuan, Yuting Gao, Yuting Xiao, Yunxiao Sun, Yipeng Chen, Yifan Mao, Yifei Wu, Yongjie Lyu, Yingying Zhang, YuQian Li, Ziping Ma, Zhiqiang Fang, Zhihao Qiu, Ziyuan Huang, Zizheng Yang, Zhengyu He

TL;DR

Ming-Flash-Omni is a 100-billion-parameter sparse Mixture-of-Experts (MoE) model that unifies multimodal comprehension and generation across text, vision, and audio. By activating only 6.1 billion parameters per token and introducing continuous audio latents alongside generative semantic segmentation, it achieves performance on par with proprietary giants like Gemini 2.5 Pro while dramatically lowering computational overhead.

What Problem Does This Paper Address?

Building a truly unified "Omni" Multimodal Large Language Model (MLLM) that seamlessly handles both understanding (perception) and generation across text, vision, and audio remains a grand challenge in AI. Most current systems suffer from representational disparities and modality imbalances.

For instance, to maintain training stability, image-understanding pathways are often frozen during generation training. This creates a bottleneck: fine-grained visual knowledge (like spatial relationships and object attributes) cannot easily transfer to high-precision image generation or editing. Furthermore, processing audio via discrete tokens introduces quantization artifacts, and training on highly heterogeneous data types causes severe hardware inefficiencies and pipeline bottlenecks.

Methods

Ming-Flash-Omni tackles these challenges through a redesigned foundation and several architectural innovations:

  • Sparse MoE Backbone: The model scales up using Ling-Flash-2.0, a sparse MoE architecture with 100B total parameters, routing just 6.1B active parameters per token to balance capacity and inference latency.
  • Unified Understanding: It introduces time-interleaved VideoRoPE, which interleaves timestamps directly with video frames to better capture temporal dynamics. It also integrates context-aware Automatic Speech Recognition (ASR) and a "think mode" for multi-step reasoning.
  • Continuous Audio Generation: Instead of relying on discrete acoustic tokens that degrade quality, the model utilizes continuous acoustic latents via a low-frame-rate (12.5 Hz) VAE tokenizer. This enables the joint, single-channel generation of speech, sound effects, and music.
  • Generative Segmentation & Single-Stream Diffusion: For vision generation, the authors adopt a native single-stream diffusion transformer that maps all modalities into a unified token space. Crucially, they reframe image segmentation as a generative editing task (e.g., changing "segment the banana" to "color the banana purple"), forcing the model to tightly couple visual perception with generation.

The entire framework is illustrated below:

The overall framework of Ming-Flash-Omni, featuring a sparse MoE architecture and unified generation heads.

To support this complex architecture, the authors enhanced Megatron-LM with Sequence Packing to handle variable-length multimodal samples and Flexible Parallelization to optimize workload distribution across encoders and decoders.

Datasets

The model was trained on a massive, curated corpus spanning multiple modalities:

  • Image & Video: Includes a 1.6-million-sample long Chain-of-Thought (CoT) dataset for STEM reasoning, ChartQA-PoT, and 200k paired lifestyle-ID photos for portrait preservation. Long video data included dialogues up to 30 minutes (256k tokens).
  • Audio: 3 million Chinese and English Context ASR samples formatted as <audio, text, context, entity_list>.

New Benchmarks Introduced

  1. StreamingMultiturnBench: A benchmark consisting of 380 videos and 2,200 video QA pairs designed to evaluate the interactive, multi-turn video streaming experience across five dimensions (accuracy, completeness, relevance, naturalness, and proactivity).
  2. Multi-Dialect and Multi-Domain Audio Understanding Benchmark: Contains over 51k multi-dialect samples from 15 regions and over 10k multi-domain samples to test robust real-world speech recognition.

Key Insights and Ideas

  • Segmentation-as-Editing: Reframing abstract binary mask generation as a semantic-preserving image edit unifies the optimization objectives of understanding and generation. Visual comprehension becomes a strict prerequisite for high-quality editing.
  • Continuous beats Discrete for Audio: Moving away from discrete tokens to continuous representations circumvents quantization loss, yielding significantly more natural, expressive, and dialect-aware Text-to-Speech (TTS) outputs.
  • Single-Stream over Dual-Stream Transformers: Processing noisy latents and reference latents in a single stream with full cross-modal attention eliminates the "copy-paste" collage artifacts common in dual-stream architectures (like MMDiT), achieving global semantic coherence in lighting and layout.

Key Results

Ming-Flash-Omni demonstrates highly competitive performance across all evaluated domains:

Performance overview of Ming-Flash-Omni across vision, text, speech, and generation benchmarks.
  • Vision-Language Perceptual Tasks: Achieves overall scores on par with Gemini 2.5 Pro. It leads open models on MMMU (77.1), sets a new high on OCRBench (891), and dominates the in-house Wiki Knowledge benchmark (64.6 vs. $\le$ 48.7 for competitors).
  • Image Generation & Editing: Surpasses all previous unified models on the GenEval benchmark (0.94 AVG) and delivers state-of-the-art image editing performance on GEdit-Bench.
  • Audio & Dialect ASR: Achieves a Character Error Rate (CER) of 2.12 on the ZipVoice-Dia(zh) podcast task and hits an accuracy of 84.42% on the challenging Sichuanese dialect generation benchmark, outperforming CosyVoice3.
  • Video Streaming: Outperforms Qwen3-Omni and Gemini 2.5 Pro on the new StreamingMultiturnBench with an average score of 75.92.
  • Hardware Efficiency: Custom CUDA kernels for LayerNorm and RMSNorm raised memory bandwidth utilization to 80%, delivering a 28% Model FLOPs Utilization (MFU)—a 4x throughput improvement over the baseline Megatron-LM.

Qualitative examples of these capabilities are showcased below:

Qualitative results across world knowledge grounding, math reasoning, dialect ASR, and multi-turn dialogue.

Limitations and Future Work

Despite its strong performance, the authors note a few limitations:

  • Text Rendering Preprocessing: For high-fidelity text rendering via Glyph-byT5, the text must be explicitly extracted from the user's prompt in advance during inference, adding an extra pipeline step.
  • Minor Benchmark Lags: While dominant in most categories, Ming-Flash-Omni still slightly lags behind specialized models like GLM-4.6-V on specific document/chart tasks (ChartQA, DocVQA) and multi-image understanding (MuirBench).

Future work will likely focus on removing the text-rendering preprocessing bottleneck and further optimizing the unified training dynamics to completely close the gap with proprietary systems.

Summary updated

Abstract

We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. Notably, it achieves strong performance on vision-language understanding benchmarks, with overall scores on par with Gemini 2.5 Pro, and enables seamless switching among multimodal tasks in multi-turn interactions. In speech, it achieves strong performance in contextual and dialect-aware ASR while enabling joint, continuous-generation of speech, sound, and music. In vision, it introduces generative semantic segmentation that achieves competitive standalone performance and enhances spatial control and editing consistency, alongside marked improvements in identity preservation, and high-fidelity in-image text rendering. Together, these capabilities demonstrate that a single unified model can serve as a practical foundation for general-purpose multimodal intelligence.
BibTeX
@misc{ai2025ming,
  author = {Inclusion AI and Bowen Ma and Cheng Zou and ChengKun Du and Canxiang Yan and Chunxiang Jin and Chunjie Shen and Chenyu Lian and Chengxiang Fan and Dandan Zheng and Fudong Wang and Furong Xu and Guangming Yao and Haohao Liu and Han Peng and Jun Zhou and Junluan Xia and Jingdong Chen and Jianing Li and Jianxin Sun and Jianjiang Zhu and Jianping Jiang and Jinpeng Ou and Jun Peng and Jin Peng and Kaixiang Ji and Li Tang and Libin Wang and Lixiang Ru and Longhua Tan and Lu Ma and Lan Wang and Mochen Bai and Minghong Cai and Mingxue Yang and Ning Gao and Qingpei Guo and Qinglong Zhang and Qiang Xu and Qin Zhao and Rui Liu and Ruijie Xiong and Ruobing Zheng and Sirui Gao and Shaoxiong Lin and Tao Zhang and Tianqi Li and Tinghao Liu and Tongli Wang and Taoye Huang and Weilong Chai and Xiaomei Wang and Xiaolong Wang and Xiaojian Liu and Xiao Lu and Xiaoyu Li and Xingning Dong and Xuzheng Yu and Xuezhi Wang and Yi Yuan and Yuting Gao and Yuting Xiao and Yunxiao Sun and Yipeng Chen and Yifan Mao and Yifei Wu and Yongjie Lyu and Yingying Zhang and YuQian Li and Ziping Ma and Zhiqiang Fang and Zhihao Qiu and Ziyuan Huang and Zizheng Yang and Zhengyu He},
  title = {{Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation}},
  year = {2025},
  eprint = {2510.24821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.24821},
}
Figure