TL;DR
Ming-Flash-Omni is a 100-billion-parameter sparse Mixture-of-Experts (MoE) model that unifies multimodal comprehension and generation across text, vision, and audio. By activating only 6.1 billion parameters per token and introducing continuous audio latents alongside generative semantic segmentation, it achieves performance on par with proprietary giants like Gemini 2.5 Pro while dramatically lowering computational overhead.
What Problem Does This Paper Address?
Building a truly unified "Omni" Multimodal Large Language Model (MLLM) that seamlessly handles both understanding (perception) and generation across text, vision, and audio remains a grand challenge in AI. Most current systems suffer from representational disparities and modality imbalances.
For instance, to maintain training stability, image-understanding pathways are often frozen during generation training. This creates a bottleneck: fine-grained visual knowledge (like spatial relationships and object attributes) cannot easily transfer to high-precision image generation or editing. Furthermore, processing audio via discrete tokens introduces quantization artifacts, and training on highly heterogeneous data types causes severe hardware inefficiencies and pipeline bottlenecks.
Methods
Ming-Flash-Omni tackles these challenges through a redesigned foundation and several architectural innovations:
- Sparse MoE Backbone: The model scales up using Ling-Flash-2.0, a sparse MoE architecture with 100B total parameters, routing just 6.1B active parameters per token to balance capacity and inference latency.
- Unified Understanding: It introduces time-interleaved VideoRoPE, which interleaves timestamps directly with video frames to better capture temporal dynamics. It also integrates context-aware Automatic Speech Recognition (ASR) and a "think mode" for multi-step reasoning.
- Continuous Audio Generation: Instead of relying on discrete acoustic tokens that degrade quality, the model utilizes continuous acoustic latents via a low-frame-rate (12.5 Hz) VAE tokenizer. This enables the joint, single-channel generation of speech, sound effects, and music.
- Generative Segmentation & Single-Stream Diffusion: For vision generation, the authors adopt a native single-stream diffusion transformer that maps all modalities into a unified token space. Crucially, they reframe image segmentation as a generative editing task (e.g., changing "segment the banana" to "color the banana purple"), forcing the model to tightly couple visual perception with generation.
The entire framework is illustrated below:
To support this complex architecture, the authors enhanced Megatron-LM with Sequence Packing to handle variable-length multimodal samples and Flexible Parallelization to optimize workload distribution across encoders and decoders.
Datasets
The model was trained on a massive, curated corpus spanning multiple modalities:
- Image & Video: Includes a 1.6-million-sample long Chain-of-Thought (CoT) dataset for STEM reasoning, ChartQA-PoT, and 200k paired lifestyle-ID photos for portrait preservation. Long video data included dialogues up to 30 minutes (256k tokens).
- Audio: 3 million Chinese and English Context ASR samples formatted as
<audio, text, context, entity_list>.
New Benchmarks Introduced
- StreamingMultiturnBench: A benchmark consisting of 380 videos and 2,200 video QA pairs designed to evaluate the interactive, multi-turn video streaming experience across five dimensions (accuracy, completeness, relevance, naturalness, and proactivity).
- Multi-Dialect and Multi-Domain Audio Understanding Benchmark: Contains over 51k multi-dialect samples from 15 regions and over 10k multi-domain samples to test robust real-world speech recognition.
Key Insights and Ideas
- Segmentation-as-Editing: Reframing abstract binary mask generation as a semantic-preserving image edit unifies the optimization objectives of understanding and generation. Visual comprehension becomes a strict prerequisite for high-quality editing.
- Continuous beats Discrete for Audio: Moving away from discrete tokens to continuous representations circumvents quantization loss, yielding significantly more natural, expressive, and dialect-aware Text-to-Speech (TTS) outputs.
- Single-Stream over Dual-Stream Transformers: Processing noisy latents and reference latents in a single stream with full cross-modal attention eliminates the "copy-paste" collage artifacts common in dual-stream architectures (like MMDiT), achieving global semantic coherence in lighting and layout.
Key Results
Ming-Flash-Omni demonstrates highly competitive performance across all evaluated domains:
- Vision-Language Perceptual Tasks: Achieves overall scores on par with Gemini 2.5 Pro. It leads open models on MMMU (77.1), sets a new high on OCRBench (891), and dominates the in-house Wiki Knowledge benchmark (64.6 vs. $\le$ 48.7 for competitors).
- Image Generation & Editing: Surpasses all previous unified models on the GenEval benchmark (0.94 AVG) and delivers state-of-the-art image editing performance on GEdit-Bench.
- Audio & Dialect ASR: Achieves a Character Error Rate (CER) of 2.12 on the ZipVoice-Dia(zh) podcast task and hits an accuracy of 84.42% on the challenging Sichuanese dialect generation benchmark, outperforming CosyVoice3.
- Video Streaming: Outperforms Qwen3-Omni and Gemini 2.5 Pro on the new
StreamingMultiturnBenchwith an average score of 75.92. - Hardware Efficiency: Custom CUDA kernels for LayerNorm and RMSNorm raised memory bandwidth utilization to 80%, delivering a 28% Model FLOPs Utilization (MFU)—a 4x throughput improvement over the baseline Megatron-LM.
Qualitative examples of these capabilities are showcased below:
Limitations and Future Work
Despite its strong performance, the authors note a few limitations:
- Text Rendering Preprocessing: For high-fidelity text rendering via Glyph-byT5, the text must be explicitly extracted from the user's prompt in advance during inference, adding an extra pipeline step.
- Minor Benchmark Lags: While dominant in most categories, Ming-Flash-Omni still slightly lags behind specialized models like GLM-4.6-V on specific document/chart tasks (ChartQA, DocVQA) and multi-image understanding (MuirBench).
Future work will likely focus on removing the text-rendering preprocessing bottleneck and further optimizing the unified training dynamics to completely close the gap with proprietary systems.
Summary updated