UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Jiancheng Pan, Yuzheng Liu, Hengyuan Cao, Qiyuan Zhang, Yangyan Li, Biao Gong, Zheng Zhang, Ka Long Cheng, Xinyang Wang, Yujun ShenarXiv 2026 · arXiv

TL;DR

UniSpark-9B is a unified multimodal model that masters both visual understanding and high-resolution image generation by blending standard "full" attention with efficient linear attention. By introducing a novel blockwise protocol to isolate noisy generation states from clean context, it achieves state-of-the-art performance on benchmarks like MME-S while delivering up to a 3.23x inference speedup for high-resolution synthesis.

What Problem Does This Paper Address?

Traditionally, AI models have been specialists: one model understands images (like VQA), while another generates them (like Stable Diffusion). Unifying these tasks into a single "omni-modal" model is the current frontier, but it faces two major hurdles.

First, there is a computational bottleneck. Standard Transformer attention scales quadratically with sequence length, making high-resolution images (which require thousands of tokens) incredibly expensive to process. Second, there is a context pollution problem in interleaved tasks (e.g., "Generate an image, then explain it"). During the iterative process of generating an image, the model works with "noisy" intermediate states. If these noisy states are allowed to leak into the model's memory, they degrade the quality of subsequent text or image predictions.

Methods

UniSpark tackles these issues with a Hybrid Mixture-of-Transformer-Experts (MoT) architecture.

  1. Hybrid Attention: The model alternates between standard Softmax attention (for precise, explicit token interactions) and GatedDeltaNet linear attention. Linear attention propagates context through a persistent recurrent state, allowing the model to handle long visual sequences with $O(L)$ efficiency instead of $O(L^2)$.
  2. Expert Routing: It uses specialized "experts" within the layers. Text and semantic visual features are routed to an understanding expert, while VAE latents (the building blocks of images) are routed to a generation expert.
  3. Blockwise Information Flow: To solve the noise leakage problem, UniSpark implements a strict "public vs. private" boundary. As shown in Figure 5, noisy visual targets are treated as private; they can read the preceding context but are hidden from future blocks. Once an image is fully generated and "cleaned," it is re-encoded and added to the "public" context, updating the linear attention's recurrent state for future use.
The information flow mechanism separates private noisy targets from public clean context.

Datasets

The authors trained UniSpark on a massive, diverse corpus:

  • Understanding: 213.4M samples covering captioning, VQA, OCR, and GUI navigation.
  • Generation: 159.7M image generation samples.
  • Interleaved: 624.8K sequences of alternating text and images.
  • Video: An exploratory set of 24.24M video generation and 3.29M video editing samples.

Evaluation relied on standard benchmarks like MME and MMBench for understanding, and GenEval and DPG-Bench for generation fidelity.

Key Insights and Ideas

  • Linear Attention as Persistent Memory: The paper demonstrates that linear attention isn't just a compression trick; it acts as a robust recurrent state that can maintain "clean" visual history across multiple turns of a conversation or generation process.
  • Separation of Concerns: By using MoT, the model doesn't have to force "understanding" parameters to learn "generation" logic. The experts share the attention context but maintain specialized internal weights.
  • Curriculum Matters: The model was trained in four stages, starting at low resolution (256px) to establish the foundation and gradually scaling to 2K resolution. This progressive scaling is vital for stabilizing the joint training of two very different objectives (text cross-entropy and visual flow matching).

Key Results

UniSpark-9B demonstrates that you don't have to sacrifice performance for unification.

  • Understanding: It achieved a score of 2470.3 on MME-S, outperforming other unified models like BAGEL and SenseNova-U1 (see Figure 1).
  • Generation: It scored 0.83 on GenEval, showing strong object and color fidelity, though it still finds complex spatial relations (like "object A is behind object B") challenging.
  • Efficiency: The hybrid approach is a game-changer for high-res tasks. At 1024x1024 resolution, UniSpark is 3.23x faster than full-attention baselines.
UniSpark leads in multimodal understanding benchmarks while remaining competitive in generation.

Qualitatively, the model excels at "interleaved" tasks, such as generating a sequence of images that maintain visual continuity, as seen in the "silk ribbon" and "pulsing jellyfish" examples in the paper's appendix.

Limitations and Future Work

While UniSpark is powerful, the authors note that compositional control (e.g., counting objects accurately or precise positioning) remains a hurdle for the generation branch. Additionally, while the hybrid attention saves time, it comes with a higher peak memory cost at extreme resolutions (like 2048px) compared to some competitors.

Future work will likely focus on tightening the integration of video data—currently treated as a post-training extension—directly into the foundation stages to enable more fluid video understanding and editing.

Summary updated

Abstract

We present UniSpark-9B, a unified multimodal model that investigates hybrid attention for visual understanding and generation. Built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch, UniSpark combines full attention for explicit token interactions with GatedDeltaNet linear attention for compact context propagation. Autoregressive text prediction and visual flow matching jointly support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction. In interleaved generation, noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction through attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers. The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum jointly trains multimodal understanding and visual generation, refining answer accuracy and instruction following while extending image synthesis to 2K. UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. Qualitative examples further illustrate its capabilities in interleaved image generation, video synthesis, and video editing. Together, these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation.
BibTeX
@misc{pan2026unispark,
  author = {Jiancheng Pan and Yuzheng Liu and Hengyuan Cao and Qiyuan Zhang and Yangyan Li and Biao Gong and Zheng Zhang and Ka Long Cheng and Xinyang Wang and Yujun Shen},
  title = {{UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation}},
  year = {2026},
  eprint = {2611.16517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16517},
}
Figure