STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning

Jie Qin, Jiancheng Huang, Limeng Qiao, Lin Ma

Abstract

Multimodal large language models (MLLMs) play a pivotal role in advancing the quest for general artificial intelligence. However, achieving unified target for multimodal understanding and generation remains challenging due to optimization conflicts and performance trade-offs. To effectively enhance generative performance while preserving existing comprehension capabilities, we introduce STAR: a STacked AutoRegressive scheme for task-progressive unified multimodal learning. This approach decomposes multimodal learning into multiple stages: understanding, generation, and editing. By freezing the parameters of the fundamental autoregressive (AR) model and progressively stacking isomorphic AR modules, it avoids cross-task interference while expanding the model's capabilities. Concurrently, we introduce a high-capacity VQ to enhance the granularity of image representations and employ an implicit reasoning mechanism to improve generation quality under complex conditions. Experiments demonstrate that STAR achieves state-of-the-art performance on GenEval (0.91), DPG-Bench (87.44), and ImgEdit (4.34), validating its efficacy for unified multimodal learning.

Introduction

In the pursuit of Artificial General Intelligence (AGI), the development of unified models capable of seamless multimodal understanding, reasoning, generation, and editing is a primary objective. However, current Multimodal Large Language Models (MLLMs) often encounter a "capability ceiling." When models are trained to perform both understanding (image-to-text) and generation (text-to-image) tasks simultaneously, optimization objectives frequently conflict. This often results in "catastrophic interference," where improving generative capabilities leads to a degradation in the model's ability to reason or describe visual content accurately.

Existing "warm-start" methodologies attempt to mitigate this by integrating diffusion models with LLMs. However, these approaches typically require complex bridge modules and heterogeneous loss functions, complicating the training process and often failing to fully preserve the original comprehension strengths of the base model. To address these challenges, we propose STAR, a framework that utilizes a task-progressive training strategy to expand model capabilities without compromising existing intelligence.

Methodology

The STAR Architecture

The core innovation of STAR is the STacked AutoRegressive scheme. Instead of fine-tuning the entire model for new tasks, STAR takes a high-performing, frozen understanding backbone (such as Qwen2.5-VL) and stacks a set of isomorphic (architecturally identical) layers on top. These new layers are initialized from the base model's final layers, ensuring they inherit the established feature space. This design eliminates the need for complex adapters or bridge layers, allowing for an elegant expansion of the model's functional capacity.

The STAR architecture featuring stacked isomorphic autoregressive layers on top of a frozen multimodal backbone.

Task-Progressive Training

STAR expands its capabilities through four distinct stages:

  1. STAR-VQ Tokenization: We developed a 1B-parameter Vector Quantizer with a 65,536-entry codebook. This high-capacity tokenizer converts images into granular discrete tokens, providing a superior foundation for high-quality generation compared to standard VQ-GANs.
  2. Stacked AR Layers: The stacked isomorphic layers are the only components trained for generation, preventing interference with the frozen understanding backbone.
  3. Implicit Reasoning Mechanism: For complex prompts, the frozen base model generates "implicit latent tokens." This allows the model to perform semantic reasoning before the stacked layers begin predicting image tokens, effectively separating "thinking" from "drawing."
  4. Diffusion Decoder: To achieve high-fidelity 1024x1024 pixel outputs, STAR employs a diffusion-based decoder (based on Lumina2), combining the efficiency of autoregressive modeling with the visual quality of diffusion.

Experiments and Results

Datasets and Benchmarks

STAR was trained on a massive scale, utilizing 60 million general text-image pairs and 10 million high-quality alignment images. For image editing, we utilized 4 million samples from datasets like UltraEdit and HQ-Edit, supplemented by 300k proprietary high-fidelity images synthesized via GPT-4o.

Evaluation spanned multiple domains:

  • Understanding: MMBench, MMMU, and MathVista.
  • Generation: GenEval (alignment), DPG-Bench (composition), and WISE (world knowledge).
  • Editing: MagicBrush and ImgEdit.

Performance Analysis

STAR establishes new state-of-the-art (SOTA) benchmarks across unified multimodal tasks. It achieved a score of 0.91 on GenEval, a 2% improvement over previous leaders like Ovis-U1, and led on DPG-Bench with a score of 87.44. Crucially, because the base model remains frozen, STAR exhibits zero degradation in understanding benchmarks, maintaining top-tier performance in reasoning and description.

Qualitative results of STAR across text-to-image generation, image editing, and knowledge-based reasoning tasks.

In image editing, STAR-7B achieved a score of 4.34 on the ImgEdit benchmark, outperforming both specialized editing models and other unified frameworks. The implicit reasoning mechanism proved particularly effective for tasks requiring world knowledge, such as generating specific mythological weapons or identifying the fastest land animal.

Conclusion

The STAR framework demonstrates a scalable and efficient path toward unified multimodal intelligence. By stacking isomorphic layers on a frozen backbone, it successfully decouples understanding from generation, avoiding the performance trade-offs inherent in traditional joint training. While the current four-stage pipeline and diffusion-based decoding introduce some computational complexity, the architecture's ability to preserve and extend model intelligence makes it a robust candidate for future scaling to even larger base models.

Summary updated

Abstract

Multimodal large language models (MLLMs) play a pivotal role in advancing the quest for general artificial intelligence. However, achieving unified target for multimodal understanding and generation remains challenging due to optimization conflicts and performance trade-offs. To effectively enhance generative performance while preserving existing comprehension capabilities, we introduce STAR: a STacked AutoRegressive scheme for task-progressive unified multimodal learning. This approach decomposes multimodal learning into multiple stages: understanding, generation, and editing. By freezing the parameters of the fundamental autoregressive (AR) model and progressively stacking isomorphic AR modules, it avoids cross-task interference while expanding the model's capabilities. Concurrently, we introduce a high-capacity VQ to enhance the granularity of image representations and employ an implicit reasoning mechanism to improve generation quality under complex conditions. Experiments demonstrate that STAR achieves state-of-the-art performance on GenEval (0.91), DPG-Bench (87.44), and ImgEdit (4.34), validating its efficacy for unified multimodal learning.
BibTeX
@misc{qin2025star,
  author = {Jie Qin and Jiancheng Huang and Limeng Qiao and Lin Ma},
  title = {{STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning}},
  year = {2025},
  eprint = {2512.13752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.13752},
}
Figure