STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning

Jie Qin, Jiancheng Huang, Limeng Qiao, Lin MaSubmission to ICLR 2026 · Rejected

TL;DR

STAR (STacked AutoRegressive scheme) presents an interference-free framework for unified multimodal learning—combining understanding, generation, and editing within a single architecture. By freezing a pre-trained multimodal LLM and stacking isomorphic autoregressive layers on top, STAR expands into text-to-image and editing capabilities via standard next-token prediction without degrading existing visual comprehension.

What Problem Does This Paper Address?

Modern multimodal large language models (MLLMs) face a persistent dilemma: models trained from scratch to handle both understanding (image-to-text) and generation (text-to-image) suffer from optimization conflicts and feature space misalignment, leading to catastrophic forgetting or sub-par generation.

Existing "warm-start" alternatives (such as BLIP3-o or MetaQuery) typically bridge a frozen MLLM to an external diffusion backbone using complex multi-layer connectors, auxiliary diffusion losses, or flow-matching objectives. These hybrid designs create substantial training complexity and feature discrepancies between discrete token spaces and continuous noise manifolds. STAR asks: Can we expand a frozen MLLM into a unified generation and editing model cleanly, using standard next-token prediction and zero auxiliary bridging connectors?

Methods

STAR avoids external cross-modality adapters through an end-to-end autoregressive design supported by four key pillars:

  1. Stacked Isomorphic AR Layers: The base vision-language backbone (e.g., Qwen2.5-VL) remains frozen to safeguard understanding. To add generative capabilities, isomorphic transformer layers are stacked directly on top ($T_{\text{full}} = T_{\text{base}} \oplus T_{\text{stack}}$). These layers replicate the base model’s topology and are warm-started by copying its top $N$ layers (16 layers for the 3B model, 14 for the 7B model).
  2. STAR-VQ Tokenizer: To supply dense, fine-grained visual tokens, the authors train a 1B-parameter discrete vector quantizer with a 65,536-entry, 512-dimensional codebook and $16\times$ downsampling. A 2-block DiT codebook projector is applied during training to prevent codebook collapse.
  3. Task-Progressive 4-Stage Training:
  4. Stage 1: Pretrain STAR-VQ on ImageNet and OpenImages.
  5. Stage 2: Pretrain Stacked AR layers on 60M general + 0.6M synthetic text-image pairs via standard next-token prediction loss.
  6. Stage 3: AR-Diffusion alignment, training a Lumina2-Image diffusion decoder (upscaling from 384 to 1024 resolution) conditioned on discrete VQ tokens via channel-wise concatenation.
  7. Stage 4: Unified instruction tuning for generation and editing (using sequence concatenation for source-image VAE latents) with a stop-gradient between the diffusion decoder and Stacked AR.
  8. Implicit Reasoning Mechanism: For knowledge-intensive prompts, the frozen base model first infers intermediate latent tokens containing world knowledge (e.g., expanding "the fastest land animal" into semantic tokens representing a cheetah) before the stacked layers synthesize the visual tokens.

Datasets

  • Training:
  • VQ Pretraining: ImageNet and OpenImages (120 epochs).
  • AR T2I Pretraining: 60M web text-image pairs + 0.6M synthetic pairs.
  • Diffusion Alignment & Tuning: 10M text-to-image subset (Stage 3); 6M T2I + 4M editing dataset (combining UltraEdit, HQ-Edit, OmniEdit, and 300K GPT-4o re-synthesized samples) (Stage 4).
  • Evaluation:
  • Multimodal Understanding: 9 standard benchmarks, including MMBench-EN, MMStar, MathVista, SEEDBench, MME, MMMU, OCRBench, POPE, and DocVQA.
  • Text-to-Image: GenEval (553 prompts), DPG-Bench (1,065 prompts), and WISEBench (1,000 prompts probing world knowledge).
  • Image Editing: MagicBrush (1,000 pairs) and ImgEdit-Bench (737 pairs).

Key Insights and Ideas

  • Homogeneous Layer Stacking Beats External Adapters: Initializing stacked layers directly from the top layers of the base MLLM provides built-in feature alignment, outperforming random initialization (0.439 vs. 0.374 on GenEval) and eliminating the need for projection bottlenecks.
  • Stage Separation Prevents Representation Collapse: Jointly training the AR layers and diffusion decoder from scratch causes chaotic early visual representations that harm diffusion training. Staging the AR pretraining before diffusion alignment improves GenEval by +0.07 (0.86 vs 0.79).
  • Two-Pass Implicit Reasoning Unlocks Knowledge-Heavy T2I: Abstract prompts fail when generative models lack semantic grounding. Letting the frozen base model infer intermediate semantic representations jumps the WISE reasoning score from 0.46 to 0.66 without adding extra parameters.

Key Results

  • Text-to-Image Generation: STAR-7B achieves 0.91 on GenEval (surpassing Ovis-U1's 0.89 and GPT-4o's 0.84) and 87.44 on DPG-Bench. On WISEBench world-knowledge evaluation, STAR-7B reaches 0.66.
  • Image Editing: On ImgEdit-Bench, STAR-7B achieves an overall score of 4.34, outperforming GPT-4o (4.20) and Ovis-U1 (4.00). On MagicBrush, STAR achieves state-of-the-art content preservation and fidelity ($L_1$ of 0.056, CLIP-I of 0.934, DINO of 0.857).
  • Multimodal Understanding: Retains full upstream comprehension with zero catastrophic forgetting (e.g., STAR-7B scores 83.9 on MMBench, 68.1 on MathVista, and 1690.1 on MME).

Limitations and Future Work

  • Visual Text and Complex Geometry: The authors note qualitative failure cases (Figure 8) in rendering dense in-image typography and complex physical poses (e.g., precise 90-degree limb articulations).
  • Strict Multi-Entity Counting: Generating scenes with precise, high-count entities (such as "ten officers") remains a failure mode.
  • Inference Latency: Two-pass generation using implicit reasoning adds autoregressive steps before diffusion decoding, which may increase latency in interactive applications.

Summary updated

Abstract

Multimodal large language models (MLLMs) play a pivotal role in advancing the quest for general artificial intelligence. However, achieving unified target for multimodal understanding and generation remains challenging due to optimization conflicts and performance trade-offs. To effectively enhance generative performance while preserving existing comprehension capabilities, we introduce STAR: Stacked AutoRegressive scheme for task-progressive unified multimodal learning. This approach decomposes multimodal learning into multiple stages: understanding, generation, and editing. By freezing the parameters of the fundamental autoregressive (AR) model and progressively stacking isomorphic AR modules, it avoids cross-task interference while expanding the model's capabilities. Concurrently, we introduce a high-capacity VQ to enhance the granularity of image representations and employ an implicit reasoning mechanism to improve generation quality under complex conditions. Experiments demonstrate that STAR achieves state-of-the-art performance on GenEval (0.91), DPG-Bench (87.44), and ImgEdit (4.34), validating its efficacy for unified multimodal learning.
BibTeX
@misc{qin2025star,
  author = {Jie Qin and Jiancheng Huang and Limeng Qiao and Lin Ma},
  title = {{STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning}},
  year = {2025},
  eprint = {2512.13752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.13752},
}

Loading related content...

Figure