MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models

Haozhe Zhao, Zefan Cai, Shuzheng Si, Liang Chen, Jiuxiang Gu, Wen Xiao, Junjie Hu

Abstract

MENTOR is a novel autoregressive framework designed for high-quality, controllable image generation that effectively balances text and image conditioning. By employing a two-stage training paradigm and a streamlined transformer-based architecture, MENTOR achieves competitive performance on benchmarks such as DreamBench++ while significantly reducing computational overhead. Notably, the model can be trained in just 1.5 days on a single 8-GPU node, demonstrating a highly efficient path toward versatile multimodal generation.

1. Introduction

The integration of text and visual cues for image generation remains a significant challenge in creative AI. Existing models often struggle with three primary issues: modality imbalance, where the model favors one input type over another; stochasticity, which complicates precise control in diffusion-based systems; and prohibitive costs, requiring massive datasets and thousands of GPU hours for alignment. MENTOR addresses these challenges by proving that balanced, high-fidelity control can be achieved through an efficient autoregressive framework without the need for extreme computational resources.

2. Methodology

MENTOR departs from the complex cross-attention mechanisms typical of diffusion models, utilizing a unified Autoregressive (AR) framework.

2.1 Architecture

The architecture comprises three core components:

  • Multimodal Encoder: Employs frozen CLIP-Large and FlanT5-XL encoders to map images and text into a shared latent space.
  • MLP Projector: A lightweight module that projects these features into the generator's embedding space.
  • Autoregressive Generator: Based on LlamaGen-XL, this decoder-only transformer predicts image tokens sequentially, conditioned on the multimodal inputs.

2.2 Two-Stage Training Paradigm

The training process is divided into two distinct phases to ensure robust multimodal understanding:

  1. Multimodal Alignment Stage: Focuses on pixel-to-semantic mapping through tasks like image reconstruction and object segmentation.
  2. Multimodal Instruction Tuning Stage: Introduces "Image Recovery" (reconstructing distorted images via text prompts) and "Subject-driven Generation." This forces the model to integrate both modalities, preventing over-reliance on the reference image.
Qualitative results across various tasks including text-guided segmentation, multi-image generation, and subject-driven generation.

3. Experimental Setup

The researchers utilized a curated dataset of approximately 3 million samples, including CC12M, Midjourney-Niji, and synthetic data from Flux.1. For subject-driven tasks, the OminiControl (Subject200k) dataset was employed. Evaluation was conducted primarily on DreamBench and DreamBench++, with the latter specifically measuring the balance between Concept Preservation (CP) and Prompt Following (PF).

4. Key Results and Insights

MENTOR demonstrates that architectural simplicity and strategic training can lead to state-of-the-art efficiency.

4.1 Performance and Efficiency

MENTOR achieved the highest CP·PF score on DreamBench++, outperforming significantly larger models such as Emu2 (37B) and Lumina-mGPT (7B). In image reconstruction, it surpassed baselines like SEED-X and DreamEngine with lower error rates on COCO and JourneyDB.

Comparison of training data size and model size across various multimodal models, highlighting MENTOR's efficiency.

4.2 The Role of Image Recovery

A key insight is the use of the Image Recovery task as a regularizer. By requiring the model to reconstruct distorted images using text, it is forced to "fill in the blanks" with textual information, preventing visual dominance. Furthermore, the deterministic nature of AR next-token prediction provides more precise control for fine-grained tasks compared to stochastic diffusion processes.

5. Limitations and Future Work

While MENTOR is highly efficient, it is currently limited by its base generator (LlamaGen-XL), which affects absolute visual quality and spatial reasoning (e.g., complex text rendering). Future efforts will focus on integrating more powerful backbone models and scaling training data to improve aesthetics and reasoning capabilities. Additionally, further evaluation of social biases in multimodal generation is required.

Summary updated

Abstract

Recent text-to-image models produce high-quality results but still struggle with precise visual control, balancing multimodal inputs, and requiring extensive training for complex multimodal image generation. To address these limitations, we propose MENTOR, a novel autoregressive (AR) framework for efficient Multimodal-conditioned Tuning for Autoregressive multimodal image generation. MENTOR combines an AR image generator with a two-stage training paradigm, enabling fine-grained, token-level alignment between multimodal inputs and image outputs without relying on auxiliary adapters or cross-attention modules. The two-stage training consists of: (1) a multimodal alignment stage that establishes robust pixel- and semantic-level alignment, followed by (2) a multimodal instruction tuning stage that balances the integration of multimodal inputs and enhances generation controllability. Despite modest model size, suboptimal base components, and limited training resources, MENTOR achieves strong performance on the DreamBench++ benchmark, outperforming competitive baselines in concept preservation and prompt following. Additionally, our method delivers superior image reconstruction fidelity, broad task adaptability, and improved training efficiency compared to diffusion-based methods. Dataset, code, and models are available at: https://github.com/HaozheZhao/MENTOR
BibTeX
@misc{zhao2025mentor,
  author = {Haozhe Zhao and Zefan Cai and Shuzheng Si and Liang Chen and Jiuxiang Gu and Wen Xiao and Junjie Hu},
  title = {{MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models}},
  year = {2025},
  eprint = {2507.09574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09574},
}
Figure