Abstract
MENTOR is a novel autoregressive framework designed for high-quality, controllable image generation that effectively balances text and image conditioning. By employing a two-stage training paradigm and a streamlined transformer-based architecture, MENTOR achieves competitive performance on benchmarks such as DreamBench++ while significantly reducing computational overhead. Notably, the model can be trained in just 1.5 days on a single 8-GPU node, demonstrating a highly efficient path toward versatile multimodal generation.
1. Introduction
The integration of text and visual cues for image generation remains a significant challenge in creative AI. Existing models often struggle with three primary issues: modality imbalance, where the model favors one input type over another; stochasticity, which complicates precise control in diffusion-based systems; and prohibitive costs, requiring massive datasets and thousands of GPU hours for alignment. MENTOR addresses these challenges by proving that balanced, high-fidelity control can be achieved through an efficient autoregressive framework without the need for extreme computational resources.
2. Methodology
MENTOR departs from the complex cross-attention mechanisms typical of diffusion models, utilizing a unified Autoregressive (AR) framework.
2.1 Architecture
The architecture comprises three core components:
- Multimodal Encoder: Employs frozen CLIP-Large and FlanT5-XL encoders to map images and text into a shared latent space.
- MLP Projector: A lightweight module that projects these features into the generator's embedding space.
- Autoregressive Generator: Based on LlamaGen-XL, this decoder-only transformer predicts image tokens sequentially, conditioned on the multimodal inputs.
2.2 Two-Stage Training Paradigm
The training process is divided into two distinct phases to ensure robust multimodal understanding:
- Multimodal Alignment Stage: Focuses on pixel-to-semantic mapping through tasks like image reconstruction and object segmentation.
- Multimodal Instruction Tuning Stage: Introduces "Image Recovery" (reconstructing distorted images via text prompts) and "Subject-driven Generation." This forces the model to integrate both modalities, preventing over-reliance on the reference image.
3. Experimental Setup
The researchers utilized a curated dataset of approximately 3 million samples, including CC12M, Midjourney-Niji, and synthetic data from Flux.1. For subject-driven tasks, the OminiControl (Subject200k) dataset was employed. Evaluation was conducted primarily on DreamBench and DreamBench++, with the latter specifically measuring the balance between Concept Preservation (CP) and Prompt Following (PF).
4. Key Results and Insights
MENTOR demonstrates that architectural simplicity and strategic training can lead to state-of-the-art efficiency.
4.1 Performance and Efficiency
MENTOR achieved the highest CP·PF score on DreamBench++, outperforming significantly larger models such as Emu2 (37B) and Lumina-mGPT (7B). In image reconstruction, it surpassed baselines like SEED-X and DreamEngine with lower error rates on COCO and JourneyDB.
4.2 The Role of Image Recovery
A key insight is the use of the Image Recovery task as a regularizer. By requiring the model to reconstruct distorted images using text, it is forced to "fill in the blanks" with textual information, preventing visual dominance. Furthermore, the deterministic nature of AR next-token prediction provides more precise control for fine-grained tasks compared to stochastic diffusion processes.
5. Limitations and Future Work
While MENTOR is highly efficient, it is currently limited by its base generator (LlamaGen-XL), which affects absolute visual quality and spatial reasoning (e.g., complex text rendering). Future efforts will focus on integrating more powerful backbone models and scaling training data to improve aesthetics and reasoning capabilities. Additionally, further evaluation of social biases in multimodal generation is required.
Summary updated