MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, Ziwei Huang, Leilei Gan, Hao Jiang

Abstract

Auto-regressive models have made significant progress in the realm of language generation, yet they do not perform on par with diffusion models in the domain of image synthesis. In this work, we introduce MARS, a novel framework for T2I generation that incorporates a specially designed Semantic Vision-Language Integration Expert (SemVIE). This innovative component integrates pre-trained LLMs by independently processing linguistic and visual information, freezing the textual component while fine-tuning the visual component. This methodology preserves the NLP capabilities of LLMs while imbuing them with exceptional visual understanding. Building upon the powerful base of the pre-trained Qwen-7B, MARS stands out with its bilingual generative capabilities corresponding to both English and Chinese language prompts and the capacity for joint image and text generation. Notably, MARS requires only 9% of the GPU days needed by SD1.5, yet it achieves remarkable results across a variety of benchmarks, illustrating the training efficiency and the potential for swift deployment in various applications.

Introduction

While diffusion models currently dominate the text-to-image (T2I) landscape, auto-regressive (AR) models—the architecture behind Large Language Models (LLMs)—offer unique advantages due to their ability to handle discrete tokens. However, adapting AR models for image synthesis presents two major challenges: catastrophic forgetting of linguistic intelligence during fine-tuning and the high computational cost of generating long token sequences for high-resolution images. MARS (Mixture of Auto-Regressive Models) addresses these hurdles by leveraging frozen LLMs and a specialized "visual expert" module to achieve state-of-the-art results with unprecedented efficiency.

Methodology

The core innovation of MARS is the Semantic Vision-Language Integration Expert (SemVIE). Rather than fine-tuning the entire LLM, the framework introduces parallel "visual expert" layers. The original LLM weights (based on Qwen-7B) remain frozen to preserve natural language processing skills, while the SemVIE modules are trained specifically to process visual information. These experts are integrated into every layer of the transformer, enabling a deep, incremental fusion of text and vision.

The SemVIE architecture integrates visual experts into a frozen LLM backbone.

To improve efficiency and coherence, MARS employs Next K-Token Prediction (NKTP), predicting a subset of $K$ future tokens simultaneously rather than a single next token. This approach captures longer-range dependencies essential for fine-grained image details. Training follows a three-stage curriculum:

  1. Alignment: Establishing basic image-text mapping at 256x256 resolution.
  2. Aesthetic Boost: Fine-tuning on a curated dataset of high-quality images with detailed captions.
  3. High-Res Refinement: Utilizing a cascading super-resolution strategy to reach 1024px resolution.

The model was trained on a massive corpus of approximately 200 million image-text pairs, including 50 million Chinese pairs for native bilingual support. A critical component of the data pipeline involved using vision-language models to rewrite noisy captions into dense, fine-grained descriptions (averaging 110 words), which significantly improved the model's ability to render minute visual details.

Results and Efficiency

MARS demonstrates exceptional performance across standard benchmarks. On the MS-COCO dataset, it achieved a Frechet Inception Distance (FID) of 6.92, outperforming Stable Diffusion 1.5 (9.22) and the larger Parti model (7.23). In T2I-CompBench evaluations for attribute binding, MARS showed superior accuracy in rendering specific colors and textures.

Performance comparison showing MARS achieving superior FID scores with significantly lower training time.

The most significant advantage of MARS is its training efficiency. The model achieved these results using only 587 A100 GPU days, which is approximately 9% of the compute required for Stable Diffusion 1.5. This efficiency underscores the potential of repurposing pre-trained LLMs for multimodal tasks rather than training vision models from scratch. Furthermore, MARS supports "joint generation," enabling the simultaneous production of complex text (e.g., a recipe) and corresponding images in a single inference pass.

Conclusion

MARS represents a significant step forward in efficient, high-fidelity text-to-image synthesis. By integrating visual experts into frozen LLMs, it preserves linguistic reasoning while achieving state-of-the-art visual generation. While auto-regressive models still face sampling bottlenecks compared to some distilled diffusion models, the "any-to-any" potential of the MARS framework suggests a path toward more universal multimodal assistants capable of handling text, images, and potentially other modalities in the future.

Summary updated

Abstract

Auto-regressive models have made significant progress in the realm of language generation, yet they do not perform on par with diffusion models in the domain of image synthesis. In this work, we introduce MARS, a novel framework for T2I generation that incorporates a specially designed Semantic Vision-Language Integration Expert (SemVIE). This innovative component integrates pre-trained LLMs by independently processing linguistic and visual information, freezing the textual component while fine-tuning the visual component. This methodology preserves the NLP capabilities of LLMs while imbuing them with exceptional visual understanding. Building upon the powerful base of the pre-trained Qwen-7B, MARS stands out with its bilingual generative capabilities corresponding to both English and Chinese language prompts and the capacity for joint image and text generation. The flexibility of this framework lends itself to migration towards any-to-any task adaptability. Furthermore, MARS employs a multi-stage training strategy that first establishes robust image-text alignment through complementary bidirectional tasks and subsequently concentrates on refining the T2I generation process, significantly augmenting text-image synchrony and the granularity of image details. Notably, MARS requires only 9% of the GPU days needed by SD1.5, yet it achieves remarkable results across a variety of benchmarks, illustrating the training efficiency and the potential for swift deployment in various applications.
BibTeX
@inproceedings{he2025mars,
  author = {Wanggui He and Siming Fu and Mushui Liu and Xierui Wang and Wenyi Xiao and Fangxun Shu and Yi Wang and Lei Zhang and Zhelun Yu and Haoyuan Li and Ziwei Huang and Leilei Gan and Hao Jiang},
  title = {{MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i16.33882},
}
Figure