Abstract
Auto-regressive models have made significant progress in the realm of language generation, yet they do not perform on par with diffusion models in the domain of image synthesis. In this work, we introduce MARS, a novel framework for T2I generation that incorporates a specially designed Semantic Vision-Language Integration Expert (SemVIE). This innovative component integrates pre-trained LLMs by independently processing linguistic and visual information, freezing the textual component while fine-tuning the visual component. This methodology preserves the NLP capabilities of LLMs while imbuing them with exceptional visual understanding. Building upon the powerful base of the pre-trained Qwen-7B, MARS stands out with its bilingual generative capabilities corresponding to both English and Chinese language prompts and the capacity for joint image and text generation. Notably, MARS requires only 9% of the GPU days needed by SD1.5, yet it achieves remarkable results across a variety of benchmarks, illustrating the training efficiency and the potential for swift deployment in various applications.
Introduction
While diffusion models currently dominate the text-to-image (T2I) landscape, auto-regressive (AR) models—the architecture behind Large Language Models (LLMs)—offer unique advantages due to their ability to handle discrete tokens. However, adapting AR models for image synthesis presents two major challenges: catastrophic forgetting of linguistic intelligence during fine-tuning and the high computational cost of generating long token sequences for high-resolution images. MARS (Mixture of Auto-Regressive Models) addresses these hurdles by leveraging frozen LLMs and a specialized "visual expert" module to achieve state-of-the-art results with unprecedented efficiency.
Methodology
The core innovation of MARS is the Semantic Vision-Language Integration Expert (SemVIE). Rather than fine-tuning the entire LLM, the framework introduces parallel "visual expert" layers. The original LLM weights (based on Qwen-7B) remain frozen to preserve natural language processing skills, while the SemVIE modules are trained specifically to process visual information. These experts are integrated into every layer of the transformer, enabling a deep, incremental fusion of text and vision.
To improve efficiency and coherence, MARS employs Next K-Token Prediction (NKTP), predicting a subset of $K$ future tokens simultaneously rather than a single next token. This approach captures longer-range dependencies essential for fine-grained image details. Training follows a three-stage curriculum:
- Alignment: Establishing basic image-text mapping at 256x256 resolution.
- Aesthetic Boost: Fine-tuning on a curated dataset of high-quality images with detailed captions.
- High-Res Refinement: Utilizing a cascading super-resolution strategy to reach 1024px resolution.
The model was trained on a massive corpus of approximately 200 million image-text pairs, including 50 million Chinese pairs for native bilingual support. A critical component of the data pipeline involved using vision-language models to rewrite noisy captions into dense, fine-grained descriptions (averaging 110 words), which significantly improved the model's ability to render minute visual details.
Results and Efficiency
MARS demonstrates exceptional performance across standard benchmarks. On the MS-COCO dataset, it achieved a Frechet Inception Distance (FID) of 6.92, outperforming Stable Diffusion 1.5 (9.22) and the larger Parti model (7.23). In T2I-CompBench evaluations for attribute binding, MARS showed superior accuracy in rendering specific colors and textures.
The most significant advantage of MARS is its training efficiency. The model achieved these results using only 587 A100 GPU days, which is approximately 9% of the compute required for Stable Diffusion 1.5. This efficiency underscores the potential of repurposing pre-trained LLMs for multimodal tasks rather than training vision models from scratch. Furthermore, MARS supports "joint generation," enabling the simultaneous production of complex text (e.g., a recipe) and corresponding images in a single inference pass.
Conclusion
MARS represents a significant step forward in efficient, high-fidelity text-to-image synthesis. By integrating visual experts into frozen LLMs, it preserves linguistic reasoning while achieving state-of-the-art visual generation. While auto-regressive models still face sampling bottlenecks compared to some distilled diffusion models, the "any-to-any" potential of the MARS framework suggests a path toward more universal multimodal assistants capable of handling text, images, and potentially other modalities in the future.
Summary updated