Abstract
We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural changes. Unlike prior work that requires complex architectural redesigns, ARRA aligns LLM hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training from text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive LLMs like Chameleon and LlamaGen, all without framework modifications. For domain adaption, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR).
Introduction: The Challenge of Global Coherence
In the landscape of generative AI, a significant divide exists between Diffusion models and Autoregressive (AR) models. AR models are particularly compelling because they leverage the same "next-token prediction" logic that powers state-of-the-art Large Language Models (LLMs), treating images as sequences of discrete visual tokens. However, images possess inherent spatial dependencies that linear text sequences do not.
Standard AR models often suffer from a lack of global spatial coherence. Because they focus primarily on predicting the immediate next pixel-patch, they frequently lose track of the "big picture," leading to semantic mismatches or fragmented results. This is particularly evident in specialized domains like medical imaging, where a generated X-ray might exhibit anatomically impossible rib alignments or mislocated lesions because the model only optimized for local dependencies.
The ARRA Framework: Aligning Local and Global Features
To address these limitations without resorting to complex architectural redesigns—such as grafting cross-attention layers or diffusion modules—we propose Autoregressive Representation Alignment (ARRA). ARRA is a training-only intervention that introduces a novel <HYBNEXT> (Hybrid Next) token to bridge the gap between local token prediction and global semantic understanding.
During the training phase, the model is required to satisfy two simultaneous constraints for every generated token:
- Local Constraint: The standard autoregressive (AR) loss, which involves predicting the correct discrete visual token from a codebook.
- Global Constraint: A new Global Visual Alignment (GVA) loss. The hidden state of the
<HYBNEXT>token is passed through a projection layer and aligned via cosine similarity with a global feature vector extracted from a pre-trained "foundation" model (e.g., CLIP or BioMedCLIP).
Crucially, this alignment is only required during training. At inference time, the projection layer and external encoder are discarded. The LLM effectively "learns" the global structure implicitly, allowing it to generate coherent images using its original, unmodified architecture. This makes ARRA a truly "plug-and-play" solution for any autoregressive model.
Key Insights and Experimental Results
Our research highlights several critical findings for advancing AR image generation:
- Interleaving vs. Prefixing: While many models provide global context via a prefix token, ARRA finds that interleaving alignment at every step is significantly more effective. This prevents "attention decay," ensuring the model maintains global context throughout long generation sequences.
- Global Feature Extraction: Utilizing the
[CLS]token from external models—which aggregates global information via self-attention—outperformed simpler methods like average pooling of image patches.
ARRA demonstrated substantial performance gains across diverse datasets:
- Natural Images: When training the LlamaGen model from scratch, ARRA reduced the Fréchet Inception Distance (FID) by 16.6% on ImageNet and 12.0% on a high-quality subset of LAION-COCO.
- Domain Adaptation: In medical imaging (MIMIC-CXR), ARRA-Adapt achieved an 18.6% FID reduction over standard fine-tuning. Qualitatively, ARRA-generated medical images showed superior anatomical consistency, avoiding the "hallucinated" or fragmented structures common in baseline LLMs.
Conclusion
ARRA demonstrates that the challenges of cross-modal global coherence can be resolved through training objective redesign rather than architectural innovation. By distilling the spatial knowledge of vision foundation models into the hidden states of an LLM, we enable high-quality, globally coherent image generation while preserving the simplicity and scalability of the autoregressive paradigm. Future work will explore scaling ARRA to larger frontier models and extending this alignment strategy to temporal coherence in video generation.
Summary updated