Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment

Xing Xie, Jiawei Liu, Ziyue Lin, Huijie Fan, Zhi Han, Yandong Tang, Liangqiong Qu

Abstract

We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural changes. Unlike prior work that requires complex architectural redesigns, ARRA aligns LLM hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training from text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive LLMs like Chameleon and LlamaGen, all without framework modifications. For domain adaption, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR).

Introduction: The Challenge of Global Coherence

In the landscape of generative AI, a significant divide exists between Diffusion models and Autoregressive (AR) models. AR models are particularly compelling because they leverage the same "next-token prediction" logic that powers state-of-the-art Large Language Models (LLMs), treating images as sequences of discrete visual tokens. However, images possess inherent spatial dependencies that linear text sequences do not.

Standard AR models often suffer from a lack of global spatial coherence. Because they focus primarily on predicting the immediate next pixel-patch, they frequently lose track of the "big picture," leading to semantic mismatches or fragmented results. This is particularly evident in specialized domains like medical imaging, where a generated X-ray might exhibit anatomically impossible rib alignments or mislocated lesions because the model only optimized for local dependencies.

Figure 1: Comparison between standard NTP-based LLMs and the ARRA framework, highlighting the difference between local constraints and hybrid local/global constraints.

The ARRA Framework: Aligning Local and Global Features

To address these limitations without resorting to complex architectural redesigns—such as grafting cross-attention layers or diffusion modules—we propose Autoregressive Representation Alignment (ARRA). ARRA is a training-only intervention that introduces a novel <HYBNEXT> (Hybrid Next) token to bridge the gap between local token prediction and global semantic understanding.

During the training phase, the model is required to satisfy two simultaneous constraints for every generated token:

  1. Local Constraint: The standard autoregressive (AR) loss, which involves predicting the correct discrete visual token from a codebook.
  2. Global Constraint: A new Global Visual Alignment (GVA) loss. The hidden state of the <HYBNEXT> token is passed through a projection layer and aligned via cosine similarity with a global feature vector extracted from a pre-trained "foundation" model (e.g., CLIP or BioMedCLIP).
Figure 2: Detailed architecture of the ARRA framework, showing the integration of the HYBNEXT token, the VQ-Tokenizer, and the representation extraction process for global alignment.

Crucially, this alignment is only required during training. At inference time, the projection layer and external encoder are discarded. The LLM effectively "learns" the global structure implicitly, allowing it to generate coherent images using its original, unmodified architecture. This makes ARRA a truly "plug-and-play" solution for any autoregressive model.

Key Insights and Experimental Results

Our research highlights several critical findings for advancing AR image generation:

  • Interleaving vs. Prefixing: While many models provide global context via a prefix token, ARRA finds that interleaving alignment at every step is significantly more effective. This prevents "attention decay," ensuring the model maintains global context throughout long generation sequences.
  • Global Feature Extraction: Utilizing the [CLS] token from external models—which aggregates global information via self-attention—outperformed simpler methods like average pooling of image patches.

ARRA demonstrated substantial performance gains across diverse datasets:

  • Natural Images: When training the LlamaGen model from scratch, ARRA reduced the Fréchet Inception Distance (FID) by 16.6% on ImageNet and 12.0% on a high-quality subset of LAION-COCO.
  • Domain Adaptation: In medical imaging (MIMIC-CXR), ARRA-Adapt achieved an 18.6% FID reduction over standard fine-tuning. Qualitatively, ARRA-generated medical images showed superior anatomical consistency, avoiding the "hallucinated" or fragmented structures common in baseline LLMs.

Conclusion

ARRA demonstrates that the challenges of cross-modal global coherence can be resolved through training objective redesign rather than architectural innovation. By distilling the spatial knowledge of vision foundation models into the hidden states of an LLM, we enable high-quality, globally coherent image generation while preserving the simplicity and scalability of the autoregressive paradigm. Future work will explore scaling ARRA to larger frontier models and extending this alignment strategy to temporal coherence in video generation.

Summary updated

Abstract

We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural changes. Unlike prior work that requires complex architectural redesigns, ARRA aligns LLM hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training from text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive LLMs like Chameleon and LlamaGen, all without framework modifications. For domain adaption, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR). By demonstrating that training objective redesign -- not just architectural innovation -- can resolve cross-modal global coherence challenges, ARRA offers a complementary paradigm for advancing autoregressive models. Code and models will be released to advance autoregressive image generation.
BibTeX
@inproceedings{xie2026unleashing,
  author = {Xing Xie and Jiawei Liu and Ziyue Lin and Huijie Fan and Zhi Han and Yandong Tang and Liangqiong Qu},
  title = {{Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i13.38089},
}

Loading related content...

Figure