Unleashing the Potential of Large Language Models for Text-to-Image Generation Through Autoregressive Representation Alignment

Xing Xie, Jiawei Liu, Ziyue Lin, Huijie Fan, Zhi Han, Yandong Tang, Liangqiong Qu

TL;DR

Autoregressive Representation Alignment (ARRA) is a "plug-and-play" training framework that allows standard Large Language Models (LLMs) to generate high-quality, globally coherent images without any architectural modifications. By aligning the LLM’s internal hidden states with global visual features from external models (like CLIP) during training, ARRA fixes the "fragmented image" problem common in autoregressive generation.

What Problem Does This Paper Address?

In the world of generative AI, there is a divide between Diffusion models (like Stable Diffusion) and Autoregressive (AR) models (like LlamaGen). AR models are attractive because they use the same "next-token prediction" logic that makes ChatGPT so powerful: they treat an image as a sequence of discrete visual "words."

However, images aren't exactly like sentences. While a sentence follows a linear flow, an image requires global spatial coherence. Standard AR models often struggle with this; they focus so much on predicting the next pixel-patch that they lose track of the "big picture." This leads to semantic mismatches or fragmented results—for example, a generated medical X-ray might have anatomically impossible, misaligned ribs because the model was only looking at local dependencies.

Previous attempts to fix this usually involve "grafting" new parts onto the LLM, such as cross-attention layers or diffusion modules. These architectural changes are messy, often breaking the model's compatibility with pre-trained weights and scaling laws.

Methods

The authors propose ARRA, a training-only intervention. The core innovation is the <HYBNEXT> (Hybrid Next) token.

During training, the model is forced to satisfy two constraints simultaneously for every generated token:

  1. Local Constraint: The standard autoregressive loss—predicting the correct discrete visual token from a codebook.
  2. Global Constraint: A new Global Visual Alignment (GVA) loss. The hidden state of the <HYBNEXT> token is passed through a small projection layer and forced to align (via cosine similarity) with a global feature vector extracted from a pre-trained "foundation" model like CLIP or BioMedCLIP.

Crucially, this alignment happens only during training. At inference time, the projection layer and the external encoder are discarded. The LLM has "learned" the global structure implicitly, allowing it to generate coherent images using its original, unmodified architecture.

Datasets

The researchers evaluated ARRA across both natural and specialized domains:

  • Natural Images: ImageNet and a 2.4M high-quality subset of LAION-COCO.
  • Medical Imaging: MIMIC-CXR (Chest X-rays) and DeepEyeNet (Fundus/eye images).
  • Encoders: They used CLIP-L for natural images and specialized encoders like BioMedCLIP and MedSAM for medical tasks.

Key Insights and Ideas

  • Interleaving is Better than Prefixing: Many models try to provide global context by putting a special token at the very beginning of a sequence. ARRA finds that interleaving the alignment at every step is far more effective. This prevents "attention decay," where the model "forgets" the global context over long sequences.
  • The Power of the [CLS] Token: When extracting features from external models, using the [CLS] token (which aggregates global information via self-attention) significantly outperformed simple average pooling of image patches.
  • Training Objective vs. Architecture: The paper argues that we don't need more complex models; we need better training objectives. By "distilling" the spatial knowledge of a vision model into the hidden states of an LLM, we can solve the coherence problem without touching a single line of the LLM's inference code.

Key Results

ARRA demonstrated significant improvements across the board:

  • From Scratch: When training a T2I model (LlamaGen) from scratch, ARRA reduced the Fréchet Inception Distance (FID)—a key measure of image quality—by 16.6% on ImageNet and 12.0% on LAION-COCO.
  • Domain Adaptation: On medical datasets, ARRA-Adapt achieved an 18.6% FID reduction over standard fine-tuning on MIMIC-CXR.
  • Qualitative Success: In medical imaging, where precision is vital, ARRA-generated images showed much better anatomical consistency (e.g., correctly placed lesions and realistic bone structures) compared to baseline LLMs which produced "hallucinated" or fragmented anatomy.

Limitations and Future Work

While ARRA is highly effective, it relies on the availability of a strong external "teacher" model (the visual foundation model). If the external encoder has its own biases or lacks understanding of a specific niche domain, the alignment might be suboptimal.

Future work could explore scaling ARRA to even larger "frontier" models or investigating if this alignment strategy can be used for other modalities, like video or 3D generation, where global temporal and spatial coherence are even harder to maintain.

Summary updated

Abstract

We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural changes. Unlike prior work that requires complex architectural redesigns, ARRA aligns LLM hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training from text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive LLMs like Chameleon and LlamaGen, all without framework modifications. For domain adaption, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR). By demonstrating that training objective redesign -- not just architectural innovation -- can resolve cross-modal global coherence challenges, ARRA offers a complementary paradigm for advancing autoregressive models. Code and models will be released to advance autoregressive image generation.
BibTeX
@inproceedings{xie2026unleashing,
  author = {Xing Xie and Jiawei Liu and Ziyue Lin and Huijie Fan and Zhi Han and Yandong Tang and Liangqiong Qu},
  title = {{Unleashing the Potential of Large Language Models for Text-to-Image Generation Through Autoregressive Representation Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i13.38089},
}

Loading related content...

Figure