TL;DR
Autoregressive Representation Alignment (ARRA) is a "plug-and-play" training framework that allows standard Large Language Models (LLMs) to generate high-quality, globally coherent images without any architectural modifications. By aligning the LLM’s internal hidden states with global visual features from external models (like CLIP) during training, ARRA fixes the "fragmented image" problem common in autoregressive generation.
What Problem Does This Paper Address?
In the world of generative AI, there is a divide between Diffusion models (like Stable Diffusion) and Autoregressive (AR) models (like LlamaGen). AR models are attractive because they use the same "next-token prediction" logic that makes ChatGPT so powerful: they treat an image as a sequence of discrete visual "words."
However, images aren't exactly like sentences. While a sentence follows a linear flow, an image requires global spatial coherence. Standard AR models often struggle with this; they focus so much on predicting the next pixel-patch that they lose track of the "big picture." This leads to semantic mismatches or fragmented results—for example, a generated medical X-ray might have anatomically impossible, misaligned ribs because the model was only looking at local dependencies.
Previous attempts to fix this usually involve "grafting" new parts onto the LLM, such as cross-attention layers or diffusion modules. These architectural changes are messy, often breaking the model's compatibility with pre-trained weights and scaling laws.
Methods
The authors propose ARRA, a training-only intervention. The core innovation is the <HYBNEXT> (Hybrid Next) token.
During training, the model is forced to satisfy two constraints simultaneously for every generated token:
- Local Constraint: The standard autoregressive loss—predicting the correct discrete visual token from a codebook.
- Global Constraint: A new Global Visual Alignment (GVA) loss. The hidden state of the
<HYBNEXT>token is passed through a small projection layer and forced to align (via cosine similarity) with a global feature vector extracted from a pre-trained "foundation" model like CLIP or BioMedCLIP.
Crucially, this alignment happens only during training. At inference time, the projection layer and the external encoder are discarded. The LLM has "learned" the global structure implicitly, allowing it to generate coherent images using its original, unmodified architecture.
Datasets
The researchers evaluated ARRA across both natural and specialized domains:
- Natural Images: ImageNet and a 2.4M high-quality subset of LAION-COCO.
- Medical Imaging: MIMIC-CXR (Chest X-rays) and DeepEyeNet (Fundus/eye images).
- Encoders: They used CLIP-L for natural images and specialized encoders like BioMedCLIP and MedSAM for medical tasks.
Key Insights and Ideas
- Interleaving is Better than Prefixing: Many models try to provide global context by putting a special token at the very beginning of a sequence. ARRA finds that interleaving the alignment at every step is far more effective. This prevents "attention decay," where the model "forgets" the global context over long sequences.
- The Power of the [CLS] Token: When extracting features from external models, using the
[CLS]token (which aggregates global information via self-attention) significantly outperformed simple average pooling of image patches. - Training Objective vs. Architecture: The paper argues that we don't need more complex models; we need better training objectives. By "distilling" the spatial knowledge of a vision model into the hidden states of an LLM, we can solve the coherence problem without touching a single line of the LLM's inference code.
Key Results
ARRA demonstrated significant improvements across the board:
- From Scratch: When training a T2I model (LlamaGen) from scratch, ARRA reduced the Fréchet Inception Distance (FID)—a key measure of image quality—by 16.6% on ImageNet and 12.0% on LAION-COCO.
- Domain Adaptation: On medical datasets, ARRA-Adapt achieved an 18.6% FID reduction over standard fine-tuning on MIMIC-CXR.
- Qualitative Success: In medical imaging, where precision is vital, ARRA-generated images showed much better anatomical consistency (e.g., correctly placed lesions and realistic bone structures) compared to baseline LLMs which produced "hallucinated" or fragmented anatomy.
Limitations and Future Work
While ARRA is highly effective, it relies on the availability of a strong external "teacher" model (the visual foundation model). If the external encoder has its own biases or lacks understanding of a specific niche domain, the alignment might be suboptimal.
Future work could explore scaling ARRA to even larger "frontier" models or investigating if this alignment strategy can be used for other modalities, like video or 3D generation, where global temporal and spatial coherence are even harder to maintain.
Summary updated