In the world of artificial intelligence, language models like GPT possess a "unified" superpower: the same training objective—predicting the next word—makes them excellent at both understanding language and generating it. In computer vision, however, these two worlds are often split. Methods like Masked Autoencoders (MAE) or Contrastive Learning (DINO) are fantastic at "perception" (learning features for classification or detection) but cannot generate new images. Conversely, Diffusion models are the kings of generation, but their internal features aren't always the most efficient for downstream tasks.
Can we build a single, simple vision model that excels at both, just like we do in NLP? The paper introduces DARL (Denoising Autoregressive Representation Learning), a unified framework that bridges the gap between image generation and visual perception.
The DARL Architecture
DARL treats an image as a sequence of patches (similar to words in a sentence) and predicts them one by one autoregressively.
Key technical components of the framework include:
- Decoder-Only Transformer: DARL utilizes a standard "decoder-only" Vision Transformer (ViT). Unlike the bidirectional attention in MAE, DARL uses causal masking, meaning each patch can only "see" the patches that came before it in a raster-scan (left-to-right, top-to-bottom) order.
- 2D Rotary Positional Embeddings (2D RoPE): Instead of standard absolute positions, the authors adapted RoPE (popular in LLMs) to 2D. This provides a strong inductive bias for spatial relationships, which is crucial for causal vision models.
- Denoising Objective: While the model can be trained to predict the next patch exactly using Mean Squared Error (MSE), replacing this with a diffusion objective—where the model predicts a clean patch from a noisy version—significantly boosts both generation quality and representation strength in larger models.
Tailoring Noise for Representation
There is a fundamental tension between learning high-level features (good for classification) and low-level details (good for generation). The authors discovered that the "optimal" noise schedule for learning representations differs from the one used for generating purely aesthetic pictures.
For representation learning, the model benefits from a noise schedule that "destroys" more of the image, forcing it to rely on global context and learn high-level abstractions rather than memorizing local details.
Key Results and Insights
The researchers evaluated DARL primarily on ImageNet-1K for pre-training and evaluation, and tested transfer learning on the VTAB benchmark and COCO.
- Closing the Gap: DARL achieved an 85.9% top-1 accuracy on ImageNet-1K fine-tuning with a ViT-Huge backbone. This is remarkably close to the state-of-the-art MAE (86.9%), despite the inherent "handicap" of causal masking.
- Superior Transfer: On the VTAB benchmark, DARL outperformed supervised pre-training on 10 out of 11 "Natural" and "Specialized" tasks, showing that its generative pre-training produces highly versatile features.
- Generative Capability: Unlike its competitors in the representation learning space (MAE, DINO), DARL can perform image completion. When given the top half of an image, it can generate multiple plausible, high-quality completions for the bottom half.
- Scaling Wins: While MSE loss is strong for smaller models, the diffusion objective scales better. As the model size and training time increase, the denoising approach pulls ahead.
Limitations and Future Work
Despite its success, there is still a minor performance gap (about 1%) between DARL and the best masked prediction models. The authors hypothesize this might be due to the "causal" constraint, which is less efficient for vision than bidirectional attention. Furthermore, the model still has to "choose" between being a great perceiver and a great generator due to capacity competition.
Future work could explore if scaling to billions of parameters can eliminate this trade-off, or if alternative "vision-native" patch orderings (like spiral or quad-tree) can outperform the standard raster scan.
Summary updated