Denoising Autoregressive Representation Learning

Yazhe Li, Jörg Bornschein, Ting Chen

Abstract

In this paper, we explore a new generative approach for learning visual representations. Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We find that training with Mean Squared Error (MSE) alone leads to strong representations. To enhance the image generation ability, we replace the MSE loss with the diffusion objective by using a denoising patch decoder. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models. Notably, the optimal schedule differs significantly from the typical ones used in standard image diffusion models. Overall, despite its simple architecture, DARL delivers performance remarkably close to state-of-the-art masked prediction models under the fine-tuning protocol. This marks an important step towards a unified model capable of both visual perception and generation, effectively combining the strengths of autoregressive and denoising diffusion models.

Introduction

In Natural Language Processing, the "next-token prediction" paradigm has successfully unified representation learning and generation. However, Computer Vision has traditionally seen these goals pursued separately. Methods like Masked Image Modeling (MIM) or Contrastive Learning excel at feature extraction but lack generative capabilities. Conversely, Diffusion Models are generative powerhouses but often produce features that are less effective for downstream classification tasks. Denoising Autoregressive Representation Learning (DARL) bridges this gap by training a decoder-only Transformer to predict image patches using a diffusion-based objective.

Methodology

DARL utilizes a straightforward decoder-only Vision Transformer (ViT) architecture. Images are decomposed into patches and processed in a "raster scan" order. Causal masking is applied to ensure each patch only attends to preceding patches, mimicking the autoregressive nature of LLMs.

The DARL architecture: a decoder-only Transformer predicts patches autoregressively using a denoising objective.

Key components of the methodology include:

  • 2D Rotary Positional Embedding (2D RoPE): This provides a strong inductive bias for spatial coordinates, outperforming standard absolute embeddings in causal vision tasks.
  • Denoising Patch Decoder: Instead of simple pixel prediction via MSE, DARL uses a denoising decoder. This component takes a noisy version of the target patch and reconstructs the clean version, conditioned on the features from the causal transformer.
  • Flexible Noise Scheduling: Noise levels are sampled from a Beta distribution during training, allowing the model to balance high-level abstraction and low-level detail.

Key Insights and Noise Scheduling

A critical finding in DARL is the "Conflict of Objectives": the optimal noise schedule for learning high-quality features differs significantly from the schedules typically used for image generation. To learn robust representations, the model benefits from working on highly noisy patches, which forces it to capture high-level semantic abstractions.

Analysis of noise schedule parameters (a, b) on representation learning performance across different patch sizes.

As shown in the parameter sweeps for the Beta distribution (parameters $a$ and $b$), certain configurations lead to significantly better downstream performance. Furthermore, the authors found that while complex patch orderings are possible, a simple raster scan is near-optimal when combined with 2D RoPE.

Experimental Results

DARL was evaluated primarily on ImageNet-1K and the Visual Task Adaptation Benchmark (VTAB).

  • Competitive Representation: Using a ViT-H backbone, DARL achieved 85.9% top-1 accuracy on ImageNet-1K. This is within 1% of the state-of-the-art Masked Autoencoder (MAE) at 86.9%, despite DARL's additional generative capabilities.
  • Generative Capability: Unlike MIM models, DARL can perform high-quality image completion, generating multiple plausible continuations for a given image prompt.
  • Scaling: The diffusion objective shows superior scaling properties compared to MSE as model size and training duration increase.

Discussion and Limitations

While DARL represents a significant step toward unified vision models, a small performance gap remains compared to bidirectional masked models. This may be due to the inherent difficulty of causal prediction versus "filling in the blanks." Additionally, while the model excels at natural and specialized tasks, it shows some weakness in structured tasks like counting, suggesting that causal ordering might influence specific types of spatial reasoning. Future work will explore whether increasing model capacity beyond ViT-H can further resolve the trade-offs between understanding and generation.

Summary updated

Abstract

In this paper, we explore a new generative approach for learning visual representations. Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We find that training with Mean Squared Error (MSE) alone leads to strong representations. To enhance the image generation ability, we replace the MSE loss with the diffusion objective by using a denoising patch decoder. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models. Notably, the optimal schedule differs significantly from the typical ones used in standard image diffusion models. Overall, despite its simple architecture, DARL delivers performance remarkably close to state-of-the-art masked prediction models under the fine-tuning protocol. This marks an important step towards a unified model capable of both visual perception and generation, effectively combining the strengths of autoregressive and denoising diffusion models.
BibTeX
@inproceedings{li2024denoising,
  author = {Yazhe Li and Jörg Bornschein and Ting Chen},
  title = {{Denoising Autoregressive Representation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=dW29JZj0G5},
}
Figure