Discrete Representations Strengthen Vision Transformer Robustness

Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, Irfan Essa

TL;DR

While Vision Transformers (ViTs) are powerful, they are often "texture-biased," making them fragile when faced with real-world distribution shifts. By augmenting the input layer with discrete tokens (visual words) from a vector-quantized encoder, the authors force the model to prioritize global shape over local texture. This simple architectural change boosts robustness by up to 12% across major benchmarks without sacrificing standard accuracy.

The Problem: Texture Bias in Vision Transformers

Deep neural networks are notorious for "shortcut learning." Instead of learning that a "cat" has a specific shape and structure, they often learn to identify a cat based on the specific texture of its fur. This works fine on standard datasets like ImageNet, but the model breaks the moment it sees a cat in a different environment—like a sketch, a stylized painting, or a low-resolution photo.

Vision Transformers (ViTs) were initially thought to be more robust than Convolutional Neural Networks (CNNs) due to their global self-attention mechanism. However, this paper reveals that ViTs are still overly reliant on high-dimensional, individually informative pixel patches. Because these continuous patches contain so much local detail, the model can minimize training error using local "nuisance" signals (textures) rather than learning the invariant global structures (shapes) that humans use for recognition.

The Solution: Dr. ViT Architecture

To address this deficiency, the authors introduce Dr. ViT (Discrete representation ViT), which modifies the standard ViT input pipeline to incorporate discrete representations.

Dr. ViT Architecture
  1. Discrete Tokenization: Instead of just linearly projecting raw pixels into embeddings, Dr. ViT uses a pre-trained Vector Quantized (VQ) encoder (similar to VQ-VAE or VQ-GAN). This encoder maps an image patch to a discrete "visual word" in a learned codebook.
  2. Information Bottleneck: Because a discrete token represents a cluster of similar patches, it acts as an information bottleneck. It discards fine-grained, high-frequency texture information and retains the essential structural category.
  3. Hybrid Input: To ensure the model doesn't lose too much detail (which might hurt performance on small objects), the architecture concatenates a low-dimensional version of the original continuous pixel embedding with the discrete token embedding.
  4. Two-Stage Training: First, the VQ-model is pre-trained on unlabelled data to learn the codebook. Second, the ViT is trained for classification using the hybrid discrete-continuous tokens as input.

Experimental Setup & Datasets

The models were trained on ImageNet-1K and ImageNet-21K. To rigorously test robustness, the authors evaluated the models across seven out-of-distribution (OOD) benchmarks:

  • Stylized-ImageNet: Images where textures are replaced by artistic styles (tests shape bias).
  • ImageNet-Sketch: Black and white sketches.
  • ImageNet-C: Images with common corruptions like fog, snow, or blur.
  • ImageNet-R & ImageNet-A: Renditions (art/cartoons) and "Natural Adversarial" examples.
  • ObjectNet: Real-world photos with unusual backgrounds and rotations.
  • ImageNet-V2: A new test set to check for overfitting to the original ImageNet.

Key Insights & Results

  • Discrete is Robust: Unlike continuous pixel values, discrete tokens are invariant to small perturbations. If you add a bit of noise to a patch, it likely still maps to the same "visual word," making the model naturally more stable.
  • Forcing Global Context: Because an individual discrete token carries less information than a raw pixel patch, the Transformer must look at the relationships between tokens (the global context) to make an accurate prediction. This shifts the model's "inductive bias" toward shape.
  • Human-like Shape Bias: The authors found that Dr. ViT has a "shape bias" of 0.62—significantly higher than vanilla ViT (0.42) or ResNet (0.20), moving much closer to the human average of 0.96.
  • Robustness Gains: Dr. ViT improved OOD performance across the board. Most notably, it achieved a 12% boost on Stylized-ImageNet and a 10% boost on ImageNet-Sketch and ImageNet-C.
  • No "Robustness Tax": Often, making a model robust makes it less accurate on clean data. Dr. ViT maintains (and sometimes slightly improves) standard ImageNet accuracy.
  • State-of-the-Art: The method outperformed specialized data augmentation techniques like CutMix and Mixup on robustness tasks, despite being a much simpler architectural change.

Limitations & Future Directions

While highly effective, the approach has a few limitations. The quantization process can sometimes "hallucinate" or smooth over fine details, causing the model to struggle with text, human faces, parallel lines, and very small objects. Additionally, the two-stage training process is more computationally intensive than standard training.

However, the generative potential of this approach is exciting. Since discrete tokens are also the foundation of high-end image generation models (like DALL-E), the authors suggest this work could pave the way for a "unified" Transformer architecture that excels at both visual recognition and image generation.

Summary updated

Abstract

Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs are overly reliant on local features (\eg, nuisances and texture) and fail to make adequate use of global context (\eg, shape and structure). As a result, ViTs fail to generalize to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12\% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.
BibTeX
@inproceedings{mao2022discrete,
  author = {Chengzhi Mao and Lu Jiang and Mostafa Dehghani and Carl Vondrick and Rahul Sukthankar and Irfan Essa},
  title = {{Discrete Representations Strengthen Vision Transformer Robustness}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=8hWs60AZcWk},
}
Figure