TL;DR
While Vision Transformers (ViTs) are powerful, they are often "texture-biased," making them fragile when faced with real-world distribution shifts. By augmenting the input layer with discrete tokens (visual words) from a vector-quantized encoder, the authors force the model to prioritize global shape over local texture. This simple architectural change boosts robustness by up to 12% across major benchmarks without sacrificing standard accuracy.
The Problem: Texture Bias in Vision Transformers
Deep neural networks are notorious for "shortcut learning." Instead of learning that a "cat" has a specific shape and structure, they often learn to identify a cat based on the specific texture of its fur. This works fine on standard datasets like ImageNet, but the model breaks the moment it sees a cat in a different environment—like a sketch, a stylized painting, or a low-resolution photo.
Vision Transformers (ViTs) were initially thought to be more robust than Convolutional Neural Networks (CNNs) due to their global self-attention mechanism. However, this paper reveals that ViTs are still overly reliant on high-dimensional, individually informative pixel patches. Because these continuous patches contain so much local detail, the model can minimize training error using local "nuisance" signals (textures) rather than learning the invariant global structures (shapes) that humans use for recognition.
The Solution: Dr. ViT Architecture
To address this deficiency, the authors introduce Dr. ViT (Discrete representation ViT), which modifies the standard ViT input pipeline to incorporate discrete representations.
- Discrete Tokenization: Instead of just linearly projecting raw pixels into embeddings, Dr. ViT uses a pre-trained Vector Quantized (VQ) encoder (similar to VQ-VAE or VQ-GAN). This encoder maps an image patch to a discrete "visual word" in a learned codebook.
- Information Bottleneck: Because a discrete token represents a cluster of similar patches, it acts as an information bottleneck. It discards fine-grained, high-frequency texture information and retains the essential structural category.
- Hybrid Input: To ensure the model doesn't lose too much detail (which might hurt performance on small objects), the architecture concatenates a low-dimensional version of the original continuous pixel embedding with the discrete token embedding.
- Two-Stage Training: First, the VQ-model is pre-trained on unlabelled data to learn the codebook. Second, the ViT is trained for classification using the hybrid discrete-continuous tokens as input.
Experimental Setup & Datasets
The models were trained on ImageNet-1K and ImageNet-21K. To rigorously test robustness, the authors evaluated the models across seven out-of-distribution (OOD) benchmarks:
- Stylized-ImageNet: Images where textures are replaced by artistic styles (tests shape bias).
- ImageNet-Sketch: Black and white sketches.
- ImageNet-C: Images with common corruptions like fog, snow, or blur.
- ImageNet-R & ImageNet-A: Renditions (art/cartoons) and "Natural Adversarial" examples.
- ObjectNet: Real-world photos with unusual backgrounds and rotations.
- ImageNet-V2: A new test set to check for overfitting to the original ImageNet.
Key Insights & Results
- Discrete is Robust: Unlike continuous pixel values, discrete tokens are invariant to small perturbations. If you add a bit of noise to a patch, it likely still maps to the same "visual word," making the model naturally more stable.
- Forcing Global Context: Because an individual discrete token carries less information than a raw pixel patch, the Transformer must look at the relationships between tokens (the global context) to make an accurate prediction. This shifts the model's "inductive bias" toward shape.
- Human-like Shape Bias: The authors found that Dr. ViT has a "shape bias" of 0.62—significantly higher than vanilla ViT (0.42) or ResNet (0.20), moving much closer to the human average of 0.96.
- Robustness Gains: Dr. ViT improved OOD performance across the board. Most notably, it achieved a 12% boost on Stylized-ImageNet and a 10% boost on ImageNet-Sketch and ImageNet-C.
- No "Robustness Tax": Often, making a model robust makes it less accurate on clean data. Dr. ViT maintains (and sometimes slightly improves) standard ImageNet accuracy.
- State-of-the-Art: The method outperformed specialized data augmentation techniques like CutMix and Mixup on robustness tasks, despite being a much simpler architectural change.
Limitations & Future Directions
While highly effective, the approach has a few limitations. The quantization process can sometimes "hallucinate" or smooth over fine details, causing the model to struggle with text, human faces, parallel lines, and very small objects. Additionally, the two-stage training process is more computationally intensive than standard training.
However, the generative potential of this approach is exciting. Since discrete tokens are also the foundation of high-end image generation models (like DALL-E), the authors suggest this work could pave the way for a "unified" Transformer architecture that excels at both visual recognition and image generation.
Summary updated