IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogério Feris, Aude Oliva

Introduction

Vision Transformers (ViTs) have revolutionized computer vision, but their performance comes with a steep computational tax. The core self-attention mechanism computes relationships between every single pair of image patches, leading to computational complexity that scales quadratically with the number of patches. However, much of this computation is inherently wasted on redundant visual information—such as an empty sky behind a bird or a static background in a video.

Furthermore, standard ViTs often suffer from "vague interpretability." Their internal attention maps do not consistently align with what humans consider the most important parts of an image. Existing compression methods frequently exacerbate this issue by creating opaque architectures that are even harder to interpret. To address these dual challenges, researchers introduce IA-RED² (Interpretability-Aware Redundancy Reduction), a novel framework that makes ViTs faster and more transparent by dynamically dropping uninformative image patches during inference.

Methodology: The Multi-head Interpreter

The core innovation of IA-RED² is the Multi-head Interpreter. Rather than processing every single patch through every layer of the Transformer, the architecture divides the Transformer blocks into hierarchical groups and evaluates patch importance dynamically.

IA-RED² dynamically selects informative patches in images and videos
  1. The Interpreter Module: Before each group of Transformer blocks, a lightweight interpreter module utilizes a learned policy token to score the importance of each input patch.
  2. Dynamic Dropping: If a patch’s score falls below a predefined threshold, it is immediately discarded. It never reaches subsequent layers, significantly saving both memory and floating-point operations (FLOPs).
  3. Hierarchical Pruning: The dropping mechanism occurs in stages. Early stages typically discard broad, uninformative background patches, while later stages refine the focus, keeping only the most critical part-level features (e.g., the eyes or beak of an animal).
  4. Curriculum Training: Because the discrete "drop or keep" decision is non-differentiable, the authors employ Reinforcement Learning (specifically, the REINFORCE algorithm). The model is rewarded for maintaining high accuracy while utilizing as few patches as possible. A curriculum learning approach is used to train the interpreters stage-by-stage, ensuring training stability.

Unlike static pruning methods, IA-RED² is entirely input-dependent. It dynamically determines the computational budget based on the specific complexity of the current image or video frame.

Experimental Setup and Key Results

The framework was rigorously evaluated across three major benchmarks: ImageNet-1k for image classification, Kinetics-400 for video action recognition (where temporal redundancy is exceptionally high), and ImageNet-Segmentation to quantitatively assess model interpretability via weakly-supervised segmentation.

Speed and Accuracy Trade-offs

IA-RED² demonstrates exceptional efficiency gains. On ImageNet, the method achieves a 1.4x speedup (measured in frames per second) for DeiT-S and DeiT-B models, sacrificing less than 0.7% in accuracy. The gains are even more pronounced in video tasks. For the TimeSformer model evaluated on Kinetics-400, IA-RED² achieved a massive 4x speedup. This is primarily because the model rapidly learns to ignore static background patches that persist across multiple frames.

Efficiency Breeds Interpretability

A striking insight from this research is that efficiency and interpretability are not mutually exclusive; in fact, they can be synergistic. By forcing the model to discard data to save computational energy, it is compelled to learn what truly matters for the prediction.

The resulting "keep" maps naturally form high-quality saliency maps. In segmentation tasks, IA-RED² achieved a 49.42 mIoU, significantly outperforming the raw attention maps of standard ViTs (46.37 mIoU) and other popular interpretability tools like GradCAM or LIME. Because the model literally cannot see the dropped patches, researchers can be certain that the remaining patches are the sole drivers of the final prediction, offering a highly faithful explanation.

Furthermore, the authors demonstrated that this "data-level" pruning is orthogonal to "weight-level" pruning (removing neural network connections), suggesting the two optimization strategies can be combined for even greater efficiency.

Limitations and Future Directions

While highly effective, IA-RED² has a few limitations:

  • Training Complexity: The reliance on multi-stage curriculum training and the REINFORCE algorithm makes the training pipeline more complex and potentially slower than standard fine-tuning approaches.
  • Hardware Specialization: Unstructured patch dropping can sometimes be challenging to optimize for certain hardware accelerators that favor dense, predictable matrix computations.
  • Hyperparameter Sensitivity: The number of hierarchical groups and the penalty coefficient for keeping patches are manual hyperparameters that may require careful tuning when adapting the framework to niche datasets.

Future work aims to make the framework more "plug-and-play" for a wider variety of transformer-based architectures, including those utilized in complex multi-modal tasks.

Summary updated

Abstract

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED$^2$). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.
BibTeX
@inproceedings{pan2021ia,
  author = {Bowen Pan and Rameswar Panda and Yifan Jiang and Zhangyang Wang and Rogério Feris and Aude Oliva},
  title = {{IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=9vhkn0jnR6},
}
Figure