Introduction
Vision Transformers (ViTs) have revolutionized computer vision but suffer from high computational costs that scale quadratically with the number of input patches. While image redundancy reduction—the process of discarding less informative patches—offers a path toward efficiency, existing methods often rely on supervised signals. This reliance introduces "categorical bias," where models prioritize regions associated with known training labels while discarding potentially critical but unlabeled information.
Researchers from Fudan University and the University of Maryland have introduced Learning to Rank Patches (LTRP), a self-supervised framework designed to identify and preserve informative image regions without human-provided labels. By reframing redundancy reduction as a ranking problem, LTRP achieves significant speedups with minimal impact on accuracy across diverse datasets.
The Challenge of Categorical Bias
Most state-of-the-art patch pruning methods are trained to retain patches that contribute most to a classification decision (e.g., identifying a "dog"). However, this creates a narrow focus. In real-world applications, a model might encounter objects it wasn't specifically trained to recognize—such as a rare tool or a specific traffic sign. Supervised models often treat these "unseen" objects as redundant background, leading to poor performance in open-world or multi-label scenarios. LTRP addresses this by seeking a "fair" way to judge informativeness regardless of whether a model knows the name of the object.
The LTRP Framework
LTRP moves away from binary "keep or drop" classification, instead learning the relative importance of every patch through a self-supervised pipeline.
1. Quantifying Semantic Density
The authors observed that Masked Autoencoders (MAE) become highly sensitive to individual patches when the masking ratio is extremely high (e.g., 90%). At this threshold, the reconstruction becomes fragile; removing a single informative patch significantly degrades the output quality. LTRP leverages this by calculating a "semantic density score" for each patch, measured by the $\ell_1$ distance between reconstructions with and without that specific patch visible.
2. Learning to Rank
To avoid the high computational cost of running a full MAE reconstruction during inference, the authors train a lightweight Ranking Model (a small ViT or CNN). This model is trained using a List-wise loss (ListMLE) to predict the relative importance order of patches based on the pseudo-labels generated by the frozen MAE. This approach allows the model to learn the structural nuances of images rather than just identifying specific object categories.
3. Representative Selection via DPC-KNN
Ranking models naturally gravitate toward high-contrast edges and boundaries. To ensure the model doesn't ignore "homogeneous" but structurally important regions (like the smooth surface of an object), LTRP incorporates a clustering step called DPC-KNN. This ensures a diverse selection of patches that maintain the global context of the image.
Experimental Results
The framework was evaluated on ImageNet-1K, MS-COCO, PASCAL VOC, and ADE20K to test its efficiency and lack of bias.
- Unbiased Performance: In tests on PASCAL VOC and MS-COCO involving categories not present in the training set, LTRP significantly outperformed supervised methods like Grad-CAM and EViT. While supervised methods saw a sharp drop in precision when removing patches, LTRP maintained high performance, confirming its ability to recognize "importance" independent of labels.
- Efficiency and Accuracy: When integrated into standard architectures (ViT-B, ViT-L, and ViT-H), LTRP provided a 1.19x to 1.36x GPU speedup. This was achieved with a negligible accuracy loss of only 0.1% to 0.5%.
- Multi-label Success: On the MS-COCO dataset, LTRP achieved 81.8% mAP at a 0.5 keep ratio, surpassing previous unsupervised benchmarks by 1.4%.
Limitations and Future Work
While LTRP offers a robust solution for inference, it requires a computationally intensive offline phase to generate pseudo-labels via the large MAE. Additionally, the ranking model's quality is inherently linked to the pre-trained MAE's ability to reconstruct specific textures. Future research may extend this "learning to rank" logic to video redundancy reduction or multimodal tasks where importance is defined by both textual and visual cues.
Conclusion
LTRP represents a significant step toward more robust and efficient vision models. By utilizing self-supervised reconstruction sensitivity to rank patch importance, it overcomes the categorical biases inherent in supervised pruning. The result is a framework that provides significant computational savings while remaining "fair" to the diverse visual information found in real-world images.
Summary updated