Abstract
Vision Transformers (ViTs) have demonstrated state-of-the-art performance in image recognition but often lack the interpretability required for high-stakes applications. Existing "case-based" interpretability methods like Prototypical Part Networks (ProtoPNets), originally designed for CNNs, fail when applied to ViTs due to a "distraction problem" caused by token homogeneity. We present ProtoPFormer, a novel framework that introduces a global-to-local guidance mechanism and a specialized Prototypical Part Concentration (PPC) loss. This approach forces the model to focus on specific, discriminative foreground parts rather than background noise, resulting in a model that is both more accurate and significantly more transparent to human users.
Introduction: The Distraction Problem
In fields like medical diagnosis or autonomous driving, understanding why a model makes a decision is as critical as the decision itself. Prototypical Part Networks (ProtoPNets) address this by classifying images through comparisons to learned "prototypes" (e.g., identifying a bird by its specific wing pattern).
However, transitioning these methods to Vision Transformers (ViTs) reveals a significant hurdle: the distraction problem. Because ViTs utilize global self-attention, internal representations (tokens) tend to become highly homogeneous. Consequently, prototypes often activate on irrelevant background patches like grass or sky rather than the object of interest. ProtoPFormer is designed to restore this focus by enforcing spatial concentration and foreground preservation.
Methodology
ProtoPFormer employs a dual-branch architecture to achieve interpretable, part-based recognition:
- Global & Local Branches: The model bifurcates the ViT's processing. A Global Branch analyzes the class token to grasp the overall object context, while a Local Branch utilizes image tokens to isolate specific discriminative parts.
- Foreground Preserving (FP) Mask: To mitigate background interference, the Global Branch generates a mask via "Attention Rollout." This mask guides the Local Branch to focus exclusively on tokens belonging to the foreground object.
- Prototypical Part Concentration (PPC) Loss: This loss function models prototype activations as a bivariate Gaussian distribution. It aims to minimize variance (forcing prototypes to focus on tight, specific areas) and maximize divergence (ensuring different prototypes identify distinct parts, such as the beak versus the wing).
- Mutual Correction: The final classification is derived from a joint decision where global and local perspectives cross-validate each other.
Experimental Results
Performance and Interpretability
ProtoPFormer was evaluated on fine-grained recognition benchmarks including CUB-200-2011 (birds), Stanford Cars, and Stanford Dogs. Using a DeiT-Ti backbone on the CUB dataset, ProtoPFormer achieved 82.26% accuracy, surpassing both the base ViT (80.57%) and the original ProtoPNet (81.36%).
Qualitatively, the heatmaps produced by ProtoPFormer are significantly cleaner and more localized than those of standard ViTs or CNN-based ProtoPNets. While previous methods often produced scattered activations, ProtoPFormer tightly bounds human-recognizable features like heads, bellies, and wings.
Human Validation (HIVE)
A user study involving 186 participants further validated the model's transparency. When asked to identify a class based solely on the model's prototypes, users succeeded 56.67% of the time with ProtoPFormer, compared to only 8.89% for the original ProtoPNet. This suggests that ProtoPFormer learns features that align much more closely with human visual reasoning.
Conclusion
ProtoPFormer successfully addresses the distraction problem in Vision Transformers by leveraging global context to guide local part discovery. By enforcing prototypical concentration, the framework provides a path toward deploying powerful Transformer architectures in domains where interpretability is a prerequisite. Future work will explore applying this framework to medical imaging and model debugging to further enhance trust in AI-driven decisions.
Summary updated