Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task

Chuanguang Yang, Zhulin An, Xiaolong Hu, Hui Zhu, Yongjun Xu

TL;DR

This paper introduces AnchorNet, a novel CNN architecture that localizes multi-scale informative patches within images and text, making the model's decision-making process interpretable. By using these localized patches for downstream tasks, AnchorNet achieves significant inference acceleration with minimal performance degradation, demonstrating that these patches retain crucial semantic information.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (CNNs) achieve impressive performance but often act as "black boxes," making it difficult to understand why they make a particular prediction. This lack of interpretability is a significant hurdle for critical applications like healthcare and autonomous driving. A key reason for this opacity is the large receptive fields (RFs) of CNNs, which blur the connection between high-level features and specific input regions, and the complex interactions within fully-connected layers. The paper aims to bridge this gap by developing a method to pinpoint and extract the most informative parts of an input (patches) that contribute to the classification decision, thereby enhancing interpretability and enabling more efficient processing.

Methods

The core innovation is AnchorNet, an interpretable CNN framework designed for patch localization. Key methodological aspects include:

  • Carefully Designed Receptive Fields (RFs): AnchorNet uses RFs that are smaller than the input image size, achieved by avoiding padding in convolutional layers. This ensures a direct, interpretable mapping from each spatial location in the feature map back to a specific patch in the input.
  • Linearly Spatial Aggregation: Instead of complex fully-connected layers, AnchorNet employs a simple linear aggregation (average pooling) of class-specific feature maps. This preserves the disentangled nature of patch contributions.
  • Multi-Scale Localization: AnchorNet incorporates three localization branches with different RF sizes. This allows it to adaptively capture informative patches at various scales, from fine-grained details to broader contextual regions.
  • Media-Level Supervision: The localization process is trained using only media-level labels (e.g., image class or sentence sentiment), eliminating the need for expensive bounding box annotations.
  • Attention Branches: Additional attention branches are used to further refine the localization process by emphasizing informative regions. These branches are trained with auxiliary supervision.
  • Localized Patch Selection (LIP): A specific algorithm (Algorithm 1) is used to select informative patches based on activation values and non-overlapping criteria (IoU), ensuring coverage and relevance.

Datasets

  • ImageNet (ILSVRC 2012): Used for training and evaluating image classification and localization.
  • MR (Movie Reviews): Used for training and evaluating text classification and localization.

Key Insights and Ideas

  • Interpretability through RF Control: By carefully controlling RF sizes and avoiding padding, the authors demonstrate a direct link between CNN feature map locations and input image patches, a crucial step towards interpretability.
  • Multi-Scale Patches Capture Diverse Information: The multi-branch approach effectively captures both coarse-grained global features (larger RFs) and fine-grained local details (smaller RFs), which are essential for comprehensive understanding.
  • Localized Patches Enable Efficient Inference: Using only the identified informative patches for downstream classification significantly reduces computational load (FLOPs) with minimal accuracy loss, suggesting a path towards faster AI models.
  • Misclassification Analysis: The framework allows for analyzing why a misclassification occurred by examining the localized patches that contributed to the incorrect decision, offering valuable debugging insights.
  • Robustness to Texture Synthesis: The method shows robustness to texture-synthesized images, where human recognition suffers greatly, highlighting the power of learned semantic features over raw visual texture.

Key Results

  • Image Localization Performance: AnchorNet-I's localization performance on ImageNet is comparable to object detection methods, despite using only classification supervision.
  • Text Localization Performance: AnchorNet-T effectively localizes informative text segments for sentiment recognition.
  • Inference Acceleration: Replacing original inputs with localized patches for downstream classification (e.g., ResNet-50, ResNeXt-50, DenseNet-169) resulted in approximately a 2x acceleration in FLOPs with only a small drop in top-1 accuracy (e.g., 1.5% to 3.3% on ImageNet).
  • Comparison with Saccader: AnchorNet-I outperforms the state-of-the-art Saccader in terms of classification accuracy for a given patch coverage and is significantly more computationally efficient (1.6M parameters, 0.5G FLOPs vs. 33.6M parameters, 21.6G FLOPs).
  • Robustness to Adversarial Attacks: AnchorNet shows good robustness to adversarial attacks (FGSM), with downstream classifier accuracy dropping only slightly compared to a significant drop for a vanilla CNN.

Limitations and Future Work

  • Limitations:
    • While the paper demonstrates effectiveness, the choice of hyperparameters for the LIP algorithm (percentage of coverage, IoU threshold) can influence the trade-off between accuracy and FLOPs, requiring careful tuning.
  • The paper notes that for some cases, particularly with narrow RF branches, localized patches might include unrelated background noise, potentially affecting classification.
  • Future Work:
    • The authors suggest that the AnchorNet framework can be extended to other media types beyond images and text, such as audio and speech recognition.
  • Further exploration of the hyperparameter sensitivity of the LIP algorithm for optimal trade-offs is suggested.
  • The framework's potential for inspiring future research in interpretable feature localization and network acceleration is highlighted.

Summary updated

Abstract

Deep convolutional neural networks (CNN) always depend on wider receptive field (RF) and more complex non-linearity to achieve state-of-the-art performance, while suffering the increased difficult to interpret how relevant patches contribute the final prediction. In this paper, we construct an interpretable AnchorNet equipped with our carefully designed RFs and linearly spatial aggregation to provide patch-wise interpretability of the input media meanwhile localizing multi-scale informative patches only supervised on media-level labels without any extra bounding box annotations. Visualization of localized informative image and text patches show the superior multi-scale localization capability of AnchorNet. We further use localized patches for downstream classification tasks across widely applied networks. Experimental results demonstrate that replacing the original inputs with their patches for classification can get a clear inference acceleration with only tiny performance degradation, which proves that localized patches can indeed retain the most semantics and evidences of the original inputs.
BibTeX
@misc{yang2020localizing,
  author = {Chuanguang Yang and Zhulin An and Xiaolong Hu and Hui Zhu and Yongjun Xu},
  title = {{Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task}},
  year = {2020},
  eprint = {2002.03737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03737},
}
Figure