Interpretation of Feature Space using Multi-Channel Attentional Sub-Networks

Masanari Kimura, Masayuki TanakaWorkshops @ CVPR 2019

TL;DR

This paper introduces a novel multi-channel attention mechanism for Convolutional Neural Networks (CNNs) that allows for a more granular interpretation of how different features contribute to recognizing specific facial attributes. By generating attribute-specific attention masks for each feature channel, the authors provide deeper insights into the CNN's internal workings and demonstrate improved interpretability and competitive performance on facial attribute recognition tasks.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) have achieved remarkable success in various computer vision tasks, but their "black box" nature makes it difficult to understand why they make certain predictions. Existing visualization methods often provide a single, aggregated attention map for all features, obscuring the nuanced contributions of individual feature channels to specific predictions. This paper addresses the challenge of interpreting CNNs by developing a method that can reveal which specific feature channels are most important for recognizing particular attributes, thereby enhancing transparency and interpretability.

Methods

The core of this paper is a novel network architecture that incorporates a multi-channel attention mechanism. Instead of a single attention mask for the entire feature map, this approach generates an attention mask for each channel of the feature map, tailored to a specific attribute. This allows the network to learn which feature channels are relevant for, say, detecting "black hair" versus "smiling." The proposed architecture includes:

  • Feature Extractor: A standard component (using Dilation Network) that extracts generic features from the input image.
  • Binary Classifiers with Attention Mechanism: This is the key innovation. For each attribute (e.g., "Smiling"), a binary classifier is trained. Crucially, it uses an attention mask that is element-wise multiplied with the feature map. This mask is attribute-specific and channel-specific, allowing for fine-grained focus. The attention mask is applied as 1 + Mk(ff(xi)) to emphasize important areas while still allowing for low attention values.
  • Multi-Label Classifier: A component to classify multiple labels simultaneously, contributing to better feature representation.
  • Reconstructor: A component that reconstructs the input image from the extracted features, aiming to improve feature representation.

The overall loss function combines losses from these components, with a sparsity penalty (||M||1) on the attention masks to encourage the selection of truly important features.

Datasets

The experiments were conducted using the CelebA dataset, which contains 40 facial attribute labels and over 200,000 images. The authors used 182,637 images for training and 19,962 for testing.

Key Insights and Ideas

  • Attribute-Specific Feature Channel Importance: The most significant insight is that different facial attributes rely on distinct sets of feature channels within the CNN. This is a departure from methods that provide a single, global attention map.
  • Human Intuition Alignment: The attention masks generated by the proposed method align well with human intuition. For instance, the "Mouth Slightly Open" attribute primarily focuses on the mouth region, while "Smiling" involves a broader area including the mouth and eyes.
  • Revealing Attribute Relationships: By analyzing the feature importance across different attributes, the method can reveal correlations and relationships between attributes. For example, certain feature IDs are used for "Smiling" but not for "Mouth Slightly Open," suggesting nuanced differences in their underlying visual cues.
  • Versatile Applications: The ability to understand feature importance for specific attributes opens doors for various applications beyond just interpretation, including improving classification accuracy, network pruning, and image generation.

Key Results

  • High Interpretability: The paper demonstrates that the proposed multi-channel attention mechanism provides high interpretability, allowing humans to gain deeper insights into the CNN's feature space.
  • Competitive Performance: On the CelebA dataset, the proposed method achieved competitive classification accuracy across many attributes, outperforming or matching existing methods like MT-RBM PCA, LNets+ANet, and FaceTracer in terms of average accuracy. For example, for the "Eyeglasses" attribute, the proposed method achieved 98.67% accuracy, surpassing the comparison methods.
  • Visualization of Feature Correlations: Figure 4 visually demonstrates the correlation within the feature space, with deeper colors indicating higher correlation, offering a novel way to explore feature relationships.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the introduction of multiple attention masks per attribute and channel could potentially increase computational overhead.
  • Future Work: The authors suggest that their method is highly versatile and can be applied to a broad range of research, including improving classification accuracy, network pruning, and image generation. The ability to group highly correlated attributes and reveal potential relationships among them is also highlighted as a direction for future exploration.

Summary updated

BibTeX
@inproceedings{kimura2019interpretation,
  author = {Masanari Kimura and Masayuki Tanaka},
  title = {{Interpretation of Feature Space using Multi-Channel Attentional Sub-Networks}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure