Unleashing the Potential of CNNs for Interpretable Few-Shot Learning

Boyang Deng, Qing Liu, Siyuan Qiao, Alan L. Yuille

Abstract

Convolutional neural networks (CNNs) are one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes hard to obtain, and it is desirable to learn to recognize objects from few examples. In this work, we address these limitations of CNNs by developing novel, flexible, and interpretable models for few-shot learning. Our models are based on the idea of encoding objects in terms of visual concepts (VCs), which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of VCs to the few-shot setting, and then uncover two key properties of feature encoding using VCs, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using VCs helps expose the natural capability of CNNs for few-shot learning.

Introduction

Standard Convolutional Neural Networks (CNNs) are notoriously data-hungry and difficult to interpret. While they excel at classification when given thousands of examples, they struggle with few-shot learning—the ability to recognize a new category from just one or five examples. Existing few-shot methods often rely on complex "meta-learning" or metric-learning schemes that are tailored to specific scenarios, lacking flexibility and interpretability. This research proposes that an ideal system should be interpretable (explaining why a classification was made) and flexible (handling varying tasks without extensive retraining).

Methodology: Visual Concepts (VCs)

The core innovation is the use of Visual Concepts (VCs) as a bridge between raw pixels and high-level categories. VCs are interpretable visual cues represented by feature vectors within CNNs.

Extracting and Encoding VCs

The process begins with a pre-trained CNN (e.g., VGG-13). Intermediate feature vectors from layers like Pool-3 or Pool-4 are clustered using a von Mises-Fisher mixture model. Each cluster center represents a "Visual Concept"—essentially a template for a specific visual part, such as a wheel or a window.

Visual Concept VC_174 representing a bicycle wheel across various instances.

An image is then converted into a VC-Encoding, a binary map where a "1" indicates the presence of a specific VC at a specific spatial location.

Proposed Models

Two intuitive models are developed based on these encodings:

  1. VC-NN (Nearest Neighbor): A template-matching approach using a "fuzzy" similarity metric. This allows for small spatial shifts, recognizing that parts like wheels may appear in slightly different positions across images.
  2. VC-LH (Factorizable Likelihood): A probabilistic model based on Naive Bayes that learns the likelihood of specific VCs appearing at specific locations for a given category.

Experiments and Results

The models were evaluated on two primary benchmarks: Mini-ImageNet (100 categories) and PASCAL3D+ (12 rigid object categories).

Examples of object categories from the datasets used for evaluation.

On Mini-ImageNet, the proposed models achieved competitive results:

  • 5-way 5-shot: The VC-LH model reached 63.07% accuracy, comparable to the state-of-the-art MAML (63.11%).
  • 5-way 1-shot: The VC-NN model achieved 46.39%, outperforming baselines like MatchingNet (43.56%).

On PASCAL3D+, the VC-based models consistently outperformed raw feature nearest-neighbor baselines, demonstrating that part-based decomposition significantly aids generalization from limited data.

Discussion

The research uncovers two key properties of VC encoding:

  • Category Sensitivity: Unsupervised VCs naturally align with categories; certain VCs fire frequently for specific objects (e.g., airplanes) and rarely for others.
  • Spatial Patterns: The arrangement of parts is crucial. A car is defined not just by the presence of wheels, but by their specific spatial configuration.

Unlike meta-learning algorithms, these models are highly flexible and can be applied to tasks with varying numbers of categories without retraining. Furthermore, they provide inherent interpretability by exposing the internal features CNNs naturally learn.

Conclusion

By leveraging Visual Concepts, this work demonstrates that standard CNNs possess a natural capability for few-shot learning. The resulting models are not only competitive in performance but also offer superior flexibility and interpretability compared to traditional "black-box" approaches. Future work will focus on improving VC extraction and extending the framework to tasks like few-shot object detection.

Summary updated

Abstract

Convolutional neural networks (CNNs) are one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes hard to obtain, and it is desirable to learn to recognize objects from few examples. In this work, we address these limitations of CNNs by developing novel, flexible, and interpretable models for few-shot learning. Our models are based on the idea of encoding objects in terms of visual concepts (VCs), which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of VCs to the few-shot setting, and then uncover two key properties of feature encoding using VCs, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using VCs helps expose the natural capability of CNNs for few-shot learning.
BibTeX
@misc{deng2017unleashing,
  author = {Boyang Deng and Qing Liu and Siyuan Qiao and Alan L. Yuille},
  title = {{Unleashing the Potential of CNNs for Interpretable Few-Shot Learning}},
  year = {2017},
  eprint = {1711.08277},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.08277},
}
Figure