Few-shot Learning by Exploiting Visual Concepts within CNNs

Boyang Deng, Qing Liu, Siyuan Qiao, Alan Yuille

Abstract

Convolutional neural networks (CNNs) are one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes hard to obtain, and it is desirable to learn to recognize objects from few examples. In this work, we address these limitations of CNNs by developing novel, flexible, and interpretable models for few-shot learning. Our models are based on the idea of encoding objects in terms of visual concepts (VCs), which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of VCs to the few-shot setting, and then uncover two key properties of feature encoding using VCs, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using VCs helps expose the natural capability of CNNs for few-shot learning.

Introduction

Deep learning, specifically Convolutional Neural Networks (CNNs), typically suffers from two major bottlenecks: extreme data hunger and the "black box" problem. Standard CNNs require thousands of labeled examples to learn new categories, and it is notoriously difficult to understand the logic behind their decisions. Few-shot learning (FSL) aims to recognize new objects from just one or five examples, but many existing methods like MAML or MatchingNet remain opaque. This paper addresses both issues by leveraging the internal representations of CNNs to identify human-understandable "parts"—Visual Concepts (VCs)—that can be reused for rapid learning of new categories.

Methodology: Visual Concept Encoding

The core of this approach is the extraction and utilization of Visual Concepts (VCs). Instead of using raw, high-dimensional CNN features, the authors cluster intermediate feature vectors (from the Pool-4 layer of a VGG-16) using a von Mises-Fisher mixture model (vMFM). This treats feature vectors as points on a unit sphere, which proves more effective than standard K-means for normalized neural network activations.

Visual Concept (VC) representing train and bus window segments across different images.

Once VCs are extracted, an image is transformed into a set of binary maps. If a specific VC (e.g., a window segment) is detected at a spatial location with high confidence, that bit is flipped to 1. Based on this encoding, two models were developed:

  1. VC-NN (Nearest Neighbor): A template-matching approach using a "fuzzy" similarity metric that allows for small spatial shifts.
  2. VC-LH (Factorizable Likelihood): A probabilistic model based on Naive Bayes that learns the likelihood of seeing specific VCs at specific locations for a given category.

Key Insights and Interpretability

A major advantage of this framework is its inherent transparency. Because VCs represent semantic parts, researchers can visualize exactly which components of an image triggered a classification.

Visual Concept (VC) capturing semantic parts of sofas, such as cushions and armrests.

Beyond interpretability, the research uncovered two critical properties:

  • Category Sensitivity: Unsupervised VCs naturally align with specific object classes (e.g., a "wing" VC fires for airplanes but not cars).
  • Spatial Patterns: The spatial arrangement of VCs is as important as their presence, allowing the models to distinguish between objects with similar parts but different structures.

Experimental Results

The models were evaluated on Mini-ImageNet and PASCAL3D+. On the standard 5-shot Mini-ImageNet task, the VC-LH model achieved 63.07% accuracy, performing on par with the state-of-the-art MAML (63.11%) at the time.

Furthermore, the VC-based models demonstrated superior flexibility. Unlike many meta-learning algorithms that are hard-coded for specific N-way K-shot scenarios, these models are task-agnostic. They significantly outperformed baselines in "cross-scenario" tests where the number of categories changed, proving that relying on stable semantic parts improves generalization and robustness.

Conclusion

By distilling CNN features into interpretable Visual Concepts, this work bridges the gap between high-performance few-shot learning and human-understandable AI. While the current models use relatively simple spatial logic, the VC-encoding framework provides a foundation for more sophisticated spatial modeling and applications in object detection and occlusion handling.

Summary updated

Abstract

Convolutional neural networks (CNNs) are one of the driving forces for the advancement of computer vision. Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence. One is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts of annotated data, which is sometimes hard to obtain, and it is desirable to learn to recognize objects from few examples. In this work, we address these limitations of CNNs by developing novel, flexible, and interpretable models for few-shot learning. Our models are based on the idea of encoding objects in terms of visual concepts (VCs), which are interpretable visual cues represented by the feature vectors within CNNs. We first adapt the learning of VCs to the few-shot setting, and then uncover two key properties of feature encoding using VCs, which we call category sensitivity and spatial pattern. Motivated by these properties, we present two intuitive models for the problem of few-shot learning. Experiments show that our models achieve competitive performances, while being more flexible and interpretable than alternative state-of-the-art few-shot learning methods. We conclude that using VCs helps expose the natural capability of CNNs for few-shot learning.
BibTeX
@misc{deng2017few,
  author = {Boyang Deng and Qing Liu and Siyuan Qiao and Alan Yuille},
  title = {{Few-shot Learning by Exploiting Visual Concepts within CNNs}},
  year = {2017},
  eprint = {1711.08277},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.08277},
}
Figure