Few-Shot Learning With Geometric Constraints

Honggyu Jung, Seong-Whan Lee

TL;DR

This paper tackles the challenge of few-shot learning by proposing a novel fine-tuning strategy that adds new categories to an already trained network without degrading performance on existing categories. Their key innovation lies in two geometric constraints that ensure new category features cluster around their weights while remaining distinct from existing category weights, achieving state-of-the-art results.

What Problem Does This Paper Address?

Deep learning models excel at image classification when trained on massive datasets. However, adapting these models to recognize new categories with very few examples (few-shot learning) is a significant challenge. The core problem is that fine-tuning a network with limited data for new categories can easily corrupt the well-learned feature representations of the original, base categories. This means that while the model might learn the new categories, it could forget what it previously knew. Existing methods often focus solely on learning new categories or struggle to maintain performance on the original ones. This paper aims to bridge this gap by enabling the seamless addition of novel categories to a pre-trained network while preserving its existing knowledge.

Methods

The authors propose a two-stage training procedure.

Stage 1: Base Category Training: A network is trained on a large dataset of base categories using standard cross-entropy loss. Crucially, features and weights are normalized to lie on a high-dimensional sphere, focusing on angular relationships rather than magnitudes.

Stage 2: Novel Category Fine-tuning: 1. Network Adaptation: The top layers of the pre-trained network are duplicated, and the bottom layers are frozen. A new classifier is added for the novel categories. 2. Geometric Constraints: To fine-tune with few examples, two novel geometric constraints are introduced: * Weight-Centric Feature Clustering (WCFC): This constraint encourages features of a novel category to cluster closely around its corresponding category weight. This is achieved by maximizing the cosine similarity between features and weights. * Angular Weight Separation (AWS): This constraint ensures that the weights of novel categories are sufficiently separated (in terms of angular distance) from the weights of the base categories. This prevents interference and maintains distinctiveness. 3. Combined Loss: The final loss function for Stage 2 is a combination of the standard cross-entropy loss, WCFC, and AWS.

The testing procedure uses a "two-stream" approach, where an input is passed through both the base and novel category classifiers, and the highest score determines the predicted class.

Datasets

The paper evaluates its method on two standard few-shot learning datasets:

  • miniImageNet: A subset of ImageNet commonly used for few-shot learning research.
  • Bharath & Girshick's dataset: Another dataset derived from ImageNet, offering a larger number of base and novel categories.

The authors also mention using subsets of ImageNet for their experiments.

Key Insights and Ideas

  • Geometric Constraints as a Solution: The core insight is that by enforcing specific geometric relationships between features and weights in a normalized feature space, one can effectively learn new categories without disrupting existing ones.
  • Preserving Base Knowledge is Crucial: The paper strongly emphasizes that simply fine-tuning for new categories can be detrimental to existing knowledge. Their method explicitly addresses this by keeping base category features intact.
  • Angular Relationships Matter: Normalizing features and weights to a sphere simplifies the problem by focusing on angular distances, making it easier to control the relationships between categories.
  • WCFC and AWS Complement Each Other: WCFC helps in learning the new categories well, while AWS ensures they don't interfere with the old ones. Together, they provide a robust fine-tuning mechanism.
  • Deeper Networks Benefit More: The proposed method shows significantly better performance when used with more powerful feature extractors like ResNet, suggesting that the geometric constraints can better leverage the capacity of deeper models.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results on the miniImageNet dataset, outperforming prevalent few-shot learning methods by a significant margin.
  • Preservation of Base Category Performance: Crucially, the method maintains high accuracy on the original base categories even after fine-tuning for novel categories. For instance, with ResNetS on miniImageNet, base category accuracy remains around 79% while novel categories also see substantial gains.
  • Improved Novel Category Accuracy: The method significantly boosts accuracy on novel categories, demonstrating its effectiveness in learning from limited examples. For example, on miniImageNet, novel category accuracy reaches up to 68.16% for 5-way 5-shot with ResNetS.
  • Effectiveness with Deeper Networks: The performance gains are more pronounced with deeper architectures like ResNetS compared to shallower CNNs (C64F).
  • Data Augmentation Helps: Simple data augmentation techniques (zero-padding, cropping, flipping) applied to novel category training examples further improve performance.

Limitations and Future Work

  • Network Capacity Dependency: The authors acknowledge that the proposed method's effectiveness is tied to the capacity of the network. While it works well with powerful networks like ResNet, its gains might be less pronounced with simpler models.
  • Need for Data Generation: The paper suggests that future work could explore more advanced example generation techniques to further boost performance, especially given the inherent imbalance in few-shot learning scenarios.
  • Imbalanced Categories: The paper notes that few-shot learning inherently deals with imbalanced categories, and future research could focus on addressing this more directly.
  • Generalization to Novel Categories: While the method is designed for novel categories, the paper implicitly suggests that the quality of the base category training is paramount for effective generalization.

Summary updated

Abstract

In this article, we consider the problem of few-shot learning for classification. We assume a network trained for base categories with a large number of training examples, and we aim to add novel categories to it that have only a few, e.g., one or five, training examples. This is a challenging scenario because: 1) high performance is required in both the base and novel categories; and 2) training the network for the new categories with a few training examples can contaminate the feature space trained well for the base categories. To address these challenges, we propose two geometric constraints to fine-tune the network with a few training examples. The first constraint enables features of the novel categories to cluster near the category weights, and the second maintains the weights of the novel categories far from the weights of the base categories. By applying the proposed constraints, we extract discriminative features for the novel categories while preserving the feature space learned for the base categories. Using public data sets for few-shot learning that are subsets of ImageNet, we demonstrate that the proposed method outperforms prevalent methods by a large margin.
BibTeX
@article{jung2020few,
  author = {Honggyu Jung and Seong-Whan Lee},
  title = {{Few-Shot Learning With Geometric Constraints}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2020},
  url = {https://openreview.net/forum?id=xe4qBjbRovX},
}
Figure