Incorporating Prototype Theory in Convolutional Neural Networks

Babak Saleh, Ahmed M. Elgammal, Jacob FeldmanIJCAI 2016

Introduction: The Fragility of Modern CNNs

While Convolutional Neural Networks (CNNs) have achieved human-level performance on standard benchmarks like ImageNet, they remain surprisingly fragile when encountering "atypical" objects. A state-of-the-art CNN might misclassify a chair shaped like a motorbike or a boat that resembles a car with 100% confidence, whereas humans categorize these with ease.

Examples of atypical objects and classification discrepancies between CNNs and humans.

The authors argue that standard CNN training focuses on "unstructured loss minimization," prioritizing decision boundaries between classes. This approach emphasizes "hard" examples near the boundary, which are often atypical. In contrast, human learning is often described by Prototype Theory, where we form a concept of a "typical" category member (e.g., a robin for "bird") and reason about atypical members (e.g., penguins) based on their deviation from that prototype. This research proposes bridging this gap by incorporating graded typicality into the CNN learning process.

Methodology: Sample-Based Weighted Loss

The core innovation is a Sample-Based Weighted Loss function. Instead of treating every training image as equally important, a weight ($\tau$) is assigned to each sample during the fine-tuning phase. The researchers explored three weighting strategies:

  1. Typicality Emphasis: Weighting representative samples more heavily to help the model learn the category's "core."
  2. Atypicality Emphasis: Weighting unusual samples more heavily to complement the boundary-seeking nature of standard loss functions.
  3. Polynomial "Both" Strategy: Using an even-degree polynomial function to weight both highly typical and highly atypical samples heavily, while downplaying "middle-of-the-road" images.

Typicality was measured using external scores (one-class SVMs on traditional visual features) and internal scores (CNN output probability or entropy). Additionally, the study compared Softmax log loss with Multi-class structured hinge loss, finding the latter more effective for atypical cases.

Experimental Setup and Datasets

To test generalization, the authors utilized:

  • Training Sets: ImageNet (16,153 images) and PASCAL VOC 2011 (4,950 images).
  • Test Typical: 8,570 images from the ILSVRC 2015 detection challenge.
  • Test Atypical: The Abnormal Object Dataset, containing images difficult for standard classifiers but easy for humans.

Key Insights and Results

A critical insight is that atypicality is not synonymous with being a "boundary case." An atypical image might simply be a highly unusual version of a single class. Standard CNNs often ignore these internal variations, but prototype-based weighting captures them.

Mean accuracy comparison of various weighting functions used in fine-tuning.

The results demonstrate a significant performance boost on atypical images:

  • Baseline: AlexNet achieved ~56.4% accuracy on the atypical test set after one epoch without weighting.
  • Weighting Impact: Typicality weighting boosted this to 67.2%. Atypicality weighting performed even better, reaching 70.6% after ten epochs.
  • Polynomial Weighting: On the PASCAL dataset, the polynomial approach achieved ~55-56% accuracy, a massive jump from the 30% unweighted baseline.

Notably, these improvements were achieved without ever showing the model an "abnormal" image during training, demonstrating superior out-of-distribution generalization.

Conclusion and Future Directions

By adjusting training data weights based on prototype theory, models can become significantly more "human-like" in handling strange data. The study also found that "going deeper" isn't always beneficial; fine-tuning too many layers led to overfitting on the small training sets. Future work includes designing loss functions specifically for typicality and applying this framework to complex tasks like image captioning.

Summary updated

BibTeX
@inproceedings{saleh2016incorporating,
  author = {Babak Saleh and Ahmed M. Elgammal and Jacob Feldman},
  title = {{Incorporating Prototype Theory in Convolutional Neural Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure