CactusNets: Layer Applicability as a Metric for Transfer Learning

Edward Collier, Robert DiBiano, Supratik Mukhopadhyay

TL;DR

This paper introduces "layer applicability," a novel metric to quantify how well features learned by a specific layer in a neural network can be used to classify a particular input. This metric is then used to build CactusNets, a new unsupervised learning architecture that dynamically branches and adapts based on input applicability, enabling more efficient knowledge transfer and learning.

What Problem Does This Paper Address?

Deep neural networks learn hierarchical features, with lower layers capturing generic patterns and higher layers becoming more specialized. This specialization is what makes transfer learning (like fine-tuning) possible, as generic features can be adapted to new tasks. However, a critical gap exists: there's no quantitative way to measure how applicable a learned feature set from a specific layer is to a specific input or class, especially when dealing with unknown or novel classes. This makes it difficult to intelligently route inputs through a network or to build systems that can learn new concepts without explicit labels, mimicking human-like learning by analogy.

Methods

The core contribution is the layer applicability metric. For a given layer $n_i$ and an input $x$ belonging to a class $c$, the metric measures how well the network, with layers up to $n_i$ frozen, can differentiate $c$ from a set of other potential classes ($u_n$). This is done by fine-tuning the network for each pair $(x, u_{nj})$ and recording the accuracy. The class applicability is then the average of these accuracies across all $u_{nj}$.

To enable real-time prediction of this applicability, the authors train applicability predictor networks. These are small CNNs that take the feature activations from a specific layer as input and output a predicted applicability score.

These components are integrated into the CactusNet architecture. This is a branching network where, at each layer, the predicted applicability of the input is compared against predefined thresholds. If applicability is low, a new branch is created, allowing the network to learn new classes or adapt to different problem types in an unsupervised manner. This branching mechanism aims to maximize feature reuse by sharing generic lower-layer features across all branches.

Datasets

  • ImageNet 2012 dataset: Used for training and evaluating the framework. The dataset was split into "man-made" and "natural" halves.
  • The "natural" half was used to train the base convolutional neural network.
  • A subset of "natural" classes not used in training was held out to represent "unknown objective" classes.
  • The "man-made" half served as the "unknown nonobjective" set.
  • A set of 20 classes (10 natural, 10 man-made) was used to approximate the space of all possible inputs for calculating class applicability.

Key Insights and Ideas

  • Layer Applicability as a Measurable Concept: The paper successfully defines and quantifies "layer applicability," providing a concrete metric for a previously abstract notion. This opens the door for more intelligent network design and adaptation.
  • Feature Specialization is Layer-Dependent: The applicability metric confirms that features become more specialized (and thus less broadly applicable) as you move deeper into the network, aligning with existing understanding but now quantitatively demonstrated.
  • Predicting Applicability is Feasible: The success of training small predictor networks to estimate layer applicability in real-time is a significant finding, enabling dynamic network behavior.
  • Unsupervised Learning via Branching: CactusNets demonstrate a novel approach to unsupervised learning by using applicability to dynamically create new branches for novel inputs, effectively growing the network as needed.
  • Mimicking Human Analogy: The concept of learning by analogy, where humans leverage existing knowledge to understand new concepts, is directly addressed by the CactusNet's ability to route inputs based on applicability.

Key Results

  • Applicability Predictors: The applicability predictor networks achieved a training Mean Squared Error (MSE) of 0.1485 and a testing MSE of 0.4889, indicating they can effectively learn to predict applicability.
  • CactusNet Branching: The framework demonstrated the ability to identify inputs as "known," "unknown but applicable," or "non-applicable," leading to appropriate branching decisions. For example, images with applicability below a certain threshold ($T_2$) triggered branching, signifying the need for new class learning.
  • Qualitative Success: The paper shows examples where the CactusNet correctly identifies known classes (e.g., furry mammals) and appropriately branches for novel or less applicable inputs (e.g., moths, screwdrivers).
  • Thresholds: Specific thresholds ($T_1=0.985$, $T_2=0.965$ at conv4) were used to guide branching decisions, separating known classes from new ones.

Limitations and Future Work

  • Approximation of Input Space: The paper acknowledges that approximating the set of all possible classes with a finite set of 20 classes is a simplification.
  • Image Applicability Measurement: While class applicability is defined, the authors note they haven't found a direct way to explicitly define or measure individual image applicability, relying on averaging class applicability.
  • Future Work: The authors suggest that the CactusNet architecture can be expanded and applied to other deep neural networks and datasets. They also highlight that the branching mechanism can be combined with existing one-shot learning methods for more efficient learning within new branches. The paper also mentions that branching realistically starts to occur beyond conv4, suggesting further investigation into earlier layer branching.

Summary updated

Abstract

Deep neural networks trained over large datasets learn features that are both generic to the whole dataset, and specific to individual classes in the dataset. Learned features tend towards generic in the lower layers and specific in the higher layers of a network. Methods like fine-tuning are made possible because of the ability for one filter to apply to multiple target classes. Much like the human brain this behavior, can also be used to cluster and separate classes. However, to the best of our knowledge there is no metric for how applicable learned features are to specific classes. In this paper we propose a definition and metric for measuring the applicability of learned features to individual classes, and use this applicability metric to estimate input applicability and produce a new method of unsupervised learning we call the CactusNet.
BibTeX
@inproceedings{collier2018cactusnets,
  author = {Edward Collier and Robert DiBiano and Supratik Mukhopadhyay},
  title = {{CactusNets: Layer Applicability as a Metric for Transfer Learning}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=PNNEJj6O33},
}

Loading related content...

Figure