On the Behavior of Convolutional Nets for Feature Extraction

Dario Garcia-Gasulla, Ferran Parés, Armand Vilalta, Jonathan Moreno, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, Toyotaro Suzumura

TL;DR

This paper moves beyond the "black box" approach of using only the final layers of a CNN for transfer learning. By statistically analyzing every single feature across 11 diverse datasets, the authors demonstrate that low-level convolutional features are more discriminative than previously thought and that a feature’s absence is just as informative as its presence.

What Problem Does This Paper Address?

In the world of computer vision, "transfer learning" usually involves taking a pre-trained model (like VGG16 trained on ImageNet) and using the activations from the very last layers as a "feature vector" for a new task. While effective, this approach is largely empirical and ignores the vast amount of visual knowledge stored in the earlier convolutional layers.

The authors identify a major gap: we don't actually know how individual features across the entire depth of a network behave when applied to new domains. Are early layers just generic edge detectors, or do they hold specific discriminative power? How much "noise" is being passed along in these massive feature vectors? This paper seeks to provide a statistical foundation for understanding feature behavior across all layers.

Methods

The researchers used the VGG16 architecture as their primary subject. To make convolutional layers comparable to fully connected (FC) layers, they applied spatial average pooling to every convolutional filter, resulting in a single scalar value per filter per image.

The core of their analysis is the Signed Kolmogorov-Smirnov (KS) statistic ($D_{KS}$). For every feature-class pair, they compared two distributions:

  1. Inner-class behavior: How the feature activates for images inside a specific class.
  2. Outer-class behavior: How the feature activates for all other images in the dataset.

The $D_{KS}$ value ranges from -1 to 1:

  • Positive $D_{KS}$: The feature is more prevalent/stronger in the target class (presence).
  • Negative $D_{KS}$: The feature is characteristically weaker or missing in the target class (absence).
  • Near Zero: The feature is irrelevant or noisy for that class.

Datasets

The study evaluated features against 11 different datasets to ensure the findings weren't specific to one type of image:

  • General/Object: imagenet (validation split), caltech101, caltech256.
  • Fine-grained: cub200 (birds), flowers102, catsdogs, stanforddogs, food101.
  • Specialized/Texture: mit67 (indoor scenes), textures, wood (knots in spruce).

They also validated their findings by swapping the source task from ImageNet to Places2 (a scene-centric dataset).

Key Insights and Ideas

  • The Information of Absence: One of the most striking findings is that a feature being absent is a powerful descriptor. For example, a "flying animal" feature might have a strongly negative $D_{KS}$ for a specific species of ground-dwelling bird. This effectively doubles the information a single CNN feature can provide.
  • Conv Layers are Generalists, FC Layers are Specialists: Convolutional features remain remarkably stable and discriminative across almost all datasets. In contrast, FC layers are highly sensitive to how similar the new task is to the original training task (ImageNet). If the tasks are disjoint (e.g., training on objects but testing on textures), FC layers become significantly noisier.
  • Defining Noise via Randomization: The authors proposed a clever way to measure noise: they randomized the labels of the datasets and recalculated the $D_{KS}$ values. Any discriminative power found in these "shuffled" datasets is pure noise. This allowed them to set a data-driven threshold ($t^+$ and $t^-$) to prune irrelevant features.
  • The "Small Data" Penalty: The amount of noise in a feature is strongly correlated with the number of images per class. Fewer samples lead to more "extreme" but statistically unreliable feature activations.

Key Results

  • Pruning Power: By applying their proposed thresholds, the authors could discard more than half of the features in an embedding while retaining the most relevant information. On average, only about 3,300 negative features and 4,000 positive features (out of 12,416) remained after pruning.
  • Layer Consistency: Convolutional layers conv1_1 through conv3_3 look almost identical across all 11 datasets, reinforcing the idea that these early layers learn a "universal" visual language.
  • Discriminative Richness: The study found that every class in every dataset had a set of highly discriminative features at every layer. No class was "indescribable" by the network, showcasing the immense richness of the learned representation.

Limitations and Future Work

  • Architecture Scope: The study focuses heavily on VGG16/19. While these are canonical, modern architectures like ResNets or Transformers might exhibit different feature distributions due to skip connections or attention mechanisms.
  • Computational Cost: Calculating the KS statistic for every feature-class pair across 12,000+ features is computationally intensive, though the authors suggest this is a one-time "characterization" cost.
  • Future Direction: The authors suggest these insights can be used to build "full-network embeddings" that intelligently pick features from different layers based on the target task, potentially outperforming standard single-layer transfer learning.

Summary updated

BibTeX
@article{garciagasulla2018behavior,
  author = {Dario Garcia-Gasulla and Ferran Parés and Armand Vilalta and Jonathan Moreno and Eduard Ayguadé and Jesús Labarta and Ulises Cortés and Toyotaro Suzumura},
  title = {{On the Behavior of Convolutional Nets for Feature Extraction}},
  journal = {J. Artif. Intell. Res.},
  year = {2018},
  doi = {10.1613/jair.5756},
}
Figure