On the Behavior of Convolutional Nets for Feature Extraction

Dario Garcia-Gasulla, Ferran Parés, Armand Vilalta, Jonathan Moreno, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, Toyotaro Suzumura

Abstract

While transfer learning typically leverages the final layers of Convolutional Neural Networks (CNNs) as generic feature extractors, the behavior of features across the entire network depth remains poorly understood. This paper presents a statistical analysis of every feature in a VGG16 model across 11 diverse datasets. Using the Signed Kolmogorov-Smirnov (KS) statistic, the authors demonstrate that early convolutional layers provide highly stable and discriminative features across domains. Furthermore, they show that the absence of a feature is as informative as its presence, and that data-driven pruning can significantly reduce feature vector dimensionality without compromising performance.

Introduction

In computer vision, the standard approach to transfer learning involves using pre-trained models, such as VGG16 trained on ImageNet, and extracting activations from the final fully connected layers. This empirical approach often ignores the rich visual information encoded in earlier convolutional layers. This study addresses this gap by providing a statistical foundation for understanding how individual features behave across the entire depth of a network when applied to new, diverse domains. The authors investigate whether early layers are merely generic edge detectors or if they hold specific discriminative power, and how much "noise" is passed along in massive feature vectors.

Methodology

The researchers utilized the VGG16 architecture as their primary subject. To enable a direct comparison between convolutional and fully connected (FC) layers, spatial average pooling was applied to each convolutional filter, reducing it to a single scalar value per filter per image. This process results in a total of 12,416 embedding features across the network.

VGG16 Architecture and Feature Distribution

The core analytical tool used was the Signed Kolmogorov-Smirnov (KS) statistic ($D_{KS}$). For every feature-class pair, the researchers compared the distribution of activations for images within a class against those outside the class. The $D_{KS}$ value ranges from -1 to 1:

  • Positive $D_{KS}$: The feature is more prevalent/stronger in the target class (presence).
  • Negative $D_{KS}$: The feature is characteristically weaker or missing in the target class (absence).
  • Near Zero: The feature is irrelevant or noisy for that class.

Experimental Setup: Datasets

To ensure the generalizability of their findings, the authors evaluated the features against 11 datasets covering general objects, fine-grained categories, and specialized textures. This variety ensures that the findings are not specific to a single type of image or task.

Overview of the 11 Datasets used for evaluation

The datasets included ImageNet (validation), Caltech101/256, CUB200 (birds), Flowers102, and specialized sets like MIT67 (indoor scenes) and Wood (texture). The study also validated findings by swapping the source task from ImageNet to Places2.

Key Insights

  • The Information of Absence: A negative $D_{KS}$ value—indicating a feature is characteristically missing—is a powerful descriptor. For example, a "flying animal" feature might have a strongly negative $D_{KS}$ for a specific species of ground-dwelling bird. This effectively doubles the information available from a single CNN feature.
  • Generalists vs. Specialists: Convolutional features are remarkably stable across datasets, acting as "universal" visual descriptors. In contrast, FC layers are highly sensitive to the similarity between the source and target tasks. If the tasks are disjoint (e.g., training on objects but testing on textures), FC layers become significantly noisier.
  • Quantifying Noise: By randomizing dataset labels, the authors established a noise baseline. This allowed them to set data-driven thresholds ($t^+$ and $t^-$) to prune irrelevant features.
  • Small Data Penalty: Feature noise is strongly correlated with the number of images per class; fewer samples lead to more extreme but statistically unreliable activations.

Results and Discussion

The application of the proposed thresholds allowed for discarding over 50% of features while retaining essential information. On average, only about 3,300 negative features and 4,000 positive features (out of 12,416) remained after pruning.

The study also found that early layers (conv1_1 to conv3_3) are nearly identical across all 11 datasets, reinforcing their role as a universal visual language. Crucially, every class across all datasets was found to have a set of highly discriminative features at every layer, showcasing the immense richness of the learned representation.

Conclusion

This research moves beyond the "black box" view of CNNs, offering a detailed characterization of feature behavior across all layers. The insights gained suggest that "full-network embeddings" could significantly improve transfer learning performance by intelligently selecting features from various layers based on the target task. While focused on VGG architectures, future work should extend this analysis to modern architectures like ResNets or Transformers to see how skip connections or attention mechanisms affect feature distribution.

Summary updated

BibTeX
@article{garciagasulla2018behavior,
  author = {Dario Garcia-Gasulla and Ferran Parés and Armand Vilalta and Jonathan Moreno and Eduard Ayguadé and Jesús Labarta and Ulises Cortés and Toyotaro Suzumura},
  title = {{On the Behavior of Convolutional Nets for Feature Extraction}},
  journal = {J. Artif. Intell. Res.},
  year = {2018},
  doi = {10.1613/jair.5756},
}
Figure