Deep Features for training Support Vector Machine

Loris Nanni, Stefano Ghidoni, Sheryl Brahnam

Abstract

Features play a crucial role in computer vision. Initially designed to detect salient elements by means of handcrafted algorithms, features are now often learned by different layers in Convolutional Neural Networks (CNNs). This paper develops a generic computer vision system based on features extracted from trained CNNs. Multiple learned features are combined into a single structure to work on different image classification tasks. The proposed system was experimentally derived by testing several approaches for extracting features from the inner layers of CNNs and using them as inputs to SVMs that are then combined by sum rule. Dimensionality reduction techniques are used to reduce the high dimensionality of inner layers. The resulting vision system is shown to significantly boost the performance of standard CNNs across a large and diverse collection of image data sets. An ensemble of different topologies using the same approach obtains state-of-the-art results on a virus data set.

Introduction

In the field of computer vision, the paradigm has shifted from "handcrafted" features (such as SIFT or LBP) to "learned" features via Convolutional Neural Networks (CNNs). However, standard deep learning workflows typically only utilize the final layer of a network for classification. This paper argues that this approach discards a wealth of generalizable information stored in intermediate layers.

Early layers in a CNN capture low-level textures and edges, while middle layers capture complex patterns and parts of objects. For specialized tasks—like classifying medical images or viruses—the high-level semantic features in the final layer might be too specific to the original training data (often ImageNet), leading to sub-optimal performance on niche datasets. The challenge lies in effectively extracting and utilizing the massive, high-dimensional data from these intermediate layers without overwhelming a traditional classifier.

Proposed Methodology

The authors propose a hybrid pipeline that combines the feature-learning power of CNNs with the robust classification capabilities of Support Vector Machines (SVMs).

Proposed architecture for extracting deep features and training SVM ensembles.

The methodology involves several key stages:

  1. Feature Extraction: Pre-trained architectures such as GoogleNet, ResNet50, and DenseNet201 are used. Features are extracted from the middle layer, every tenth layer thereafter, and the final four layers.
  2. Dimensionality Reduction: Because intermediate layers are high-dimensional, the authors tested several techniques to make the data digestible for an SVM, including Discrete Cosine Transform (DCT), PCA, and global pooling methods like Entropy (GEP) or Mean Thresholding (GMTP).
  3. Classification Ensemble: Each selected layer’s reduced features are used to train a separate SVM. The final prediction is made by combining these SVMs using a simple sum rule.
  4. Optimization: Sequential Forward Floating Selection (SFFS) was used to automatically pick the best combination of layers and classifiers for a given task.

Experimental Results

The system was stress-tested across 13 diverse datasets, including 12 medical image datasets and a virus benchmark dataset. The results consistently showed a clear advantage for the "Deep Feature + SVM" approach.

Performance comparison of various ensemble methods using ResNet50 across multiple datasets.

Key insights from the experiments include:

  • The "Middle" Matters: The best performance (labeled TunLayer-3) was consistently obtained by looking three layers back from the end, rather than just using the final layer. This confirms that the most "abstract" features aren't always the most useful for transfer learning.
  • General Performance Boost: On ResNet50, the proposed ensemble methods (like DC+PC+GMTP) achieved an average accuracy of ~94% across the medical datasets, compared to just 85.5% for a standard standalone CNN.
  • State-of-the-Art (SOTA): On the Virus dataset, an ensemble of DenseNet201 and ResNet50 using this feature extraction method reached 89.60% accuracy, surpassing previous benchmarks in the literature.

Conclusion

This research demonstrates that an ensemble of SVMs trained on different layers of a fixed CNN can provide a much higher accuracy floor than standard fine-tuning, especially on small or specialized datasets. While there is a computational overhead associated with training multiple SVMs, the significant performance gains across diverse datasets justify the approach. Future work will explore more compact representations to reduce the memory footprint of these intermediate layers.

Summary updated

Abstract

Features play a crucial role in computer vision. Initially designed to detect salient elements by means of handcrafted algorithms, features are now often learned by different layers in Convolutional Neural Networks (CNNs). This paper develops a generic computer vision system based on features extracted from trained CNNs. Multiple learned features are combined into a single structure to work on different image classification tasks. The proposed system was experimentally derived by testing several approaches for extracting features from the inner layers of CNNs and using them as inputs to SVMs that are then combined by sum rule. Dimensionality reduction techniques are used to reduce the high dimensionality of inner layers. The resulting vision system is shown to significantly boost the performance of standard CNNs across a large and diverse collection of image data sets. An ensemble of different topologies using the same approach obtains state-of-the-art results on a virus data set.
BibTeX
@misc{nanni2021deep,
  author = {Loris Nanni and Stefano Ghidoni and Sheryl Brahnam},
  title = {{Deep Features for training Support Vector Machine}},
  year = {2021},
  eprint = {2104.03488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.03488},
}
Figure