Deep Features for Training Support Vector Machines

Loris Nanni, Stefano Ghidoni, Sheryl BrahnamJournal of Imaging 2021 · DOI

TL;DR

This paper proposes a novel method for image classification by extracting features from the inner layers of pre-trained Convolutional Neural Networks (CNNs) and feeding them into Support Vector Machines (SVMs). By combining these deep features with dimensionality reduction techniques like Discrete Cosine Transform (DCT) applied channel-wise, the system achieves state-of-the-art results on various medical image datasets and a virus benchmark.

What Problem Does This Paper Address?

Traditional computer vision systems relied on handcrafted features, which were often task-specific and computationally expensive. While deep learning, particularly CNNs, has revolutionized feature extraction by learning hierarchical representations, effectively utilizing the rich information from inner CNN layers for classification tasks has remained a challenge. These inner layers produce high-dimensional feature vectors that are difficult to directly feed into traditional classifiers like SVMs, which prefer fixed-length, lower-dimensional inputs. This paper aims to bridge this gap by developing a robust system that leverages these inner-layer features for improved image classification accuracy.

Methods

The core of this work lies in extracting features from various layers of pre-trained CNNs (GoogleNet, ResNet50, DenseNet201) and then using these features to train SVMs. To handle the high dimensionality of inner-layer features, several dimensionality reduction techniques were explored:

  • Classic Transforms: Principal Component Analysis (PCA) and Discrete Cosine Transform (DCT) were applied to reduce feature dimensions. A key innovation is applying DCT separately to each channel of a layer's output, which proved more effective than global DCT.
  • Feature Selection: Chi-square feature selection (CHI) was used to rank and select the most informative features.
  • Texture Descriptors: Local Binary Patterns (LBP) were employed as a texture descriptor, followed by CHI for feature selection (LBPCHI).
  • Deep Representation: Deep Co-occurrence (CoOC) was explored, which captures relationships between activation values within a region.
  • Global Pooling: Global Entropy Pooling (GEP) and Global Mean Thresholding Pooling (GMTP) were used to aggregate feature information from layers.
  • Ensembling: Various combinations of these methods and features from different layers were ensembled using a sum rule to further boost performance. Sequential Forward Floating Selection (SFFS) was used to select optimal layer/classifier combinations.

Datasets

The study utilized a diverse set of 12 publicly available medical image datasets, including:

  • CH: Chinese hamster ovary cells
  • HE: HeLa cells
  • RN: Fly cells (D. melanogaster)
  • MA: C. Elegans muscle age
  • TB: C. Elegans terminal bulb aging
  • LY: Lymphoma
  • LG: Liver gender (caloric restriction)
  • LA: Liver aging (ad-libitum)
  • BGR: Breast grading carcinoma
  • LAR: Laryngeal tissues
  • LO: Endogenous cells
  • TR: Transfected cells

Additionally, a popular virus benchmark dataset (object scale) was used to test generalizability.

Key Insights and Ideas

  • Inner Layers Matter: Features extracted from the inner layers of CNNs, not just the final layers, contain valuable information for classification and can significantly boost performance when properly processed.
  • Channel-wise DCT is Powerful: Applying DCT independently to each channel of a CNN layer's output proved to be a highly effective dimensionality reduction technique, outperforming global DCT.
  • Ensembling is Key: Combining features from multiple layers and different dimensionality reduction methods, especially through sum rule, leads to substantial performance gains and can achieve state-of-the-art results.
  • Tuning vs. Pretraining: While tuning CNNs on specific datasets can improve performance, the proposed feature extraction and SVM approach can also achieve excellent results with pre-trained models, offering a more generalizable solution.
  • SVMs Remain Relevant: Despite the rise of end-to-end deep learning, SVMs, when fed with well-engineered deep features, remain a competitive and computationally efficient choice for classification.

Key Results

The proposed system demonstrated significant improvements over standard CNN classification. For instance, using ResNet50, the proposed channel-based DCT method achieved an accuracy of 93.1%, a substantial increase from the baseline CNN's 85.5%. The best ensemble, (DCT + GMTP)-2, reached an accuracy of 93.9%. Similar improvements were observed with GoogleNet. On the virus benchmark dataset, the (DCT+GMTP)-2 ensemble achieved state-of-the-art results, reaching 89.60% accuracy with DenseNet201. The study also found that discarding the SVMs trained on the two last layers slightly improved performance in some cases.

Limitations and Future Work

The paper acknowledges that while the proposed method achieves excellent results, further research could explore combining this approach with other deep neural networks. Additionally, testing different methods for representing inner-layer features in a compact way for SVM training is suggested. The authors also note that for the GoogleNet experiments, only the most interesting approaches tested on ResNet50 were evaluated to save computation time, implying that a full exploration might yield further insights.

Summary updated

Abstract

Features play a crucial role in computer vision. Initially designed to detect salient elements by means of handcrafted algorithms, features now are often learned using different layers in convolutional neural networks (CNNs). This paper develops a generic computer vision system based on features extracted from trained CNNs. Multiple learned features are combined into a single structure to work on different image classification tasks. The proposed system was derived by testing several approaches for extracting features from the <i>inner layers</i> of CNNs and using them as inputs to support vector machines that are then combined by sum rule. Several dimensionality reduction techniques were tested for reducing the high dimensionality of the inner layers so that they can work with SVMs. The empirically derived generic vision system based on applying a discrete cosine transform (DCT) separately to each channel is shown to significantly boost the performance of standard CNNs across a large and diverse collection of image data sets. In addition, an ensemble of different topologies taking the same DCT approach and combined with global mean thresholding pooling obtained state-of-the-art results on a benchmark image virus data set.
BibTeX
@article{nanni2021deep,
  author = {Loris Nanni and Stefano Ghidoni and Sheryl Brahnam},
  title = {{Deep Features for Training Support Vector Machines}},
  journal = {Journal of Imaging},
  year = {2021},
  doi = {10.3390/jimaging7090177},
}
Figure