TL;DR
This paper proposes a novel method for image classification by extracting features from the inner layers of pre-trained Convolutional Neural Networks (CNNs) and feeding them into Support Vector Machines (SVMs). By combining these deep features with dimensionality reduction techniques like Discrete Cosine Transform (DCT) applied channel-wise, the system achieves state-of-the-art results on various medical image datasets and a virus benchmark.
What Problem Does This Paper Address?
Traditional computer vision systems relied on handcrafted features, which were often task-specific and computationally expensive. While deep learning, particularly CNNs, has revolutionized feature extraction by learning hierarchical representations, effectively utilizing the rich information from inner CNN layers for classification tasks has remained a challenge. These inner layers produce high-dimensional feature vectors that are difficult to directly feed into traditional classifiers like SVMs, which prefer fixed-length, lower-dimensional inputs. This paper aims to bridge this gap by developing a robust system that leverages these inner-layer features for improved image classification accuracy.
Methods
The core of this work lies in extracting features from various layers of pre-trained CNNs (GoogleNet, ResNet50, DenseNet201) and then using these features to train SVMs. To handle the high dimensionality of inner-layer features, several dimensionality reduction techniques were explored:
- Classic Transforms: Principal Component Analysis (PCA) and Discrete Cosine Transform (DCT) were applied to reduce feature dimensions. A key innovation is applying DCT separately to each channel of a layer's output, which proved more effective than global DCT.
- Feature Selection: Chi-square feature selection (CHI) was used to rank and select the most informative features.
- Texture Descriptors: Local Binary Patterns (LBP) were employed as a texture descriptor, followed by CHI for feature selection (LBPCHI).
- Deep Representation: Deep Co-occurrence (CoOC) was explored, which captures relationships between activation values within a region.
- Global Pooling: Global Entropy Pooling (GEP) and Global Mean Thresholding Pooling (GMTP) were used to aggregate feature information from layers.
- Ensembling: Various combinations of these methods and features from different layers were ensembled using a sum rule to further boost performance. Sequential Forward Floating Selection (SFFS) was used to select optimal layer/classifier combinations.
Datasets
The study utilized a diverse set of 12 publicly available medical image datasets, including:
- CH: Chinese hamster ovary cells
- HE: HeLa cells
- RN: Fly cells (D. melanogaster)
- MA: C. Elegans muscle age
- TB: C. Elegans terminal bulb aging
- LY: Lymphoma
- LG: Liver gender (caloric restriction)
- LA: Liver aging (ad-libitum)
- BGR: Breast grading carcinoma
- LAR: Laryngeal tissues
- LO: Endogenous cells
- TR: Transfected cells
Additionally, a popular virus benchmark dataset (object scale) was used to test generalizability.
Key Insights and Ideas
- Inner Layers Matter: Features extracted from the inner layers of CNNs, not just the final layers, contain valuable information for classification and can significantly boost performance when properly processed.
- Channel-wise DCT is Powerful: Applying DCT independently to each channel of a CNN layer's output proved to be a highly effective dimensionality reduction technique, outperforming global DCT.
- Ensembling is Key: Combining features from multiple layers and different dimensionality reduction methods, especially through sum rule, leads to substantial performance gains and can achieve state-of-the-art results.
- Tuning vs. Pretraining: While tuning CNNs on specific datasets can improve performance, the proposed feature extraction and SVM approach can also achieve excellent results with pre-trained models, offering a more generalizable solution.
- SVMs Remain Relevant: Despite the rise of end-to-end deep learning, SVMs, when fed with well-engineered deep features, remain a competitive and computationally efficient choice for classification.
Key Results
The proposed system demonstrated significant improvements over standard CNN classification. For instance, using ResNet50, the proposed channel-based DCT method achieved an accuracy of 93.1%, a substantial increase from the baseline CNN's 85.5%. The best ensemble, (DCT + GMTP)-2, reached an accuracy of 93.9%. Similar improvements were observed with GoogleNet. On the virus benchmark dataset, the (DCT+GMTP)-2 ensemble achieved state-of-the-art results, reaching 89.60% accuracy with DenseNet201. The study also found that discarding the SVMs trained on the two last layers slightly improved performance in some cases.
Limitations and Future Work
The paper acknowledges that while the proposed method achieves excellent results, further research could explore combining this approach with other deep neural networks. Additionally, testing different methods for representing inner-layer features in a compact way for SVM training is suggested. The authors also note that for the GoogleNet experiments, only the most interesting approaches tested on ResNet50 were evaluated to save computation time, implying that a full exploration might yield further insights.
Summary updated