An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification

Abien Fred AgaraparXiv 2017 · OpenReview

TL;DR

This paper explores replacing the traditional Softmax classifier at the end of a Convolutional Neural Network (CNN) with a linear Support Vector Machine (SVM). The authors found that while their CNN-SVM architecture performed slightly worse than a CNN-Softmax on the MNIST dataset, it achieved comparable results on the more challenging Fashion-MNIST dataset, suggesting SVMs are a viable alternative for image classification.

What Problem Does This Paper Address?

Deep learning models, particularly Convolutional Neural Networks (CNNs), have achieved state-of-the-art performance across various tasks like image classification. Conventionally, the final layer of these networks uses a Softmax function to output class probabilities. However, some research has investigated using Support Vector Machines (SVMs) as an alternative classifier. SVMs are inherently binary classifiers, and their application in multi-class scenarios typically involves a "one-versus-all" strategy. This paper aims to further explore the effectiveness of combining a CNN with a linear SVM, specifically investigating its performance against the standard CNN-Softmax architecture on both a well-established and a newer, more challenging image dataset.

Methods

The core of the methodology involves building a CNN architecture and then comparing two different classification heads: a standard Softmax and a linear SVM.

  • CNN Architecture: The authors implemented a relatively simple CNN consisting of two convolutional layers with max pooling, followed by fully connected layers. The activation function used is the Rectified Linear Unit (ReLU), chosen for its efficiency in accelerating gradient descent convergence.
  • SVM Integration: For the CNN-SVM model, the output of the CNN's final fully connected layer is fed into a linear SVM. The SVM's objective is to find an optimal hyperplane to separate classes. The paper notes that for multi-class SVM, a "one-versus-all" approach is used. The SVM's parameters are learned using the Adam optimizer.
  • Softmax Baseline: For comparison, a standard CNN architecture ending with a Softmax classifier is used.
  • Loss Function: The CNN-SVM uses the L2-SVM loss function, which is a differentiable version of the hinge loss, aiming for more stable results.

Datasets

The study utilized two popular image classification datasets:

  • MNIST: A widely used dataset of handwritten digits, consisting of 60,000 training images and 10,000 test images.
  • Fashion-MNIST: A newer dataset proposed as a more challenging alternative to MNIST. It comprises images of clothing items and shares the same distribution, number of classes (10), and image format as MNIST.

Crucially, the paper states that both datasets were used "as they were, with no preprocessing such as normalization or dimensionality reduction."

Key Insights and Ideas

  • SVM as a Viable Classifier: The results suggest that a linear SVM can be a competitive classifier when integrated with a CNN, especially on more complex datasets like Fashion-MNIST, where its performance is close to that of Softmax.
  • Fashion-MNIST's Challenge: The significant drop in accuracy for both models on Fashion-MNIST compared to MNIST highlights the dataset's increased difficulty and its better representation of modern computer vision tasks.
  • Impact of Preprocessing: The authors explicitly mention that the lack of data preprocessing (like normalization) might have limited the performance of both models, particularly on MNIST, where the CNN-Softmax slightly outperformed the CNN-SVM, contrary to some prior findings.

Key Results

  • MNIST Performance:
    • CNN-Softmax achieved a test accuracy of 99.23%.
  • CNN-SVM achieved a test accuracy of 99.04%.
  • Fashion-MNIST Performance:
    • CNN-Softmax achieved a test accuracy of 91.86%.
  • CNN-SVM achieved a test accuracy of 90.72%.
  • Training Time: Both models completed training in approximately 4 minutes and 16 seconds on MNIST, and the CNN-SVM was slightly faster (4 minutes 29 seconds) than CNN-Softmax (4 minutes 47 seconds) on Fashion-MNIST.
  • Training Loss: On MNIST, CNN-Softmax had a lower average training loss (0.1368) than CNN-SVM (0.2690). On Fashion-MNIST, CNN-Softmax also had a lower average training loss (0.2598) compared to CNN-SVM (0.7937).

Limitations and Future Work

  • Simplicity of CNN: The authors acknowledge that the base CNN model used in this study was relatively simple. They hypothesize that using a more sophisticated CNN architecture and employing data preprocessing techniques (like normalization) could lead to improved results, potentially reproducing findings from prior work that showed a stronger advantage for CNN-SVM.
  • No Data Preprocessing: The decision to not preprocess the datasets might have impacted the performance, especially on MNIST. Future work could explore the benefits of normalization and other preprocessing steps.
  • Contradiction with Prior Work: The study's findings on MNIST, where CNN-Softmax outperformed CNN-SVM, contradict some previous research. This discrepancy is attributed to the simpler CNN architecture and lack of preprocessing in this study. Future work should aim to replicate the conditions of prior studies to further validate the CNN-SVM approach.

Summary updated

Abstract

Convolutional neural networks (CNNs) are similar to "ordinary" neural networks in the sense that they are made up of hidden layers consisting of neurons with "learnable" parameters. These neurons receive inputs, performs a dot product, and then follows it with a non-linearity. The whole network expresses the mapping between raw image pixels and their class scores. Conventionally, the Softmax function is the classifier used at the last layer of this network. However, there have been studies (Alalshekmubarak and Smith, 2013; Agarap, 2017; Tang, 2013) conducted to challenge this norm. The cited studies introduce the usage of linear support vector machine (SVM) in an artificial neural network architecture. This project is yet another take on the subject, and is inspired by (Tang, 2013). Empirical data has shown that the CNN-SVM model was able to achieve a test accuracy of ~99.04% using the MNIST dataset (LeCun, Cortes, and Burges, 2010). On the other hand, the CNN-Softmax was able to achieve a test accuracy of ~99.23% using the same dataset. Both models were also tested on the recently-published Fashion-MNIST dataset (Xiao, Rasul, and Vollgraf, 2017), which is suppose to be a more difficult image classification dataset than MNIST (Zalandoresearch, 2017). This proved to be the case as CNN-SVM reached a test accuracy of ~90.72%, while the CNN-Softmax reached a test accuracy of ~91.86%. The said results may be improved if data preprocessing techniques were employed on the datasets, and if the base CNN model was a relatively more sophisticated than the one used in this study.
BibTeX
@misc{agarap2017architecture,
  author = {Abien Fred Agarap},
  title = {{An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification}},
  year = {2017},
  eprint = {1712.03541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.03541},
}
Figure