SVM and ELM: Who Wins? Object Recognition with Deep Convolutional Features from ImageNet

Lei Zhang, David Zhang

Abstract

Deep learning with a convolutional neural network (CNN) has been proved to be very effective in feature extraction and representation of images. For image classification problems, this work aims at finding which classifier is more competitive based on high-level deep features of images. In this report, we discuss Nearest Neighbor (NN), Support Vector Machines (SVM), and Extreme Learning Machines (ELM) for image classification under deep convolutional activation feature representation. Specifically, we adopt a benchmark object recognition dataset from multiple sources with domain bias for evaluating different classifiers. The deep features are obtained by a well-trained CNN with five convolutional layers and three fully-connected layers on the challenging ImageNet dataset. Experiments demonstrate that ELMs outperform SVMs in cross-domain recognition tasks. In particular, state-of-the-art results are obtained by Kernel ELM (KELM), which outperforms SVMs by approximately 4% in average accuracy.

Introduction: The Challenge of Domain Bias

In modern computer vision, Convolutional Neural Networks (CNNs) are often utilized as powerful feature extractors. A common workflow involves taking a pre-trained network (e.g., trained on ImageNet) and using its high-level activations as input for a separate classifier. However, a significant challenge remains: domain bias. This occurs when the training data (source domain) differs significantly in quality, lighting, or perspective from the test data (target domain).

While SVMs have long been the standard choice for classification on top of deep features, this paper investigates whether Extreme Learning Machines (ELMs) might be better suited for the high-dimensional, abstract activation features produced by deep networks, particularly when facing domain shifts.

Methodology and Feature Extraction

The researchers compared five classification techniques: Nearest Neighbor (NN), Support Vector Machine (SVM), Least Square SVM (LSSVM), Extreme Learning Machine (ELM), and Kernel ELM (KELM). Unlike SVMs, which require complex quadratic programming, ELMs use a single-hidden-layer feed-forward structure where input weights are randomly assigned and only output weights are analytically calculated, leading to significantly faster training.

Diagram of the training and testing protocol, showing the CNN architecture and the extraction of deep features for SVM/ELM classification.

Features were extracted from the 6th ($f_6$) and 7th ($f_7$) fully-connected layers of a CNN pre-trained on ImageNet. These activations, known as Deep Convolutional Activation Features (DeCAF), represent high-level semantic information about the images.

Benchmarking on the 4DA Dataset

To test robustness against domain shift, the study utilized the 4DA (Domain Adaptation) dataset, consisting of 10 object classes across four distinct domains:

  • Amazon (A): High-quality merchant site images.
  • DSLR (D): High-resolution professional photos.
  • Webcam (W): Low-resolution, noisy images.
  • Caltech-256 (C): A standard object recognition benchmark.

Experimental Results: ELM vs. SVM

The experiments focused on cross-domain recognition tasks (e.g., training on Amazon images and testing on Webcam images). The results consistently showed that higher-level features ($f_7$) provided better discriminative power than $f_6$.

Accuracy comparison across various cross-domain tasks (e.g., A->D, C->D) for different classifiers using f6 and f7 features.

As shown in the performance charts, Kernel ELM (KELM) emerged as the clear winner. In the "Source Only" cross-domain setting—the most difficult scenario where no target data is seen during training—KELM outperformed SVM by an average of 4% in accuracy. Even when a few labeled samples from the target domain were added (Source + Target), KELM maintained a lead of approximately 1.5%.

Conclusion

The findings suggest that the random projection philosophy of ELMs, combined with the mathematical efficiency of kernels, makes them more robust to domain shifts than the margin-maximization approach of SVMs when dealing with high-dimensional deep features. While CNNs excel at feature representation, the choice of the final classifier remains a critical factor in achieving state-of-the-art performance in real-world, cross-domain applications.

Summary updated

Abstract

Deep learning with a convolutional neural network (CNN) has been proved to be very effective in feature extraction and representation of images. For image classification problems, this work aim at finding which classifier is more competitive based on high-level deep features of images. In this report, we have discussed the nearest neighbor, support vector machines and extreme learning machines for image classification under deep convolutional activation feature representation. Specifically, we adopt the benchmark object recognition dataset from multiple sources with domain bias for evaluating different classifiers. The deep features of the object dataset are obtained by a well-trained CNN with five convolutional layers and three fully-connected layers on the challenging ImageNet. Experiments demonstrate that the ELMs outperform SVMs in cross-domain recognition tasks. In particular, state-of-the-art results are obtained by kernel ELM which outperforms SVMs with about 4% of the average accuracy. The features and codes are available in http://www.escience.cn/people/lei/index.html
BibTeX
@misc{zhang2015svm,
  author = {Lei Zhang and David Zhang},
  title = {{SVM and ELM: Who Wins? Object Recognition with Deep Convolutional Features from ImageNet}},
  year = {2015},
  eprint = {1506.02509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1506.02509},
}
Figure