Image Classification Based on KPCA and SVM with Randomized Hyper-parameter Optimization

Lin Li, Jin Lian, Yue Wu, Mao YeInternational Journal of Signal Processing Image Processing and Pattern Recognition 2014 · DOI

Abstract

Image classification is one of the most fundamental and useful activities in the computer vision domain. For better accuracy and executing efficiency under the circumstance of high-dimensional feature descriptors, we propose a novel framework for multi-class image classification based on Kernel Principal Component Analysis (KPCA) for feature descriptor post-processing and Support Vector Machine (SVM) with randomized hyper-parameter optimization for classification. We produce the image feature representation by extracting Pyramid Histogram of Visual Word (PHOW) descriptors, then map the descriptors through additive kernels. At the third step, we use KPCA for feature dimensionality reduction. Finally, we classify images by SVM with randomized hyper-parameter optimization. Extensive experiments on the Msrcv2 and 15-Scenes datasets justify that: (1) feature descriptors with KPCA are superior to those with PCA for dimensionality reduction; and (2) SVM with randomized hyper-parameter optimization greatly saves time while maintaining high accuracy.

Introduction

In computer vision, representing an image accurately often requires high-dimensional descriptors. This paper addresses two major hurdles in processing such data: the "curse of dimensionality" and the computational bottleneck of hyper-parameter tuning. For instance, using PHOW descriptors enhanced with additive kernels results in a staggering 36,000-dimensional feature vector per image. Standard linear PCA often fails to capture the complex, non-linear relationships in such data, and traditional "Grid Search" for SVM optimization is computationally exhausting.

Proposed Methodology

The proposed framework follows a four-stage pipeline to streamline the classification process:

  1. Feature Extraction (PHOW): Extracts local textures at multiple scales and spatial layouts using an evolution of dense SIFT.
  2. Additive Kernel Transformation: Applies an additive $\chi^2$ kernel transformation to make features more discriminative, which expands the dimensionality to 36,000.
  3. Non-linear Dimensionality Reduction (KPCA): Projects the data into a lower-dimensional space (typically 100 principal components) to preserve non-linear structures that linear projections might squash.
  4. Classification with Randomized Search: Trains an SVM using Randomized Hyper-parameter Optimization. This samples the parameter space randomly, which is more efficient than brute-force grid searching as it focuses on the most impactful hyper-parameters.
The proposed image classification framework consists of a training phase to generate models and a testing phase for classification.

Experimental Setup and Datasets

The framework was evaluated on two well-known benchmarks:

  • Msrcv2 (Microsoft Research Cambridge): 591 images across 23 object classes.
  • 15-Scenes: 4,485 images across 15 different scene categories (e.g., kitchen, mountain, office).

The researchers tested the model's robustness by using varying amounts of training data, ranging from 10% to 50% of the total sets.

Key Results and Analysis

Experimental results highlighted the efficiency and stability of the KPCA-SVM approach:

  • KPCA vs. PCA: KPCA proved significantly better at extracting discriminant features from kernel-transformed descriptors. While linear PCA showed performance dips on the 15-Scenes dataset, KPCA remained stable. Interestingly, as shown in the training time analysis, KPCA transformations were significantly faster than standard PCA when dealing with these high-dimensional vectors.
  • Dimensionality Threshold: Across different kernels, classifier performance generally stabilized once 100 principal components were reached, indicating massive redundancy in the original 36,000-dimensional space.
  • Optimization Speedup: Randomized search for SVM hyper-parameters was approximately 10 times faster than traditional Grid Search. On the Msrcv2 dataset, randomized search achieved ~49.34% accuracy in just 0.2 seconds, compared to 1.7 seconds for Grid Search.
Comparison of training time for PCA and various KPCA kernels; KPCA variants demonstrate significantly lower execution times as sampling percentage increases.

Conclusion and Future Work

This study demonstrates that combining KPCA for non-linear dimensionality reduction with randomized hyper-parameter optimization provides a robust and efficient pipeline for image classification. While the framework excels on standard benchmarks, future research should explore its scalability to "Big Data" regimes like ImageNet and provide further theoretical insights into kernel selection for KPCA. Additionally, comparing this hand-crafted feature approach against modern end-to-end Convolutional Neural Networks (CNNs) remains a valuable direction for future study.

Summary updated

Abstract

Image classification is one of the most fundamental and useful activities in computer vision domain. For better accuracy and executing efficiency under the circumstance of high dimensional feature descriptors in image classification, we proposes a novel framework for multi-class image classification based on kernel principal component analysis(KPCA) for feature descriptors post-processing and support vector machine (SVM) with randomized hyper-parameter optimization for classification. We produce the image feature representation by extracting pyramid histogram of visual word (PHOW) descriptors of image, then map the descriptors though additive kernels. At the third step we use KPCA for feature dimensionality reduction. Finally we classify image by SVM with randomized hyper-parameter optimization. Extensive experiments are tested on two data sets: Msrcv2, 15-Scenes. These experiments justify that (1) feature descriptors with KPCA is superior to that with PCA for dimensionality reduction;(2)SVM with randomized hyper-parameter optimization greatly saves time while keeping high accuracy.
BibTeX
@article{li2014image,
  author = {Lin Li and Jin Lian and Yue Wu and Mao Ye},
  title = {{Image Classification Based on KPCA and SVM with Randomized Hyper-parameter Optimization}},
  journal = {International Journal of Signal Processing Image Processing and Pattern Recognition},
  year = {2014},
  doi = {10.14257/ijsip.2014.7.4.29},
}
Figure