Kernel principal component analysis network for image classification

Dan Wu, Jiasong Wu, Rui Zeng, Longyu Jiang, Lotfi Senhadji, Huazhong ShuarXiv 2015 · arXiv

TL;DR

The paper introduces KPCANet (Kernel Principal Component Analysis Network), a deep learning architecture that extends the popular PCANet by incorporating nonlinear kernel functions. By mapping image data into higher-dimensional feature spaces before performing PCA, KPCANet achieves remarkable invariance to illumination changes and robust stability against occlusions and deformations, outperforming its linear predecessor on face and object recognition tasks.

What Problem Does This Paper Address?

A primary challenge in image classification is handling intra-class variability caused by changing illumination, occlusions, and geometric deformations. While deep Convolutional Neural Networks (CNNs) excel at learning invariant features, they often require massive datasets and extensive tuning.

As a simpler alternative, researchers previously proposed PCANet, which uses basic Principal Component Analysis (PCA) to learn data-adaptive convolutional filters. However, PCANet relies entirely on linear PCA, meaning it struggles to capture the complex, nonlinear structures inherent in real-world image variations. This paper addresses that limitation by injecting nonlinear kernel methods directly into the network's structural design.

Methods

KPCANet consists of three main phases: two cascaded Kernel PCA (KPCA) stages followed by a single pooling/hashing stage.

The detailed block diagram of the proposed KPCANet
  1. The KPCA Stages (1 & 2): Instead of computing standard linear PCA filters on image patches, KPCANet maps these patches into a high-dimensional (potentially infinite-dimensional) feature space using a kernel function. The network then extracts the principal eigenvectors of this centralized kernel matrix to serve as convolutional filter banks.
  2. The Pooling Stage: The outputs from the second stage are binarized using a Heaviside step function. These binary images are then converted into decimal values (hashing) to create composite images. Finally, the network computes block-wise histograms of these decimal values and concatenates them into a single, robust feature vector, which is fed into a linear SVM for final classification.

The authors experiment with a wide variety of kernel functions, including Gaussian, Polynomial, Laplacian, Sigmoid, Circular, and Spherical kernels.

Various kernel functions used in this paper

Datasets

The architecture was evaluated on five standard benchmarks:

  • MNIST & USPS: Standard handwritten digit datasets.
  • Yale Face Database: 165 grayscale images of 15 individuals displaying variations in lighting and facial expressions.
  • COIL-100: A dataset containing color images of 100 distinct objects viewed from 72 different angles.
  • AR Face Dataset: A challenging face database containing around 4,000 color images with highly controlled variations in illumination, expressions, and physical disguises (e.g., sunglasses and scarves).

Key Insights and Ideas

  • Task-Dependent Nonlinearity: Interestingly, standard linear PCANet actually performs slightly better on simple, structurally uniform data like handwritten digits (MNIST/USPS). However, for complex objects and faces where lighting and geometry vary nonlinearly, nonlinear kernels significantly outperform the linear baseline.
  • Flawless Illumination Invariance: By mapping patch data to a higher-dimensional kernel space, KPCANet successfully decouples identity features from environmental lighting noise, achieving perfect accuracy under pure illumination changes.
  • The Cost of Depth: While cascading more layers (up to three stages) generally improves accuracy, it comes with a steep computational penalty, as training time scales exponentially with the number of stages due to the kernel matrix diagonalization.

Key Results

The experimental results demonstrate clear advantages for specific kernel selections depending on the dataset:

Comparison of error rates of KPCANet with various kernel functions on different datasets %
  • Object and Face Recognition: On the COIL-100 dataset, the Gaussian kernel reduced the classification error rate to 0.89% (compared to 1.36% for the linear kernel). On the Yale Face dataset, Polynomial and Sigmoid kernels halved the error rate to 2.67% (down from 5.33%).
  • Robustness to Disguise: On the AR Face dataset, both the linear and Laplacian variants of KPCANet achieved a perfect 100% accuracy under illumination changes, and significantly outperformed traditional Local Binary Patterns (LBP) when handling complex expressions and disguises.
Comparison of accuracy rates of the methods on AR face database %
  • Impact of Network Depth: Increasing the filter bank layer number from one to three using a Sigmoid kernel boosted the AR Face recognition accuracy from 94.67% to 97.33%.
Comparison of accuracy rates of KPCANet with different stages number on AR face dataset %

Limitations and Future Work

The authors note that the primary bottleneck of KPCANet is its computational complexity. Because diagonalizing large kernel matrices is computationally expensive, training time grows exponentially as you add more stages to the network. Additionally, selecting the optimal kernel function and tuning its hyperparameters (such as $\sigma$ or $c$) still requires empirical trial and error, leaving room for future work in automated kernel selection.

Summary updated

Abstract

In order to classify the nonlinear feature with linear classifier and improve the classification accuracy, a deep learning network named kernel principal component analysis network (KPCANet) is proposed. First, mapping the data into higher space with kernel principal component analysis to make the data linearly separable. Then building a two-layer KPCANet to obtain the principal components of image. Finally, classifying the principal components with linearly classifier. Experimental results show that the proposed KPCANet is effective in face recognition, object recognition and hand-writing digits recognition, it also outperforms principal component analysis network (PCANet) generally as well. Besides, KPCANet is invariant to illumination and stable to occlusion and slight deformation.
BibTeX
@misc{wu2015kernel,
  author = {Dan Wu and Jiasong Wu and Rui Zeng and Longyu Jiang and Lotfi Senhadji and Huazhong Shu},
  title = {{Kernel principal component analysis network for image classification}},
  year = {2015},
  eprint = {1512.06337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1512.06337},
}
Figure