PCANet: A Simple Deep Learning Baseline for Image Classification?

Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, Yi Ma

TL;DR

PCANet is a "back-to-basics" deep learning architecture that replaces complex backpropagation and non-linear activations with simple Principal Component Analysis (PCA), binary hashing, and block-wise histograms. Despite its extreme simplicity and lack of traditional "learning," it matches or exceeds state-of-the-art performance on benchmark tasks like face recognition and digit classification, providing a highly efficient baseline for the computer vision community.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (ConvNets) have revolutionized image classification, but they often feel like "black boxes." They require massive datasets, extensive hyperparameter tuning, and significant computational power for Stochastic Gradient Descent (SGD). Furthermore, the mathematical justification for why specific architectures work is often secondary to empirical success.

The authors of PCANet ask a fundamental question: How much of the complexity in deep learning is actually necessary? They sought to design a network using only the most basic data processing components to see if a "naive" model could serve as a competitive baseline against more sophisticated, hand-crafted, or heavily tuned models.

Methods

PCANet consists of three incredibly simple stages:

  1. Cascaded PCA: Instead of learning filters via backpropagation, the network takes patches from the input images and applies Principal Component Analysis (PCA) to find the lead eigenvectors. These eigenvectors become the convolutional filters. This process is repeated for a second stage to capture higher-level features.
  2. Binary Hashing: The output of the second stage is binarized using a simple Heaviside step function (mapping values to 0 or 1). This acts as a non-linear mapping.
  3. Block-wise Histograms: The binary strings are converted into decimal values, and the image is partitioned into blocks. The network computes a histogram of these values for each block and concatenates them. This provides translation invariance and serves as the final feature vector for a linear classifier (like an SVM).

The authors also tested two variations: RandNet (using random filters) and LDANet (using filters learned via Linear Discriminant Analysis).

Datasets

The researchers evaluated PCANet across a wide variety of tasks:

  • Face Recognition/Verification: MultiPIE, Extended Yale B, AR, FERET, and LFW (Labeled Faces in the Wild).
  • Handwritten Digit Recognition: MNIST and its various noisy/rotated variations.
  • Texture Classification: CUReT.
  • Object Recognition: CIFAR-10.

Key Insights and Ideas

  • The Power of Simplicity: One of the most surprising findings is that PCANet contains no non-linear operations (like ReLU or Sigmoid) between its convolutional stages. The authors found that moving the non-linearity and pooling to the very last stage (via hashing and histograms) was sufficient for high performance.
  • Efficiency is King: Because PCA is a closed-form solution, training is orders of magnitude faster than SGD-based networks. Training PCANet on 100,000 images took only 30 minutes, whereas a comparable ConvNet took 6 hours.
  • Robustness to Occlusion: PCANet showed remarkable resilience to image corruption. In the Extended Yale B test, it maintained over 86% accuracy even when 60% of the face was randomly occluded.
  • PCA as an Auto-encoder: The paper frames PCA as the simplest possible class of auto-encoder, minimizing reconstruction error without the need for complex regularization.

Key Results

PCANet didn't just act as a baseline; it set new records at the time of publication:

  • Extended Yale B: Achieved 99.58% accuracy, proving nearly immune to lighting variations.
  • FERET: Obtained a state-of-the-art average accuracy of 97.25%.
  • MNIST: Achieved a 0.62% error rate, which was competitive with much more complex deep learning models.
  • LFW: In an unsupervised setting, it reached 86.28% accuracy, outperforming several high-dimensional hand-crafted features.
  • CIFAR-10: While it reached 78.67% accuracy, it lagged behind the best ConvNets (which hit ~89%), suggesting that for highly varied object recognition, end-to-end supervised learning still holds an edge.

Limitations and Future Work

  • Exponential Dimensionality: The size of the final feature vector grows exponentially with the number of stages and filters. This makes it difficult to stack PCANet "very deep" (e.g., 50+ layers) like modern ResNets.
  • Generalization to Complex Objects: While PCANet excels at "well-prepared" data (aligned faces or digits), it struggles more with the extreme scale and pose variations found in unconstrained object datasets like ImageNet.
  • Future Directions: The authors suggest exploring "tensor-like" filters to manage dimensionality and investigating better pre-processing for pose alignment to help the network generalize to more complex, unconstrained datasets.

Summary updated

Abstract

In this paper, we propose a very simple deep learning network for image classification that is based on very basic data processing components: 1) cascaded principal component analysis (PCA); 2) binary hashing; and 3) blockwise histograms. In the proposed architecture, the PCA is employed to learn multistage filter banks. This is followed by simple binary hashing and block histograms for indexing and pooling. This architecture is thus called the PCA network (PCANet) and can be extremely easily and efficiently designed and learned. For comparison and to provide a better understanding, we also introduce and study two simple variations of PCANet: 1) RandNet and 2) LDANet. They share the same topology as PCANet, but their cascaded filters are either randomly selected or learned from linear discriminant analysis. We have extensively tested these basic networks on many benchmark visual data sets for different tasks, including Labeled Faces in the Wild (LFW) for face verification; the MultiPIE, Extended Yale B, AR, Facial Recognition Technology (FERET) data sets for face recognition; and MNIST for hand-written digit recognition. Surprisingly, for all tasks, such a seemingly naive PCANet model is on par with the state-of-the-art features either prefixed, highly hand-crafted, or carefully learned [by deep neural networks (DNNs)]. Even more surprisingly, the model sets new records for many classification tasks on the Extended Yale B, AR, and FERET data sets and on MNIST variations. Additional experiments on other public data sets also demonstrate the potential of PCANet to serve as a simple but highly competitive baseline for texture classification and object recognition.
BibTeX
@article{chan2015pcanet,
  author = {Tsung-Han Chan and Kui Jia and Shenghua Gao and Jiwen Lu and Zinan Zeng and Yi Ma},
  title = {{PCANet: A Simple Deep Learning Baseline for Image Classification?}},
  journal = {IEEE Trans. Image Process.},
  year = {2015},
  url = {https://openreview.net/forum?id=r3VO6Bpnul9},
}

Loading related content...

Figure