Deep Linear Discriminant Analysis Matthias Dorfer

Rainer Kelz, Gerhard WidmerICLR 2016 · arXiv · OpenReview

This paper introduces Deep Linear Discriminant Analysis (DeepLDA), a novel approach that marries the feature-learning power of deep neural networks with the classical statistical properties of Linear Discriminant Analysis (LDA). By replacing the standard softmax/cross-entropy output layer with an objective function derived from the LDA eigenvalue problem, DeepLDA forces the network to learn latent representations that are explicitly linearly separable. This leads to significant performance gains, particularly on datasets with limited labeled samples.

The Problem with Standard Cross-Entropy

In standard deep learning for classification, models typically rely on Categorical Cross-Entropy (CCE) loss. While CCE is highly effective at minimizing classification error, it does not impose any specific geometric constraints on the "latent space"—the internal representation the network learns. As long as the final layer can draw some boundary between classes, the CCE objective is satisfied.

Comparison of standard Categorical Cross-Entropy network and the proposed DeepLDA architecture.

The authors argue that this lacks the beneficial properties of Linear Discriminant Analysis (LDA). Classic LDA seeks a projection where classes have low variance within themselves (tight clusters) and high variance between each other (widely separated clusters). However, classic LDA is a linear method and cannot handle the complex, non-linear patterns found in raw images. The "gap" addressed by this paper is how to perform LDA-style optimization within a non-linear, end-to-end trainable deep learning framework.

Methods: The DeepLDA Architecture

The core innovation of this work is the introduction of an LDA-layer placed on top of a Deep Neural Network (DNN).

  1. The Objective: Instead of maximizing the likelihood of individual labels, the network maximizes the eigenvalues of the LDA generalized eigenvalue problem: $S_b e = v S_w e$, where $S_b$ is the between-class scatter matrix and $S_w$ is the within-class scatter matrix.
  2. The "Smallest $k$" Trick: A naive maximization of all eigenvalues often leads the network to focus only on the "easiest" classes to separate, ignoring the harder ones. To fix this, the authors propose an objective that focuses on maximizing the smallest $k$ eigenvalues. This pushes the network to distribute discriminative power across all available dimensions of the latent space.
  3. End-to-End Training: The authors derived the partial derivatives for this eigenvalue-based loss, allowing the entire system to be trained using standard backpropagation and Stochastic Gradient Descent (SGD).
  4. Regularization: They regularize the within-class scatter matrix ($S_w + \lambda I$) to ensure numerical stability and prevent the overestimation of small eigenvalues.

Datasets

The authors evaluated DeepLDA on three standard computer vision benchmarks:

Sample images from the evaluated datasets: MNIST, CIFAR-10, and STL-10.
  • MNIST: 28x28 grayscale images of handwritten digits.
  • CIFAR-10: 32x32 color images of 10 object classes.
  • STL-10: 96x96 color images. This dataset is particularly challenging because it has very few labeled training examples (only 5,000) compared to its massive unlabeled set.

Key Insights and Results

Unlike standard networks where linear separability is merely a byproduct of training, DeepLDA makes it the primary goal. This results in internal representations where many features are orthogonal to each other. By focusing on the smallest eigenvalues, DeepLDA prevents "feature collapse," where a network might rely on just one or two dominant features to distinguish classes. Instead, it forces the network to find multiple independent ways to separate the data.

The most striking "aha moment" occurs in small data regimes, specifically with the STL-10 dataset. DeepLDA significantly outperforms standard Cross-Entropy when labeled data is scarce, suggesting that the structural "priors" imposed by LDA help the network generalize better from fewer examples.

  • STL-10 (Supervised): In a setting with only 1,000 training samples, DeepLDA achieved 66.97% accuracy, compared to just 57.44% for a standard network with the same architecture using Cross-Entropy—a massive 9.5% improvement.
  • MNIST: DeepLDA achieved a competitive error rate of 0.29%, which was state-of-the-art at the time for models not using data augmentation.
  • CIFAR-10: Results were competitive (7.29% error), though slightly behind standard Cross-Entropy (7.10%). The authors suggest DeepLDA performs better on larger image sizes (like STL-10's 96x96) than on very small ones.

Limitations and Future Work

While powerful, DeepLDA comes with a few notable constraints:

  • Batch Size Sensitivity: Because LDA relies on calculating covariance matrices ($S_w$ and $S_b$), it requires relatively large mini-batches to get stable statistical estimates. This can be a bottleneck for memory-intensive architectures.
  • Computational Complexity: Solving the generalized eigenvalue problem during every training step is more computationally expensive than a simple softmax operation.
  • Image Resolution: The performance gain seemed to diminish on lower-resolution images (like CIFAR-10), suggesting the method's effectiveness might be tied to the dimensionality of the input or the complexity of the learned features.

Looking forward, the authors suggest combining DeepLDA with semi-supervised learning—utilizing the large unlabeled portions of datasets like STL-10—as a highly promising next step.

Summary updated

BibTeX
@inproceedings{kelz2016deep,
  author = {Rainer Kelz and Gerhard Widmer},
  title = {{Deep Linear Discriminant Analysis Matthias Dorfer}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.04707},
}
Figure