Abstract
Discriminative features play an important role in image and object classification and also in other fields of research such as semi-supervised learning, fine-grained classification, out of distribution detection. Inspired by Linear Discriminant Analysis (LDA), we propose an optimization called Neural Discriminant Analysis (NDA) for Deep Convolutional Neural Networks (DCNNs). NDA transforms deep features to become more discriminative and, therefore, improves the performances in various tasks. Our proposed optimization has two primary goals for inter- and intra-class variances. The first one is to minimize variances within each individual class. The second goal is to maximize pairwise distances between features coming from different classes. We evaluate our NDA optimization in different research fields: general supervised classification, fine-grained classification, semi-supervised learning, and out of distribution detection. We achieve performance improvements in all the fields compared to baseline methods that do not use NDA. Besides, using NDA, we also surpass the state of the art on the four tasks on various testing datasets.
Introduction
While Deep Convolutional Neural Networks (DCNNs) are the gold standard for image classification, the "latent space"—the internal representation of data before the final decision—is not always optimally organized. In many standard training regimes, features from different classes can end up uncomfortably close to each other, or features within the same class can be too scattered.
This lack of discriminative power becomes a major hurdle in several critical scenarios:
- Fine-Grained Visual Classification (FGVC): Distinguishing between very similar categories, such as different breeds of dogs or models of cars, where visual differences are subtle.
- Semi-Supervised Learning (SSL): Learning effectively when labeled data is scarce.
- Out-of-Distribution (OOD) Detection: Identifying when an input image belongs to a category the network has never seen before.
By borrowing the mathematical objectives of classic Linear Discriminant Analysis (LDA), Neural Discriminant Analysis (NDA) forces the network to learn a more structured and robust feature space.
Neural Discriminant Analysis (NDA)
The core contribution of this work is the NDA optimization technique. Unlike standard training which often relies solely on Cross-Entropy loss, NDA utilizes a Siamese network architecture and a multi-component loss function to explicitly shape the latent space.
The optimization focuses on three primary criteria:
- Classification Accuracy ($L_{class}$): Standard categorical cross-entropy loss to ensure correct labeling.
- Intra-class Variance Minimization ($L_{Mean}$): This loss calculates the distance between a feature point and its class mean. It acts as an "anchor," pulling features of the same class toward a central prototype.
- Inter-class Variance Maximization ($L_{Siamese}$): By training on pairs of images, the network is taught to push features from different classes further apart while pulling features from the same class even closer together.
The final NDA loss is a weighted sum of these components: $$L_{NDA} = \alpha(L_{Class}) + \beta(L_{Mean} + \gamma * L_{Siamese})$$
NDA is a "plug-and-play" approach that can be added to almost any existing DCNN architecture (e.g., ResNet or Inception) by attaching these losses to the pre-logit layer.
Experimental Evaluation
The authors evaluated NDA across a wide variety of benchmarks, including CIFAR-10/100 for general classification and specialized datasets like Stanford-Dogs, CUB-200-2011, and Stanford-Cars for fine-grained tasks.
Key Results
- Significant Baseline Improvements: On CIFAR-10, NDA improved AlexNet's accuracy from a baseline of 60.2% to 70.9%, a massive 10.7% jump.
- State-of-the-Art in Fine-Grained Tasks: Using ResNet and Inception backbones, NDA achieved near-perfect results on several datasets, including 99.9% accuracy on Stanford-Cars.
- Robust OOD Performance: NDA outperformed popular methods like ODIN and MC Dropout in detecting out-of-distribution samples, approaching the performance of Deep Ensembles at a fraction of the computational cost.
Discussion and Insights
Beyond raw accuracy, NDA provides several qualitative benefits. The $L_{Mean}$ loss provides a stable reference point that prevents the Siamese loss from making the latent space chaotic during training. This leads to significantly reduced variance between different training runs, making models more reliable.
Furthermore, NDA improves model calibration for OOD detection. Standard networks are often overconfident on data they have never seen; NDA-trained networks tend to assign lower confidence to anomalous data, making them safer for real-world deployment.
Future work may explore applying NDA to earlier convolutional layers rather than just the pre-logit layer, and further automating the tuning of the loss weighting hyperparameters ($\alpha, \beta, \gamma$).
Summary updated