Abstract
Discriminant analysis, as a widely used approach in machine learning to extract low-dimensional features from the high-dimensional data, applies the Fisher discriminant criterion to find the orthogonal discriminant projection subspace. But most of the Euclidean-based algorithms for discriminant analysis are easily convergent to a spurious local minima and hardly obtain an unique solution. To address such problem, in this study we propose a novel method named Riemannian-based Discriminant Analysis (RDA), which transforms the traditional Euclidean-based methods to the Riemannian manifold space. In RDA, the second-order geometry of trust-region methods is utilized to learn the discriminant bases. To validate the efficiency and effectiveness of RDA, we conduct a variety of experiments on image classification tasks. The numerical results suggest that RDA can extract statistically significant features and robustly outperform state-of-the-art algorithms in classification tasks.
Introduction
Linear Discriminant Analysis (LDA) is a fundamental technique in machine learning used to find a low-dimensional subspace that best separates different classes of data. However, traditional Euclidean-based LDA algorithms face two major hurdles: the tendency to converge to spurious local minima and the singularity problem, where the within-class scatter matrix becomes non-invertible in high-dimensional, low-sample-size scenarios.
Riemannian-based Discriminant Analysis (RDA)
To address these challenges, we propose Riemannian-based Discriminant Analysis (RDA). This framework moves traditional LDA from flat Euclidean space onto curved Riemannian manifolds, specifically the Stiefel and Grassmann manifolds. These mathematical spaces inherently respect the orthogonality constraints required for projection bases, making the optimization process more natural and effective.
Optimization and Regularization
RDA introduces several key innovations to the discriminant analysis pipeline:
- Subtractive Loss Function: Instead of the traditional ratio-based Fisher criterion, RDA uses a subtractive form. This allows for efficient calculation of Riemannian gradients and Hessians without the need for matrix inversion, effectively solving the singularity problem.
- Second-Order Optimization: RDA employs a Riemannian trust-region method. By utilizing second-order geometric information (the Hessian) on the tangent space, the algorithm can navigate complex loss landscapes that often trap first-order methods.
- Sparsity Regularization: An $\ell_1$-norm penalty is incorporated into the manifold optimization to prevent overfitting and improve generalization to unseen data.
Experimental Evaluation
Datasets
The performance of RDA was evaluated across seven benchmark image datasets covering objects, handwritten digits, and faces. These datasets provide a rigorous test for dimensionality reduction across varying levels of complexity.
The datasets include:
- Objects: COIL20, ETH80.
- Digits: MNIST, USPS.
- Faces: ORL, Olivetti, CMU PIE (up to 50 classes).
Key Results
RDA demonstrated superior performance across nearly all benchmarks:
- Clustering: In $k$-means clustering tasks, RDA achieved significantly higher Accuracy (ACC) and Normalized Mutual Information (NMI) than Euclidean-based competitors. For instance, on the CMU PIE dataset, RDA achieved an ACC of 0.7866, while the next best Euclidean method (DATER) reached only 0.3764.
- Classification: Using a $k$-Nearest Neighbor (kNN) classifier, RDA-extracted features consistently led to higher classification accuracy compared to state-of-the-art multilinear and tensor-based methods.
Discussion and Conclusion
By treating the search for optimal projection bases as a geometric problem on curved manifolds, RDA overcomes the fundamental limitations of traditional LDA. The use of second-order trust-region methods ensures more robust convergence, while the subtractive loss function bypasses numerical instabilities.
While powerful, Riemannian optimization can be computationally intensive. Future work will focus on reducing the computational overhead of second-order methods and developing automated strategies for hyperparameter tuning and metric selection to further enhance the algorithm's efficiency.
Summary updated