Linear Discriminant Analysis (LDA) is a cornerstone of supervised dimensionality reduction. Its goal is simple: find a low-dimensional subspace where data points from different classes are as far apart as possible, while points within the same class are clustered tightly together.
To achieve this, classical LDA maximizes the ratio of between-class scatter ($S_B$) to within-class scatter ($S_W$). The between-class scatter matrix is defined as:
$$S_B = \sum_{c=1}^C N_c (\mu_c - \mu)(\mu_c - \mu)^T$$
where $\mu_c$ is the mean of class $c$, $\mu$ is the global mean, and $N_c$ is the number of samples in class $c$.
Mathematically, $S_B$ is constructed from the outer products of vectors pointing from the global mean to each of the $C$ class centroids. Because these $C$ vectors are linearly dependent (they sum to zero when weighted by class size), the rank of $S_B$ is strictly bounded by $C - 1$. When solving the classic generalized eigenvalue problem $S_B v = \lambda S_W v$ to find the optimal projection vectors, you can obtain at most $C - 1$ eigenvectors with non-zero eigenvalues.
This creates a severe rank bottleneck. If you are working on a binary classification problem ($C = 2$), traditional LDA forces you to project your data onto a single dimension ($2 - 1 = 1$). If your data features complex, multi-dimensional variations or non-linear structures, flattening it onto a single line destroys vital discriminative information.
To preserve rich, high-dimensional structures in low-class scenarios, a prominent research direction focuses on rank-unconstrained discriminant dimensionality reduction. By redefining how class separation is measured, these techniques break the $C - 1$ ceiling, allowing projection into arbitrary dimensions.
Shifting from Global Centroids to Local Neighborhoods
The fundamental reason for the rank bottleneck is the reliance on a single global mean $\mu_c$ per class. If we instead calculate distances locally, the number of reference points scales with the number of samples or neighborhoods rather than the number of classes, naturally expanding the rank of the scatter matrices.
In Localized Sliced Inverse Regression, Qiang Wu 0003 et al. replace global class averages with local neighborhood means. By calculating a distinct local mean for every data point, the resulting between-class covariance matrix is no longer constrained to a rank of $C - 1$. This allows the model to identify multiple informative projection directions even in binary classification tasks.
Similarly, Feiping Nie 0001 et al. introduce Neighborhood MinMax Projections, which optimizes local pairwise distances rather than class averages. This focus on local neighborhood structures yields output spaces that can freely exceed the number of classes, providing significantly more descriptive power for non-Gaussian data distributions.
Taking this to its logical conclusion, non-parametric methods abandon class distributions altogether. Seyed Omid Sadjadi et al. develop a non-parametric discriminant analysis framework in Nearest neighbor discriminant analysis for robust speaker recognition. By evaluating scatter matrices based on $k$-nearest neighbors rather than class centroids, they generate full-rank scatter matrices. As the authors demonstrate in The IBM 2016 Speaker Recognition System, maintaining a full-rank feature space is essential for tasks with a small number of classes where retaining high-dimensional information is necessary for downstream classification.
Modeling Classes as Multimodal Sub-Clusters
Another intuitive way to break the rank constraint is to acknowledge that real-world classes are rarely single, uniform Gaussian "blobs." If a class contains multiple sub-clusters, we can treat each sub-cluster as an independent entity during scatter calculation.
Fernando De la Torre et al. leverage this in Multimodal oriented discriminant analysis. By incorporating sub-class clustering, the effective number of centroids increases, allowing the model to extract a number of features exceeding the traditional $C - 1$ limit. Yu Zhang 0006 et al. formalize this structural expansion in Multimodal Linear Discriminant Analysis via Structural Sparsity. Modeling classes as multimodal distributions mathematically expands the rank of the between-class scatter matrix, enabling projections into higher-dimensional embedding spaces that preserve complex sub-class relationships.
Traditional LDA (Single Centroid per Class):
Class 1 (O) --------> μ1 \
+---> Rank bounded by C - 1 (1 Dimension)
Class 2 (X) --------> μ2 /
Multimodal / Sub-cluster Discriminant Analysis:
Class 1 (O) --------> Sub-cluster Centroids (μ1a, μ1b, μ1c) \
+---> Rank expanded to total sub-clusters - 1
Class 2 (X) --------> Sub-cluster Centroids (μ2a, μ2b) /
Instead of explicit clustering, Hitoshi Sakano et al. propose the Extended Fisher Criterion Based on Auto-correlation Matrix Information. Their approach integrates multiple eigenvectors from class-specific auto-correlation matrices directly into the between-class scatter calculation. This captures the internal "shape" and orientation of each class, increasing the mathematical rank of the optimization objective and lifting the dimensionality constraint. This concept also extends to unsupervised scenarios; Qianqian Wang 0001 et al. show in Enhanced Unsupervised Discriminant Dimensionality Reduction for Nonlinear Data that removing the rank constraint allows data to be projected into subspaces with higher dimensions than the total number of clusters, capturing intricate, detailed feature sets.
Reformulating the Optimization Objective
The $C - 1$ limit is inextricably linked to the generalized eigenvalue problem. If we change the mathematical formulation of the optimization objective, we can decouple the projection dimensionality from the class count entirely.
Pairwise Distance and Harmonic Means
Instead of maximizing the distance between individual class means and a global mean, we can optimize the distances between all pairs of classes. Shuai Zheng 0002 et al. introduce A Harmonic Mean Linear Discriminant Analysis for Robust Image Classification, which replaces the standard arithmetic mean of class separations with a harmonic mean of pairwise class distances. Because the harmonic mean prioritizes separating classes that are closest together, it avoids the dominance of already well-separated classes. Crucially, this optimization framework decouples the projection objective from the rank of the between-class scatter matrix, enabling dimensionality reduction into an arbitrary number of dimensions.
Trace Ratio Optimization
Traditional LDA optimizes the determinant ratio $\max |W^T S_B W| / |W^T S_W W|$, which requires matrix inversion and is bound by the rank of $S_B$. In Locality Adaptive Discriminant Analysis, Xuelong Li 0001 et al. frame dimensionality reduction as a trace ratio problem:
$$\max \frac{\text{Tr}(W^T S_B W)}{\text{Tr}(W^T S_W W)}$$
This mathematical shift avoids matrix inversion entirely and bypasses the rank limitations associated with class counts, allowing for more projection directions than the number of classes.
Max-Min and Worst-Case Frameworks
Rather than optimizing average behavior, we can optimize for the worst-case scenario. Bing Su 0001 et al. propose Heteroscedastic max-min distance analysis, which uses semi-definite programming to maximize the minimum Chernoff distance between any two classes. This formulation generates discriminative subspaces that easily exceed $C - 1$ dimensions. Similarly, Chunhua Shen et al. demonstrate in Worst Case Linear Discriminant Analysis as Scalable Semidefinite Feasibility Problems that worst-case discriminant frameworks are completely unrestricted by the number of classes, allowing for projections into any dimension up to the original feature count. On a macro level, Shi Qiu shows in Multi-View Auxiliary Diagnosis Algorithm for Lung Nodules that modifying the inter-class dispersion matrix within linear discriminant analysis successfully overcomes the rank limit bottleneck when distinguishing highly similar categories.
Matrix Decomposition and Alternative Scatter Metrics
A final set of approaches bypasses the rank bottleneck by altering the core matrices under evaluation or decomposing the projection operator itself.
Total Scatter Matrix Substitution
The total scatter matrix $S_T = S_B + S_W$ captures the variance of all individual samples around the global mean, making it a full-rank matrix under normal conditions. In Two-fold regularization for kernel Fisher discriminant analysis in face recognition, Sangki Kim et al. utilize the total scatter matrix instead of the between-class scatter matrix in their optimization objective. This modification expands the mathematical rank of the criterion, allowing the model to extract a larger number of features than the traditional $C - 1$ limit.
Projection Matrix Decomposition
Instead of learning a single projection matrix constrained by class labels, we can break the matrix into independent components. Zhihui Lai et al. introduce Generalized Robust Regression for Jointly Sparse Subspace Learning, where they decompose the projection matrix into two distinct components to decouple the number of extracted features from the total number of data categories. This resolves the rank bottleneck inherent in traditional regression-based dimensionality reduction, enabling detailed data representation in small-sample-size scenarios.
Feature and Variable Partitioning
Advanced correlation and partitioning strategies offer another pathway to high-dimensional feature retention:
- Correlation Analysis: Z. Wu et al. utilize advanced discriminative correlation analysis in Enhanced feature fusion through irrelevant redundancy elimination in intra-class and extra-class discriminative correlation analysis to retain a higher number of feature dimensions than the standard class-count limit, preserving intricate data relationships during feature fusion.
- Variable Clustering: In Clustered Discriminant Regression for High-Dimensional Data Feature Extraction and Its Applications in Healthcare and Additive Manufacturing, Bo Shen et al. partition variables into multiple clusters. This allows for the extraction of a significantly higher number of features than traditional discriminant regression, which is particularly advantageous in binary classification tasks where standard methods are limited to a single feature.
Deep Neural Embeddings
Finally, we can transcend linear projections entirely. Shuai Wang et al. introduce Deep Discriminant Analysis for i-vector Based Robust Speaker Recognition, replacing linear projections with deep neural networks trained on joint softmax and center losses. These neural-based non-linear transformations allow for completely arbitrary output dimensionality, permitting the preservation of rich information in the embedding space even when categories are numerous but samples per category are scarce.
Key Foundational Work
Feature Extraction Methods for Real-Time Face Detection and Classification (2005) — contributed 1 concept elements to this direction.
Multimodal oriented discriminant analysis (2005) — contributed 1 concept elements to this direction.
Neighborhood MinMax Projections (2007) — contributed 1 concept elements to this direction.