Complex Moment-Based Supervised Eigenmap for Dimensionality Reduction

Akira Imakura, Momo Matsuda, Xiucai Ye, Tetsuya SakuraiAAAI 2019 · DOI

Abstract

Dimensionality reduction (DR) is a fundamental task in data science, yet traditional methods often suffer from significant information loss or poor scalability. We present Complex Moment-Based Supervised Eigenmap (CMSE), a novel hybrid approach that integrates the structural preservation of matrix trace optimization with the discriminative power of ridge regression. By leveraging complex moment-based parallel eigensolvers, CMSE captures a richer set of eigenvectors within a specified subspace, enhancing classification accuracy and computational efficiency. Experimental results on diverse real-world datasets demonstrate that CMSE consistently outperforms existing kernel-based DR techniques.

Introduction

High-dimensional data analysis requires effective dimensionality reduction to facilitate clustering and classification. However, conventional techniques like Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) typically rely on a minimal number of eigenvectors, often discarding critical data properties. Furthermore, these methods frequently struggle to balance data structure preservation with label-based mapping. To address these gaps, researchers from the University of Tsukuba introduced CMSE, which utilizes a novel objective function and a complex moment-based subspace to provide a more robust and scalable DR framework.

Methodology

The CMSE algorithm is built upon a dual-objective function that optimizes for both trace preservation and regression accuracy.

Hybrid Objective Function

CMSE balances two primary goals:

  1. Trace Optimization: Minimizing a weighted matrix trace to maintain the inherent local structure of the data, similar to Locality Preserving Projections (LPP).
  2. Ridge Regression: Minimizing the squared error between projected features and ground truth labels to ensure the resulting low-dimensional space is highly discriminative.

A weight parameter, $\mu$, allows the model to transition between these two objectives, enabling a tuned hybrid approach that captures the benefits of both.

Complex Moment-Based Subspace

A key innovation in CMSE is the use of complex moment-based parallel eigensolvers. Instead of solving for individual eigenvectors, the method employs contour integrals in the complex plane to define a subspace $\mathcal{S}_\Omega$. This subspace includes multiple eigenvectors within a specific eigenvalue range $\Omega$, preventing the "information bottleneck" seen in methods like LDA, which are limited by the number of classes. This formulation also makes the algorithm highly parallelizable, as it involves solving independent linear systems at various quadrature points.

Experimental Evaluation

The performance of CMSE was evaluated using both artificial noisy datasets and eight real-world benchmarks spanning biological, image, and speech data (e.g., MNIST, GLIOMA, TOX-171).

Accuracy and Robustness

In comparative tests, the kernelized version (K-CMSE) demonstrated superior accuracy (ACC) and robustness. For instance, on the TOX-171 dataset, K-CMSE achieved 96.50% accuracy, significantly outperforming K-LPP's 50.92%.

Comparison of K-CMSE against other kernel methods across varying k-nearest neighbor values.

As shown in the figure above, K-CMSE maintains a consistently higher accuracy compared to K-LPP, K-LFDA, and K-RR as the $k$-nearest neighbor parameter varies, highlighting its stability and superior discriminative power.

Discussion

The effectiveness of CMSE is dependent on its hyperparameters, including the weight $\mu$ and the number of input vectors $L$.

Sensitivity analysis of K-CMSE performance with respect to the linear combination parameter mu and the number of input vectors L.

The sensitivity analysis reveals that an optimal balance (around $\mu = 0.25$) significantly improves performance over pure regression or pure trace-based methods. Additionally, the accuracy stabilizes as the number of input vectors $L$ increases, suggesting that the method effectively captures the necessary subspace information with a moderate number of vectors.

Conclusion

CMSE offers a powerful and scalable alternative for supervised dimensionality reduction. By integrating complex moment-based eigensolvers with a hybrid objective function, it overcomes the limitations of traditional DR methods regarding information loss and parallel efficiency. Future work will explore relaxing the orthogonal constraints in the underlying Procrustes problem to further streamline computation and integrating a broader range of trace optimization techniques.

Summary updated

BibTeX
@inproceedings{imakura2019complex,
  author = {Akira Imakura and Momo Matsuda and Xiucai Ye and Tetsuya Sakurai},
  title = {{Complex Moment-Based Supervised Eigenmap for Dimensionality Reduction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33013910},
}
Figure