Common Mode Patterns for Supervised Tensor Subspace Learning

Konstantinos Makantasis, Anastasios D. Doulamis, Nikolaos Doulamis, Athanasios Voulodimos

TL;DR

This paper introduces Common Mode Patterns (CMP), a novel supervised learning method for reducing the dimensionality of multi-dimensional data (tensors) in binary classification tasks. CMP effectively separates data from different classes in the reduced-dimensional space, leading to improved discriminative power compared to unsupervised methods like MPCA.

What Problem Does This Paper Address?

Modern sensing technologies generate massive, high-dimensional datasets, often represented as tensors. Effectively analyzing this data requires reducing its dimensionality while preserving or even enhancing its discriminative features for classification. Traditional unsupervised subspace learning methods, like Multilinear Principal Component Analysis (MPCA), aim to capture the most variance in the data but don't explicitly focus on separating different classes. This can lead to features that are common across classes, hindering classification performance. The paper addresses this gap by proposing a supervised approach that leverages class labels to ensure that the reduced-dimensional features are maximally separable between classes.

Methods

The core of the paper is the Common Mode Patterns (CMP) algorithm. CMP is inspired by the Common Spatial Patterns (CSP) algorithm, which is effective for 2D data but not directly applicable to higher-order tensors. CMP extends the CSP concept to tensors by extracting features that maximize the separability between two classes.

Key aspects of the method include:

  • Supervised Subspace Learning: Unlike unsupervised methods, CMP uses class labels to guide the feature extraction process.
  • Common Mode Patterns: The algorithm aims to find projection matrices for each mode (dimension) of the tensor such that the projected data from different classes are maximally separated. This is achieved by maximizing the inter-class scatter while minimizing the intra-class scatter.
  • Normalization Process: Before projection, a normalization step is applied to the tensor data. This involves calculating mode-specific scatter matrices for each class and then performing a transformation based on their eigenvalues and eigenvectors. This step is crucial for preparing the data for the subsequent projection.
  • Iterative Algorithm: The CMP algorithm iteratively estimates projection matrices for each tensor mode. The process continues until a termination criterion is met.
  • Feature Separation: A key theoretical insight is that the projection matrices derived by CMP ensure that features important for one class are deemphasized for the other, thus preventing overlap in the reduced-dimensional space.

Datasets

The paper uses the Pavia University hyperspectral imaging dataset for experimental validation. This dataset contains 103 spectral bands and has 9 classes. For their binary classification experiments, the authors grouped pixels depicting man-made objects (asphalt, metal sheets, bricks, bitumen) against all other pixels. The training set consisted of 200 samples per class.

Key Insights and Ideas

  • Supervision is Key for Discriminative Features: The paper strongly advocates for supervised subspace learning when the goal is classification. Simply reducing dimensionality based on overall data variance (unsupervised) is less effective than using labels to find features that explicitly separate classes.
  • CMP Maximizes Inter-Class Separability: The core idea is to find projections that make the classes as distinct as possible in the lower-dimensional space, rather than just preserving the most information.
  • Features for One Class are Not Features for Another: A significant theoretical outcome is that the CMP projection matrices are designed such that features that are discriminative for class 1 are deemphasized for class 2, and vice-versa. This is a powerful property for binary classification.
  • Extension of CSP to Tensors: The work successfully bridges the gap of applying CSP-like principles to multi-dimensional tensor data, opening up new avenues for tensor-based analysis.

Key Results

The CMP method was compared against MPCA using three classifiers: Rank-1 Tensor Regression (Rank-1 TR), CNN, and Rank-1 FNN.

  • Improved Classification Accuracy: CMP consistently outperformed MPCA in terms of classification accuracy across different dimensionality reduction settings (e.g., reducing to 26, 10, or 5 spectral components). For instance, with 26 components and CNN, CMP achieved 90.41% accuracy compared to MPCA's 85.08%.
  • Efficient Dimensionality Reduction: The experimental results indicate that CMP can efficiently reduce the dimensionality of tensor objects while simultaneously increasing inter-class separability.
  • Rank-1 TR Performance: For the Rank-1 TR classifier, the performance difference between CMP and MPCA was minimal, which the authors attribute to the linear nature and limited capacity of Rank-1 TR.

Limitations and Future Work

  • Binary Classification Limitation: The primary limitation acknowledged is that the current CMP algorithm is designed specifically for binary classification problems.
  • Future Work: The authors plan to extend the CMP approach to handle multi-class classification problems. Additionally, they aim to evaluate CMP on more diverse datasets and compare it against other supervised tensor subspace learning methods.

Summary updated

Abstract

In this work we propose a method for reducing the dimensionality of tensor objects in a binary classification framework. The proposed Common Mode Patterns method takes into consideration the labels' information, and ensures that tensor objects that belong to different classes do not share common features after the reduction of their dimensionality. We experimentally validate the proposed supervised subspace learning technique and compared it against Multilinear Principal Component Analysis using a publicly available hyper-spectral imaging dataset. Experimental results indicate that the proposed CMP method can efficiently reduce the dimensionality of tensor objects, while, at the same time, increasing the inter-class separability.
BibTeX
@inproceedings{makantasis2019common,
  author = {Konstantinos Makantasis and Anastasios D. Doulamis and Nikolaos Doulamis and Athanasios Voulodimos},
  title = {{Common Mode Patterns for Supervised Tensor Subspace Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ZL6VXd281D},
}
Figure