Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification

Konstantinos Makantasis, Alexandros Georgogiannis, Athanasios Voulodimos, Ioannis Georgoulas, Anastasios D. Doulamis, Nikolaos Doulamis

TL;DR

This paper introduces the Rank-R Feedforward Neural Network (FNN), a novel tensor-based model for classifying high-order data. By leveraging Canonical/Polyadic (CP) decomposition on its weights, Rank-R FNN significantly reduces the number of trainable parameters, making it highly efficient for small-sample learning scenarios, particularly in hyperspectral image classification, while achieving state-of-the-art performance.

What Problem Does This Paper Address?

Modern data science and engineering are increasingly dealing with high-dimensional, structurally rich data, often represented as tensors. Traditional machine learning algorithms, which typically expect vectorized input, struggle with this complexity. Vectorizing tensor data destroys valuable structural information and leads to a massive increase in model parameters, making algorithms computationally expensive and prone to overfitting, especially when training data is scarce. This is a significant challenge in fields like hyperspectral image analysis, where acquiring and labeling large datasets is costly and time-consuming. The paper aims to bridge this gap by proposing a model that can effectively handle tensor inputs without losing structural integrity and with a significantly reduced parameter count.

Methods

The core innovation is the Rank-R Feedforward Neural Network (FNN). Unlike standard FNNs that vectorize inputs, Rank-R FNN directly processes data in its tensor form. The key lies in imposing a Canonical/Polyadic (CP) decomposition of rank R on the weights connecting the input layer to the first hidden layer. This decomposition decomposes the weight tensor into a sum of R outer products of vectors.

This CP decomposition offers two main advantages:

  1. Preservation of Structure: It allows the model to inherently exploit the multilinear structure of the input data across its different modes (dimensions).
  2. Parameter Reduction: It dramatically reduces the number of trainable parameters compared to a fully connected FNN. For instance, a fully connected layer with an input tensor of size $P_1 \times P_2 \times P_3$ and $h$ hidden neurons would have $h \times P_1 \times P_2 \times P_3$ parameters. A Rank-R FNN, under CP decomposition, reduces this to $h \times R \times (P_1 + P_2 + P_3)$ parameters (simplified for illustration).

The paper also establishes the theoretical properties of Rank-R FNN, proving its universal approximation capabilities and learnability via empirical risk minimization, similar to standard FNNs. The training is performed using an alternating optimization algorithm based on coordinate descent.

Datasets

The paper evaluates the proposed Rank-R FNN on three publicly available hyperspectral datasets:

  • Indian Pines: Captured by AVIRIS sensor, 224 spectral bands, 10,249 labeled pixels, 16 classes.
  • Pavia University: Captured by ROSIS sensor, 103 spectral bands, 42,776 labeled pixels, 9 classes.
  • Botswana: Captured by Hyperion sensor, 145 spectral bands, 3,248 labeled pixels, 14 classes.

These datasets are standard benchmarks for hyperspectral image classification. The authors use a patch-based approach for pixel-wise classification, where a $5 \times 5$ spatial patch centered on a pixel is used as input.

Key Insights and Ideas

  • Tensor Structure is Key: The paper strongly advocates for treating high-order data as tensors rather than vectors, highlighting the loss of information incurred by vectorization.
  • Rank-R as a Regularizer: The CP decomposition acts as a form of regularization, effectively reducing model complexity and preventing overfitting, especially crucial in small-sample settings.
  • Rank-1 FNN Limitations: While Rank-1 FNN (a special case of Rank-R FNN) offers parameter reduction, it has limitations in modeling complex relationships. Increasing the rank (R) enhances the model's representation power.
  • Robustness to Noise: Tensor-based models, due to their inherent structure and reduced parameterization, show remarkable robustness to noisy input data, a significant advantage over standard CNNs.
  • Rapid Convergence: Rank-R FNN models converge much faster than conventional CNNs, often within 20 epochs, regardless of the rank R.

Key Results

  • State-of-the-Art Performance: Rank-R FNN achieves state-of-the-art classification accuracy on hyperspectral datasets, particularly in small-sample scenarios.
  • Parameter Efficiency: The number of trainable parameters is drastically reduced, being 41-57 times smaller than conventional CNNs for the Indian Pines, Pavia University, and Botswana datasets.
  • Improved Accuracy: In noisy conditions and small-sample settings, Rank-R FNN consistently outperforms CNNs. For example, with 10 samples/class and 20% noise, Rank-R FNN shows improvements of 3.9% (Pavia), 3.3% (Botswana), and 1.6% (Indian Pines) over CNN after 500 epochs. With 50 samples/class and 20% noise, the improvements are even more substantial.
  • Robustness: The model exhibits significantly smaller standard deviations in accuracy across different runs, indicating greater stability and robustness compared to CNNs.
  • Faster Training: Rank-R FNN converges in fewer epochs (typically < 20) compared to CNNs (often > 200).

Limitations and Future Work

  • Rank Selection: The optimal rank R can be application-dependent, and determining it might require experimentation. The paper suggests that for Botswana, R=1 is best, while for Pavia and Indian Pines, R=2 is optimal in noisy conditions.
  • Future Work: The paper implicitly suggests exploring the application of Rank-R FNN to other high-order data domains beyond hyperspectral imagery. Further investigation into the theoretical underpinnings of rank selection and its impact on model performance could also be a future direction.

Summary updated

Abstract

An increasing number of emerging applications in data science and engineering are based on multidimensional and structurally rich data. The irregularities, however, of high-dimensional data often compromise the effectiveness of standard machine learning algorithms. We hereby propose the Rank- R Feedforward Neural Network (FNN), a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters, thereby offering two core advantages compared to typical machine learning methods. First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension. Moreover, the number of the model's trainable parameters is substantially reduced, making it very efficient for small sample setting problems. We establish the universal approximation and learnability properties of Rank- R FNN, and we validate its performance on real-world hyperspectral datasets. Experimental evaluations show that Rank- R FNN is a computationally inexpensive alternative of ordinary FNN that achieves state-of-the-art performance on higher-order tensor data.
BibTeX
@inproceedings{makantasis2021rank,
  author = {Konstantinos Makantasis and Alexandros Georgogiannis and Athanasios Voulodimos and Ioannis Georgoulas and Anastasios D. Doulamis and Nikolaos Doulamis},
  title = {{Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=goCqd6SJD0},
}
Figure