Multi-Layered Projected Entangled Pair States for Image Classification

Lei Li, Hong LaiSustainability 2023 · DOI

TL;DR

This paper introduces Multi-Layered Projected Entangled Pair States (MLPEPS), a novel tensor network architecture for image classification. MLPEPS builds upon the strengths of Projected Entangled Pair States (PEPS) by introducing a hierarchical, multi-layered structure that captures both local pixel correlations and global image features, achieving competitive accuracy with significantly fewer parameters than traditional deep learning models.

What Problem Does This Paper Address?

Image classification is a cornerstone of computer vision, with deep convolutional neural networks (CNNs) currently dominating the field. However, CNNs often require very deep architectures, leading to a large number of parameters, computational complexity, and potential issues like gradient vanishing/exploding. Tensor networks, originating from quantum physics, offer a promising alternative due to their ability to efficiently represent complex correlations and avoid the "curse of dimensionality." While simpler tensor networks like Matrix Product States (MPS) have been explored, they struggle with preserving the 2D structure of images, leading to information loss. Projected Entangled Pair States (PEPS) are better suited for 2D data but can still face limitations in capturing hierarchical features. This paper aims to bridge the gap between the performance of mature neural networks and the efficiency of tensor networks by developing a hierarchical PEPS-based model.

Methods

The core innovation is the Multi-Layered Projected Entangled Pair States (MLPEPS) model. It extends the single-layer PEPS architecture by stacking multiple PEPS layers hierarchically. Each layer consists of identical PEPS blocks arranged in a lattice. The output of one layer serves as the input to the next, allowing for progressive abstraction of image features.

Key methodological aspects include:

  • Hierarchical Feature Extraction: Lower PEPS layers capture local image information, while upper layers process these features to extract more abstract representations, mimicking the hierarchical nature of deep learning.
  • PEPS Structure: PEPS is chosen for its inherent ability to represent 2D structures, preserving spatial correlations better than MPS.
  • Boundary MPS Approximation: To handle the computational complexity of contracting large PEPS, the authors employ the boundary MPS method, which approximates the contraction by treating the top and bottom rows as MPS and the rest as Matrix Product Operators. This reduces exponential complexity to polynomial.
  • Stochastic Gradient Descent and Backpropagation: Model parameters are optimized using standard backpropagation, with a custom SVD backpropagation process to address numerical instability issues encountered with PEPS.
  • Feature Mapping: Input images are first processed through a feature map (e.g., using sine and cosine functions) to embed pixel values into a higher-dimensional Hilbert space, and potentially a "squeeze operation" to group pixels into blocks for initial feature extraction.

Datasets

The proposed MLPEPS model was evaluated on two standard image classification datasets:

  • Fashion-MNIST: A dataset of grayscale images of clothing items, consisting of 60,000 training and 10,000 test images across 10 classes. This dataset was used to compare MLPEPS against single-layer PEPS and other tensor network models.
  • COVID-19 Radiography Dataset: A more complex dataset containing chest X-ray images for classifying COVID-19 positive cases, lung opacity, normal cases, and viral pneumonia. This dataset was used to assess MLPEPS's generalization ability against well-known neural networks.

Key Insights and Ideas

  • Hierarchy is Key for Tensor Networks: Just as deep learning benefits from layered architectures, stacking PEPS layers (MLPEPS) significantly improves performance over single-layer PEPS by enabling hierarchical feature abstraction.
  • PEPS for Structure Preservation: PEPS's 2D nature is crucial for image data, allowing it to capture spatial relationships that 1D tensor networks like MPS would lose.
  • Computational Efficiency via Approximation: The boundary MPS method is essential for making PEPS-based models computationally tractable for larger images and deeper networks, bridging the gap between theoretical potential and practical application.
  • Parameter Efficiency: MLPEPS demonstrates that tensor networks can achieve competitive accuracy with substantially fewer parameters than comparable neural networks, hinting at their potential for resource-constrained environments.
  • Overfitting as a Challenge: While MLPEPS shows strong learning ability, its generalization performance on complex datasets can be weaker than mature neural networks, suggesting overfitting is a significant area for future research in tensor network models.

Key Results

  • Fashion-MNIST: MLPEPS achieved a test set accuracy of 90.44%, outperforming single-layer PEPS (88.30%) and other tensor network models, and even surpassing AlexNet (89.90%).
  • COVID-19 Radiography Dataset: MLPEPS achieved a test set accuracy of 91.63%, which is very close to GoogLeNet (92.75%) and significantly better than single-layer PEPS.
  • Parameter Count: MLPEPS demonstrated remarkable parameter efficiency, requiring only 1/90th the parameters of VGG-16 for comparable performance on the COVID-19 dataset.
  • Training vs. Test Accuracy: MLPEPS achieved 100% training accuracy on the COVID-19 dataset, highlighting its strong fitting ability, but its test accuracy lagged behind some neural networks, indicating potential overfitting.

Limitations and Future Work

  • Generalization Ability: The paper acknowledges that MLPEPS's generalization ability on complex datasets is weaker than mature neural networks, suggesting a need to address overfitting.
  • Network Structure Exploration: The authors note that the optimal structure of MLPEPS for different tasks is still an open question. They propose exploring various PEPS block sizes and layer configurations.
  • Avoiding Overfitting: Future work should investigate regularization techniques and methods for avoiding overfitting in deeper PEPS structures, potentially drawing inspiration from deep learning architectures like residual and dense networks.
  • Lightweight Networks: The parameter efficiency of MLPEPS makes it a strong candidate for lightweight networks suitable for portable devices.

Summary updated

Abstract

Tensor networks have been recognized as a powerful numerical tool; they are applied in various fields, including physics, computer science, and more. The idea of a tensor network originates from quantum physics as an efficient representation of quantum many-body states and their operations. Matrix product states (MPS) form one of the simplest tensor networks and have been applied to machine learning for image classification. However, MPS has certain limitations when processing two-dimensional images, meaning that it is preferable for an projected entangled pair states (PEPS) tensor network with a similar structure to the image to be introduced into machine learning. PEPS tensor networks are significantly superior to other tensor networks on the image classification task. Based on a PEPS tensor network, this paper constructs a multi-layered PEPS (MLPEPS) tensor network model for image classification. PEPS is used to extract features layer by layer from the image mapped to the Hilbert space, which fully utilizes the correlation between pixels while retaining the global structural information of the image. When performing classification tasks on the Fashion-MNIST dataset, MLPEPS achieves a classification accuracy of 90.44%, exceeding tensor network models such as the original PEPS. On the COVID-19 radiography dataset, MLPEPS has a test set accuracy of 91.63%, which is very close to the results of GoogLeNet. Under the same experimental conditions, the learning ability of MLPEPS is already close to that of existing neural networks while having fewer parameters. MLPEPS can be used to build different network models by modifying the structure, and as such it has great potential in machine learning.
BibTeX
@inproceedings{li2023multi,
  author = {Lei Li and Hong Lai},
  title = {{Multi-Layered Projected Entangled Pair States for Image Classification}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su15065120},
}
Figure