Report on 2012.11841v2

Ye-Ming Meng, Jing Zhang, Peng Zhang, Chao Gao, Shi-Ju RanOpenalex Unknown 2022 · DOI · arXiv · OpenAlex

Abstract

Tensor network, which originates from quantum physics, is emerging as an efficient tool for classical and quantum machine learning. Nevertheless, there still exists a considerable accuracy gap between tensor network and the sophisticated neural network models for classical machine learning. In this work, we combine the ideas of matrix product state (MPS), the simplest tensor network structure, and residual neural network and propose the residual matrix product state (ResMPS). The ResMPS can be treated as a network where its layers map the "hidden" features to the outputs (e.g., classifications), and the variational parameters of the layers are the functions of the features of the samples (e.g., pixels of images). This is different from neural network, where the layers map feed-forwardly the features to the output. The ResMPS can equip with the non-linear activations and dropout layers, and outperforms the state-of-the-art tensor network models in terms of efficiency, stability, and expression power. Besides, ResMPS is interpretable from the perspective of polynomial expansion, where the factorization and exponential machines naturally emerge. Our work contributes to connecting and hybridizing neural and tensor networks, which is crucial to further enhance our understand of the working mechanisms and improve the performance of both models.

TL;DR

The Residual Matrix Product State (ResMPS) is a novel architecture that merges the interpretability of physics-inspired Tensor Networks with the performance-boosting "shortcuts" of Residual Neural Networks. By using data features to parameterize the network layers rather than just passing them through, ResMPS achieves state-of-the-art accuracy for Tensor Networks while remaining mathematically interpretable through polynomial expansion.

What Problem Does This Paper Address?

Tensor Networks (TNs), particularly the Matrix Product State (MPS), are powerful mathematical tools originally designed to describe quantum many-body systems. In recent years, they have gained traction in machine learning because they are highly interpretable and efficient. However, a significant "accuracy gap" remains: standard TNs often underperform compared to sophisticated deep neural networks (NNs) on classical tasks like image classification.

The challenge lies in the fact that TNs are essentially linear maps. While researchers have tried to hybridize them with NNs (e.g., using a CNN as a feature extractor for a TN), it has been difficult to incorporate core deep learning "tricks"—like residual connections, non-linear activations, and dropout—directly into the TN structure without losing its unique properties.

Methods

The authors introduce ResMPS, which rethinks how data enters a network. In a traditional neural network, data flows through layers. In ResMPS, the data features (like image pixels $x_i$) are used to parameterize the layers themselves.

Architecture of the ResMPS model showing how input features parameterize the layers.

As shown in the figure above, the architecture is built on a residual structure where the hidden state $h^{[n]}$ is updated by a "shift vector" $v^{[n]}$ determined by the $n$-th feature of the input. This creates an "information highway" that allows gradients to flow easily, preventing the vanishing gradient problem common in deep tensor models.

The paper explores two specific versions:

  1. Simple ResMPS (sResMPS): A multi-linear model that is mathematically equivalent to a standard MPS but uses only half the parameters.
  2. Activated ResMPS (aResMPS): A non-linear version that incorporates ReLU activations and dropout layers on the residual terms, significantly boosting generalization power.

Datasets

The models were evaluated on two standard computer vision benchmarks:

  • MNIST: The classic dataset of handwritten digits.
  • Fashion-MNIST: A more challenging dataset of clothing items (shirts, boots, etc.), used to test the model's ability to handle complex patterns.

Key Insights and Ideas

  • Data as Parameters: The most striking idea is that the input data determines the weights of the network for that specific sample. This allows the network to encode sample information into a "path" in high-dimensional space.
  • Channel Redundancy: The authors discovered that standard MPS models contain significant redundancy. Their sResMPS achieves identical accuracy to a standard MPS while discarding the "inactive" channels, effectively halving the parameter count.
  • The Identity Shortcut: By initializing the residual terms near zero, the network starts as an identity map. This "identity parameterization" is crucial for stable training in very deep architectures (hundreds of layers).
  • Mathematical Bridge: ResMPS can be expanded into a polynomial form. The authors show that this expansion naturally gives rise to Factorization Machines and Exponential Machines, providing a theoretical link between quantum-inspired models and classical machine learning techniques.

Key Results

ResMPS demonstrates significant improvements over existing tensor-based approaches. As shown in the table below, the aResMPS model achieves state-of-the-art results for tensor networks on the Fashion-MNIST dataset, even outperforming some classical neural network architectures like AlexNet.

Performance comparison of ResMPS against other state-of-the-art tensor network and neural network models.
  • Superior Accuracy: On Fashion-MNIST, aResMPS achieved a testing accuracy of 91.46%, outperforming other state-of-the-art Tensor Network models like CNN-PEPS (91.2%) and Bayesian TN (86.9%).
  • Efficiency: sResMPS matched the performance of a standard MPS machine but with a 50% reduction in parameter complexity.
  • Stability: Unlike standard deep TNs, ResMPS converged quickly and stably even with hundreds of tensor cores, thanks to the residual "highways."

Limitations and Future Work

While ResMPS outperforms other Tensor Networks and simple NNs like AlexNet, it still lags slightly behind the most sophisticated ResNets (which use stacked convolutional layers and reach ~93%+ on Fashion-MNIST).

The authors suggest several promising directions for future research, including replacing current weight matrices with convolutional layers within the ResMPS framework and extending the residual idea to more complex topologies like Tree Tensor Networks or PEPS.

Summary updated

Abstract

Tensor network, which originates from quantum physics, is emerging as an efficient tool for classical and quantum machine learning.Nevertheless, there still exists a considerable accuracy gap between tensor network and the sophisticated neural network models for classical machine learning.In this work, we combine the ideas of matrix product state (MPS), the simplest tensor network structure, and residual neural network and propose the residual matrix product state (ResMPS).The ResMPS can be treated as a network where its layers map the "hidden" features to the outputs (e.g., classifications), and the variational parameters of the layers are the functions of the features of the samples (e.g., pixels of images).This is different from neural network, where the layers map feed-forwardly the features to the output.The ResMPS can equip with the non-linear activations and dropout layers, and outperforms the state-of-the-art tensor network models in terms of efficiency, stability, and expression power.Besides, ResMPS is interpretable from the perspective of polynomial expansion, where the factorization and exponential machines naturally emerge.Our work contributes to connecting and hybridizing neural and tensor networks, which is crucial to further enhance our understand of the working mechanisms and improve the performance of both models.
BibTeX
@inproceedings{meng2022report,
  author = {Ye-Ming Meng and Jing Zhang and Peng Zhang and Chao Gao and Shi-Ju Ran},
  title = {{Report on 2012.11841v2}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.21468/scipost.report.4511},
}
Figure