Tensor networks (TNs), rooted in the study of quantum many-body systems, have emerged as a promising framework for machine learning. Among them, the Matrix Product State (MPS) is the most widely used due to its efficiency in representing high-dimensional data. However, a significant performance gap remains between standard TN models and state-of-the-art deep neural networks (NNs). This gap is primarily attributed to issues with gradient stability in deep TNs and the lack of hierarchical feature integration.
This report summarizes the proposal of the Residual Matrix Product State (ResMPS), which bridges this gap by incorporating residual connections and non-linearities—key components of successful deep learning architectures—into the MPS framework.
The ResMPS Architecture
The fundamental innovation of ResMPS lies in how data is processed. In traditional feed-forward neural networks (FNNs), data is typically input only at the initial layer. In contrast, ResMPS uses input features (e.g., image pixels) to parameterize the weights of the network layers themselves.
Architecture of the Residual Matrix Product State (ResMPS), showing (a) the overall flow with residual connections, (b) the corresponding tensor network diagram, and (c, d, e) details of simple and activated layer variants.
As shown in Figure 3, the ResMPS architecture introduces "information highways" or shortcuts. A layer in ResMPS is defined by the transformation:
$$h^{[n]} = h^{[n-1]} + \sigma(L^{[n]}(h^{[n-1]}; W^{[n]}(x_n)) + b^{[n]})$$
where $h^{[n-1]}$ is the hidden state from the previous layer, $x_n$ is the $n$-th input feature, and $W^{[n]}$ is a weight matrix that is a function of $x_n$. The authors propose two versions:
sResMPS (Simple): A multi-linear model that maintains mathematical equivalence to a standard MPS while utilizing a more efficient residual structure.
aResMPS (Activated): A non-linear model that incorporates ReLU activations and dropout layers to enhance representation power and stability.
Experimental Results and Performance
The ResMPS models were benchmarked on the MNIST and Fashion-MNIST datasets. The results demonstrate that ResMPS significantly outperforms traditional TN models and even some established neural network architectures.
On the Fashion-MNIST dataset, the activated ResMPS (aResMPS) achieved an accuracy of 91.42%, surpassing standard TN models (which typically hover around 88-89%) and the AlexNet neural network (88.82%). While it still slightly trails the ResNet (93.39%), the gap is significantly narrowed.
One of the most important findings is the efficiency of the sResMPS. It achieves the same accuracy as a standard MPS but requires 50% fewer parameters. This is achieved by identifying and removing redundant "channels" within the tensors that do not contribute to the final classification.
Insights into Stability and Interpretability
The residual structure directly addresses the vanishing gradient problem. By treating the hidden state updates as small perturbations (vectors $v^{[n]}$ added to $h^{[n-1]}$), the network maintains a stable signal flow even as depth increases.
Visualization of ResMPS dynamics: (a) hidden state updates as vectors, (b) path visualization in feature space, and (c) t-SNE clustering of learned representations.
As illustrated in Figure 4, the model effectively clusters data in the hidden feature space, facilitating accurate classification. Furthermore, the authors demonstrate that ResMPS is highly interpretable through polynomial expansion. The model can be decomposed into a series of terms representing interactions between features. This analysis reveals that ResMPS naturally implements the logic of Factorization and Exponential Machines, providing a clear mathematical understanding of how the model captures high-order correlations.
Conclusion
The Residual Matrix Product State represents a significant step forward in hybridizing quantum-inspired tensor networks with deep learning principles. By introducing residual connections and non-linearities, ResMPS achieves superior accuracy, improved stability, and enhanced interpretability compared to traditional TN models. Future work will explore extending this residual framework to more complex 2D tensor structures like PEPS and integrating convolutional layers to further close the performance gap with state-of-the-art neural networks.
Summary updated
Abstract
Tensor network, which originates from quantum physics, is emerging as an efficient tool for classical and quantum machine learning.Nevertheless, there still exists a considerable accuracy gap between tensor network and the sophisticated neural network models for classical machine learning.In this work, we combine the ideas of matrix product state (MPS), the simplest tensor network structure, and residual neural network and propose the residual matrix product state (ResMPS).The ResMPS can be treated as a network where its layers map the "hidden" features to the outputs (e.g., classifications), and the variational parameters of the layers are the functions of the features of the samples (e.g., pixels of images).This is different from neural network, where the layers map feed-forwardly the features to the output.The ResMPS can equip with the nonlinear activations and dropout layers, and outperforms the state-of-the-art tensor network models in terms of efficiency, stability, and representation power.Besides, ResMPS is interpretable from the perspective of polynomial expansion, where the factorization and exponential machines naturally emerge.Our work contributes to connecting and hybridizing neural and tensor networks, which is crucial to further enhance our understanding of the working mechanisms and improve the performance of both models.Contents 1 Introduction 2 2 Residual matrix product state 3 2.1 Definition of residual matrix product state 3 2.2 The working mechanism of ResMPS 4 2.3 The Architecture of ResMPS 5 2.4 Benchmarking results 6 2.4.1 Classification accuracy 6 2.4.2Redundancy of regular MPS 7 2.4.3Representation power of ResMPS 8 2.5 Comparison with the recurrent neural network and transformer 9 3 Properties of the residual structure 9 3.1 Avoiding the gradient problems by residual terms 9 3.2 Relations to polynomial expansion 10 1 SciPost Physics Submission 4 Conclusion 11 A Training details 12 B Path independency 13 C Additional benchmarks 14 References 15 adopting the convolutional neural network (CNN) as a feature extractor in TN [7, 17, 18]; compressing the linear layers of deep NN by matrix product operators [19]; and implementing the convolutional operations using TN [20], etc.These attempts further motivate us to investigate the possible hybridization of TN and NN.In this work, we incorporate the information highways (also known as shortcuts) [21,22], non-linear activations, and dropout [23] into TN (MPS in specific), and propose Residual MPS (ResMPS in short).The essential underlying idea of ResMPS is a delicate way of inputting data such that the variational parameters of the network layers are the functions of the data features.Such idea is inspired by the traditional feed-forward neural network (FNN), while in FNN, the data is input only in the initial step.We provide two specific examples of ResMPS dubbed as simple and activated ResMPS.The simple version (sResMPS in short) is a multi-linear model that can exactly be written into a standard MPS, and the activated version (aResMPS in short) is a non-linear model equipped with NN layers.The results on Fashion-MNIST show that the simple ResMPS achieves the same accuracy as MPS while its parameter complexity is half of the MPS.For the activated ResMPS, we find a significant enhancement of efficiency and accuracy by introducing the non-linear activations and the dropout layers on the residual terms.Furthermore, we determine the model interpretability of sResMPS by polynomial expression.The truncated model achieves a high level of accuracy with only a few low-order terms of sResMPS.Surprisingly, the factorization [24] and exponential machines [25] have naturally emerged in this expansion scheme.ResMPS shows the underlying connections between TN and NN for ML and can shed light on novel possibilities and flexibility for developing powerful ML models beyond NN or TN.
BibTeX
@inproceedings{meng2022report,
author = {Ye-Ming Meng and Jing Zhang and Peng Zhang and Chao Gao and Shi-Ju Ran},
title = {{Report on scipost\_202209\_00017v1}},
booktitle = {Openalex Unknown},
year = {2022},
doi = {10.21468/scipost.report.5846},
}