Long short-term enhanced memory for sequential recommendation

Jiasheng Duan, Peng-Fei Zhang, Ruihong Qiu, Zi HuangWWW 2023 · DOI · OpenReview

Abstract

Sequential recommendation systems aim to predict a user's next interaction based on their historical behavior. While Recurrent Neural Networks (RNNs) like LSTMs have been foundational in this field, they suffer from inherent limitations: the "monotonicity trap," where recent items are over-prioritized, and "aggressive forgetting," where long-term interests are prematurely discarded. This paper introduces LSTeM (Long Short-Term enhanced Memory), which re-engineers the internal circuitry of the LSTM cell. By replacing standard additive gates with a Query-Key-Value (Q-K-V) mechanism and introducing a novel "recover gate," LSTeM effectively captures complex, non-monotonic dependencies and retrieves vital historical information from a global memory embedding.

Introduction

For years, LSTMs were the gold standard for sequential recommendation. However, their architecture imposes a rigid assumption of temporal decay. In real-world scenarios, a user's current intent might be more closely related to an action taken weeks ago than to their most recent click. Standard LSTMs struggle to skip over the "noise" of recent, irrelevant actions. Furthermore, the standard "forget gate" is often too aggressive, deleting long-term preferences to prevent memory overload.

While many contemporary approaches attempt to solve these issues by adding external attention layers (extrinsic improvements), LSTeM focuses on intrinsic improvements to the LSTM cell itself. By modernizing the internal gating logic, LSTeM proves that recurrent architectures remain highly competitive with pure attention-based models like SASRec.

The LSTeM Architecture

The core innovation of LSTeM lies in its modified internal cell, which replaces the traditional linear combinations used in gating with more dynamic operations.

Q-K-V Gating Mechanism

In a standard LSTM, the Input, Forget, and Output gates are calculated by adding the current input to the previous hidden state. LSTeM replaces this with a Query-Key-Value triplet mechanism, similar to those found in Transformers. At each time step, the gate calculates the correlation between the current item and the sequence history. This allows the gate to function as a dynamic filter that understands context, deciding what information to retain or discard based on how well the past and present "match."

The Recover Gate

To combat the issue of aggressive forgetting, LSTeM introduces a fourth gate: the Recover Gate. This gate works in tandem with a dynamic global memory embedding—a summary of the user's entire history generated via self-attention. If the forget gate accidentally discards a piece of information that becomes relevant again, the recover gate can "pull" that information back from the global memory into the current hidden state.

The internal architecture of the LSTeM cell (labeled as M-LSTM), illustrating the Q-K-V gating logic and the interaction between the recover gate ($r_t$) and global memory ($m_t^{avg}$).

Experimental Evaluation

The model was evaluated on four diverse public datasets representing different domains and levels of data sparsity:

  • Amazon Beauty: High sparsity with short sequences.
  • Steam: Large-scale video game interaction data.
  • MovieLens (ML-1m & ML-20m): Standard benchmarks for movie recommendations, with ML-20m providing a massive dataset to test scalability.

Key Results

LSTeM was compared against nine state-of-the-art baselines, including traditional methods (BPR-MF), standard RNNs (GRU4Rec), and pure attention models (SASRec).

  • Superior Accuracy: LSTeM achieved the best performance across almost all metrics (MRR, Hit Rate, and NDCG) on all four datasets. The most significant improvements were observed on the ML-20m dataset, suggesting that the architecture excels at extracting value from large volumes of data.
  • Ablation Studies: Removing the recover gate or reverting to standard additive gates caused a significant drop in performance, validating that both the Q-K-V mechanism and the recovery logic are essential to the model's success.
  • Efficiency: Despite the increased mathematical complexity of the internal gates, the training time per epoch remained comparable to other deep learning-based sequential models.

Conclusion and Future Work

LSTeM demonstrates that the limitations of RNNs in sequential recommendation are not inherent to the recurrent structure itself, but rather to the traditional gating mechanisms. By integrating Q-K-V logic and a recovery path for forgotten information, LSTeM provides a robust framework for modeling complex user behavior.

Future research may address the computational overhead of the "gate math" in massive production environments and explore the integration of Graph Neural Network (GNN) properties to better model multi-relational user-item interactions within the sequence.

Summary updated

Abstract

Sequential recommendation is a stream of studies on recommender systems, which focuses on predicting the next item a user interacts with by modeling the dynamic sequence of user-item interactions. Since being born to explore the dynamic tendency of variable-length temporal sequence, Recurrent Neural Networks (RNNs) have been paid much attention in this area. However, the inherent defects caused by the network structure of RNNs have limited their applications in sequential recommendation, which are mainly shown on two factors: RNNs tend to make point-wise predictions and ignore the collective dependencies because the temporal relationships between items change monotonically; RNNs are likely to forget the essential information during processing long sequences. To solve these problems, researchers have done much work to enhance the memory mechanism of RNNs. However, although previous RNN-based methods have achieved promising performance by taking advantage of external knowledge with other advanced techniques, the improvement of the intrinsic property of existing RNNs has not been explored, which is still challenging. Therefore, in this work, we propose a novel architecture based on Long Short-Term Memories (LSTMs), a broadly-used variant of RNNs, specific for sequential recommendation, called Long Short-Term enhanced Memory (LSTeM), which boosts the memory mechanism of original LSTMs in two ways. Firstly, we design a new structure of gates in LSTMs by introducing a “Q-K-V” triplet, a mechanism to accurately and properly model the correlation between the current item and the user’s historical behaviors at each time step. Secondly, we propose a “recover gate” to remedy the inadequacy of memory caused by the forgetting mechanism, which works with a dynamic global memory embedding. Extensive experiments have demonstrated that LSTeM achieves comparable performance to the state-of-the-art methods on the challenging datasets for sequential recommendation.
BibTeX
@article{duan2023long,
  author = {Jiasheng Duan and Peng-Fei Zhang and Ruihong Qiu and Zi Huang},
  title = {{Long short-term enhanced memory for sequential recommendation}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-022-01056-9},
}

Loading related content...

Figure