Abstract
Sequential recommendation systems aim to predict a user's next interaction based on their historical behavior. While Recurrent Neural Networks (RNNs) like LSTMs have been foundational in this field, they suffer from inherent limitations: the "monotonicity trap," where recent items are over-prioritized, and "aggressive forgetting," where long-term interests are prematurely discarded. This paper introduces LSTeM (Long Short-Term enhanced Memory), which re-engineers the internal circuitry of the LSTM cell. By replacing standard additive gates with a Query-Key-Value (Q-K-V) mechanism and introducing a novel "recover gate," LSTeM effectively captures complex, non-monotonic dependencies and retrieves vital historical information from a global memory embedding.
Introduction
For years, LSTMs were the gold standard for sequential recommendation. However, their architecture imposes a rigid assumption of temporal decay. In real-world scenarios, a user's current intent might be more closely related to an action taken weeks ago than to their most recent click. Standard LSTMs struggle to skip over the "noise" of recent, irrelevant actions. Furthermore, the standard "forget gate" is often too aggressive, deleting long-term preferences to prevent memory overload.
While many contemporary approaches attempt to solve these issues by adding external attention layers (extrinsic improvements), LSTeM focuses on intrinsic improvements to the LSTM cell itself. By modernizing the internal gating logic, LSTeM proves that recurrent architectures remain highly competitive with pure attention-based models like SASRec.
The LSTeM Architecture
The core innovation of LSTeM lies in its modified internal cell, which replaces the traditional linear combinations used in gating with more dynamic operations.
Q-K-V Gating Mechanism
In a standard LSTM, the Input, Forget, and Output gates are calculated by adding the current input to the previous hidden state. LSTeM replaces this with a Query-Key-Value triplet mechanism, similar to those found in Transformers. At each time step, the gate calculates the correlation between the current item and the sequence history. This allows the gate to function as a dynamic filter that understands context, deciding what information to retain or discard based on how well the past and present "match."
The Recover Gate
To combat the issue of aggressive forgetting, LSTeM introduces a fourth gate: the Recover Gate. This gate works in tandem with a dynamic global memory embedding—a summary of the user's entire history generated via self-attention. If the forget gate accidentally discards a piece of information that becomes relevant again, the recover gate can "pull" that information back from the global memory into the current hidden state.
Experimental Evaluation
The model was evaluated on four diverse public datasets representing different domains and levels of data sparsity:
- Amazon Beauty: High sparsity with short sequences.
- Steam: Large-scale video game interaction data.
- MovieLens (ML-1m & ML-20m): Standard benchmarks for movie recommendations, with ML-20m providing a massive dataset to test scalability.
Key Results
LSTeM was compared against nine state-of-the-art baselines, including traditional methods (BPR-MF), standard RNNs (GRU4Rec), and pure attention models (SASRec).
- Superior Accuracy: LSTeM achieved the best performance across almost all metrics (MRR, Hit Rate, and NDCG) on all four datasets. The most significant improvements were observed on the ML-20m dataset, suggesting that the architecture excels at extracting value from large volumes of data.
- Ablation Studies: Removing the recover gate or reverting to standard additive gates caused a significant drop in performance, validating that both the Q-K-V mechanism and the recovery logic are essential to the model's success.
- Efficiency: Despite the increased mathematical complexity of the internal gates, the training time per epoch remained comparable to other deep learning-based sequential models.
Conclusion and Future Work
LSTeM demonstrates that the limitations of RNNs in sequential recommendation are not inherent to the recurrent structure itself, but rather to the traditional gating mechanisms. By integrating Q-K-V logic and a recovery path for forgotten information, LSTeM provides a robust framework for modeling complex user behavior.
Future research may address the computational overhead of the "gate math" in massive production environments and explore the integration of Graph Neural Network (GNN) properties to better model multi-relational user-item interactions within the sequence.
Summary updated