TL;DR
This paper introduces TriMLP, a novel MLP-based architecture for sequential recommendation that addresses the information leakage issue inherent in standard MLPs. By employing a "Triangular Mixer," TriMLP effectively models temporal dependencies without sacrificing computational efficiency, achieving state-of-the-art performance across various datasets.
What Problem Does This Paper Address?
Sequential recommendation systems aim to predict a user's next action (e.g., item to purchase, video to watch) based on their historical interactions. While Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers have been successful, recent advancements in Multi-Layer Perceptrons (MLPs), particularly "token-mixing MLPs," offer a computationally efficient alternative. However, directly applying these MLPs to sequential recommendation tasks leads to subpar performance. The core problem is that the fully-connected nature of standard MLPs allows for "anti-chronological" interactions, where future information can leak into the prediction of current or past events. This information leakage disrupts the auto-regressive training paradigm crucial for sequential modeling, hindering the MLP's ability to capture temporal dynamics effectively.
Methods
The paper proposes TriMLP, an MLP-like architecture for sequential recommendation. The key innovation is the Triangular Mixer, which modifies the standard MLP's cross-token communication mechanism.
- Triangular Mixer: This component simplifies the MLP's matrix multiplication by selectively dropping lower-triangle neurons in the weight matrix. This effectively blocks connections from future tokens to past or current tokens, preventing information leakage and enforcing chronological order.
- Global and Local Mixing: The Triangular Mixer further comprises two sub-layers:
- Global Mixing: This layer uses a triangular mask to capture long-range dependencies across the entire sequence.
- Local Mixing: This layer further refines the triangular mask to capture short-term, local patterns within specific segments (sessions) of the sequence. These two layers are serially combined to model both long-term and short-term preferences.
The architecture follows a standard pattern: an embedding layer processes input sequences, followed by the Triangular Mixer for interaction modeling, and finally a classifier to predict item probabilities.
Datasets
The authors evaluated TriMLP on 12 publicly available datasets from four benchmarks:
- Amazon: Beauty, Sports
- MovieLens: ML-100K, ML-1M, ML-10M
- Tenrec: QB-Article, QB-Video
- Location-based Social Network (LBSN): NYC, TKY, Brightkite, Yelp, Gowalla
These datasets vary in scale, ranging from approximately 50K to 10M user-item interactions.
Key Insights and Ideas
- MLPs and Auto-Regressive Training Don't Mix (Without Modification): The paper strongly argues that standard, fully-connected MLPs are fundamentally incompatible with the auto-regressive training paradigm used in sequential recommendation due to information leakage. This is a crucial insight that explains the observed performance gap.
- Triangular Masking is Key: The simple yet effective "Triangular Mixer" is the core enabler for MLPs in this domain. By enforcing chronological order through masking, it unlocks the MLP's potential for sequential modeling.
- Global and Local Mixing Complement Each Other: Combining both global (long-range) and local (short-range) mixing within the Triangular Mixer is more effective than using either alone, highlighting the need for multi-scale temporal modeling.
- Efficiency Gains are Significant: The MLP architecture, even with the proposed mixer, maintains a strong efficiency advantage over Transformer-based models, especially in terms of inference cost.
Key Results
- Performance Improvement: TriMLP consistently outperforms state-of-the-art baselines across all 12 datasets, achieving an average performance boost of up to 14.88%.
- Efficiency: TriMLP offers a significant reduction in inference cost, with an average saving of 8.65% compared to baselines. This efficiency advantage grows with dataset scale.
- Ablation Studies:
- Removing the triangular structure (SqrMLP) severely degrades performance, confirming the importance of chronological modeling.
- Using only global (TriMLP_G) or only local (TriMLP_L) mixing yields better results than a standard MLP but is inferior to the combined TriMLP.
- Positional embeddings negatively impact TriMLP's performance, suggesting the triangular structure implicitly captures positional information.
- 1-0 initialization and Softmax normalization in the mixing kernels are beneficial.
Limitations and Future Work
- Limitations:
- The paper notes that while TriMLP performs well, the "auto-encoding" training paradigm (used in models like BERT4Rec) is generally more suited for dense datasets, whereas sequential recommendation datasets are typically very sparse.
- The authors mention that the performance of local mixing can be sensitive to the choice of session number/length, and certain settings might lead to suboptimal results.
- Future Work:
- The authors plan to improve TriMLP by incorporating auxiliary information such as temporal factors and item attributes.
- They also aim to explore data augmentation and pre-training techniques.
- A key future direction is to investigate how to decouple the sequence length from the MLP's shape, enabling it to flexibly handle sequences of varying lengths.
Summary updated