Revenge of MLP in Sequential Recommendation

Yiheng Jiang, Yuanbo XuarXiv 2023 · OpenReview

TL;DR

This paper introduces TriMLP, a novel MLP-based architecture for sequential recommendation that addresses the information leakage issue inherent in standard MLPs. By employing a "Triangular Mixer," TriMLP effectively models temporal dependencies without sacrificing computational efficiency, achieving state-of-the-art performance across various datasets.

What Problem Does This Paper Address?

Sequential recommendation systems aim to predict a user's next action (e.g., item to purchase, video to watch) based on their historical interactions. While Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers have been successful, recent advancements in Multi-Layer Perceptrons (MLPs), particularly "token-mixing MLPs," offer a computationally efficient alternative. However, directly applying these MLPs to sequential recommendation tasks leads to subpar performance. The core problem is that the fully-connected nature of standard MLPs allows for "anti-chronological" interactions, where future information can leak into the prediction of current or past events. This information leakage disrupts the auto-regressive training paradigm crucial for sequential modeling, hindering the MLP's ability to capture temporal dynamics effectively.

Methods

The paper proposes TriMLP, an MLP-like architecture for sequential recommendation. The key innovation is the Triangular Mixer, which modifies the standard MLP's cross-token communication mechanism.

  • Triangular Mixer: This component simplifies the MLP's matrix multiplication by selectively dropping lower-triangle neurons in the weight matrix. This effectively blocks connections from future tokens to past or current tokens, preventing information leakage and enforcing chronological order.
  • Global and Local Mixing: The Triangular Mixer further comprises two sub-layers:
  • Global Mixing: This layer uses a triangular mask to capture long-range dependencies across the entire sequence.
  • Local Mixing: This layer further refines the triangular mask to capture short-term, local patterns within specific segments (sessions) of the sequence. These two layers are serially combined to model both long-term and short-term preferences.

The architecture follows a standard pattern: an embedding layer processes input sequences, followed by the Triangular Mixer for interaction modeling, and finally a classifier to predict item probabilities.

Datasets

The authors evaluated TriMLP on 12 publicly available datasets from four benchmarks:

  • Amazon: Beauty, Sports
  • MovieLens: ML-100K, ML-1M, ML-10M
  • Tenrec: QB-Article, QB-Video
  • Location-based Social Network (LBSN): NYC, TKY, Brightkite, Yelp, Gowalla

These datasets vary in scale, ranging from approximately 50K to 10M user-item interactions.

Key Insights and Ideas

  • MLPs and Auto-Regressive Training Don't Mix (Without Modification): The paper strongly argues that standard, fully-connected MLPs are fundamentally incompatible with the auto-regressive training paradigm used in sequential recommendation due to information leakage. This is a crucial insight that explains the observed performance gap.
  • Triangular Masking is Key: The simple yet effective "Triangular Mixer" is the core enabler for MLPs in this domain. By enforcing chronological order through masking, it unlocks the MLP's potential for sequential modeling.
  • Global and Local Mixing Complement Each Other: Combining both global (long-range) and local (short-range) mixing within the Triangular Mixer is more effective than using either alone, highlighting the need for multi-scale temporal modeling.
  • Efficiency Gains are Significant: The MLP architecture, even with the proposed mixer, maintains a strong efficiency advantage over Transformer-based models, especially in terms of inference cost.

Key Results

  • Performance Improvement: TriMLP consistently outperforms state-of-the-art baselines across all 12 datasets, achieving an average performance boost of up to 14.88%.
  • Efficiency: TriMLP offers a significant reduction in inference cost, with an average saving of 8.65% compared to baselines. This efficiency advantage grows with dataset scale.
  • Ablation Studies:
    • Removing the triangular structure (SqrMLP) severely degrades performance, confirming the importance of chronological modeling.
  • Using only global (TriMLP_G) or only local (TriMLP_L) mixing yields better results than a standard MLP but is inferior to the combined TriMLP.
  • Positional embeddings negatively impact TriMLP's performance, suggesting the triangular structure implicitly captures positional information.
  • 1-0 initialization and Softmax normalization in the mixing kernels are beneficial.

Limitations and Future Work

  • Limitations:
    • The paper notes that while TriMLP performs well, the "auto-encoding" training paradigm (used in models like BERT4Rec) is generally more suited for dense datasets, whereas sequential recommendation datasets are typically very sparse.
  • The authors mention that the performance of local mixing can be sensitive to the choice of session number/length, and certain settings might lead to suboptimal results.
  • Future Work:
    • The authors plan to improve TriMLP by incorporating auxiliary information such as temporal factors and item attributes.
  • They also aim to explore data augmentation and pre-training techniques.
  • A key future direction is to investigate how to decouple the sequence length from the MLP's shape, enabling it to flexibly handle sequences of varying lengths.

Summary updated

Abstract

In this paper, we present a MLP-like architecture for sequential recommendation, namely TriMLP, with a novel Triangular Mixer for cross-token communications. In designing Triangular Mixer, we simplify the cross-token operation in MLP as the basic matrix multiplication, and drop the lower-triangle neurons of the weight matrix to block the anti-chronological order connections from future tokens. Accordingly, the information leakage issue can be remedied and the prediction capability of MLP can be fully excavated under the standard auto-regressive mode. Take a step further, the mixer serially alternates two delicate MLPs with triangular shape, tagged as global and local mixing, to separately capture the long range dependencies and local patterns on fine-grained level, i.e., long and short-term preferences. Empirical study on 12 datasets of different scales (50K\textasciitilde 10M user-item interactions) from 4 benchmarks (Amazon, MovieLens, Tenrec and LBSN) show that TriMLP consistently attains promising accuracy/efficiency trade-off, where the average performance boost against several state-of-the-art baselines achieves up to 14.88% with 8.65% less inference cost.
BibTeX
@misc{jiang2023revenge,
  author = {Yiheng Jiang and Yuanbo Xu},
  title = {{Revenge of MLP in Sequential Recommendation}},
  year = {2023},
  eprint = {2305.14675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.14675},
}
Figure