Probabilistic Masked Attention Networks for Explainable Sequential Recommendation

Huiyuan Chen, Kaixiong Zhou, Zhimeng Jiang, Chin-Chia Michael Yeh, Xiaoting Li, Menghai Pan, Yan Zheng, Hao YangIJCAI 2023 · DOI

Introduction

Transformer-based models have become the gold standard for sequential recommendation, leveraging self-attention to capture complex dependencies in user behavior. However, these models often struggle with the inherent noise found in real-world interaction logs. Standard Transformers utilize the Softmax function to compute attention scores, which by design assigns a non-zero weight to every item in a user's history. This "dense" attention forces the model to incorporate irrelevant interactions—such as a random click on a T-shirt while researching electronics—into its final prediction.

Conceptual illustration of relevant vs. irrelevant attention in sequential recommendation.

This paper introduces the Probabilistic Masked Attention Network (PMAN), a novel architecture designed to address the limitations of dense attention. By introducing a learnable binary mask and replacing Softmax with a squared ReLU activation, PMAN creates sparse attention maps that effectively filter out noise, improving both recommendation accuracy and model explainability.

Methodology: The PMAN Architecture

The core innovation of PMAN lies in its ability to prune irrelevant items from the attention context. The authors propose several key modifications to the standard self-attention mechanism:

1. Sparse Activation with Squared ReLU

Unlike Softmax, which always outputs positive values, PMAN employs a squared ReLU activation. This allows the network to output exactly zero for certain attention weights, enabling the model to ignore items that do not contribute to the current prediction.

2. Probabilistic Masking

PMAN introduces a trainable binary mask $M \in \{0,1\}$ applied element-wise to the attention map. This mask acts as a gate: if a mask value is 0, the corresponding historical item is completely excluded from the context. To overcome the challenge of training discrete variables, the authors utilize the Gumbel-Softmax trick to approximate gradients during backpropagation.

3. Constrained Optimization via PGD

To control the level of sparsity, the authors apply Projected Gradient Descent (PGD) to enforce an $L_1$ norm constraint on the mask. This acts as a "budget," allowing the model to learn exactly how many items should be retained for optimal performance.

4. Architectural Simplification

Interestingly, the authors found that the effectiveness of sparse attention allowed them to simplify the overall Transformer architecture. PMAN operates with a single attention head and removes standard components like point-wise feed-forward layers and layer normalization, leading to a more efficient model without sacrificing performance.

Experimental Evaluation

The authors evaluated PMAN across five diverse benchmark datasets: Amazon-Beauty, Amazon-Sports, Yelp, MovieLens1M, and Steam. Using a leave-one-out evaluation strategy, they compared PMAN against state-of-the-art backbones including SASRec, TiSASRec, and BERT4Rec.

Key Results

  • Performance Gains: Integrating PMAN into the SASRec backbone resulted in an average improvement of 7.97% in Hit@10 and 9.06% in NDCG@10 across all datasets.
  • Efficiency: Despite the added complexity of mask optimization, PMAN's simplified architecture allows it to train in approximately 0.81x the time required for standard Transformer backbones.
  • Robustness to Noise: In stress tests where random noise was injected into user sequences, PMAN maintained significantly higher performance than standard models, demonstrating its superior denoising capabilities.

Explainability and Visualization

One of the primary advantages of PMAN is its inherent explainability. By forcing attention weights for irrelevant items to zero, the model provides a clear, sparse signal of which past interactions influenced a recommendation.

Comparison of attention maps between standard SASRec and SASRec+PMAN, highlighting the sparse masking of irrelevant items.

Qualitative analysis on the MovieLens dataset shows that while standard SASRec spreads attention across the entire history, PMAN successfully masks unrelated items (marked with an 'X' in the visualization), focusing only on the items most relevant to the user's current intent.

Conclusion and Future Work

PMAN demonstrates that sparsity is not just a tool for efficiency, but a powerful mechanism for improving the accuracy and interpretability of sequential recommenders. By explicitly masking noise, PMAN outperforms dense attention models while being faster to train. Future research will explore scaling these sparse masking benefits to massive sequence lengths by integrating linear-time attention approximations.

Summary updated

BibTeX
@inproceedings{chen2023probabilistic,
  author = {Huiyuan Chen and Kaixiong Zhou and Zhimeng Jiang and Chin-Chia Michael Yeh and Xiaoting Li and Menghai Pan and Yan Zheng and Xia Hu and Hao Yang},
  title = {{Probabilistic Masked Attention Networks for Explainable Sequential Recommendation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/230},
}
Figure