Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation

Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, Ji-Rong Wen

Abstract

Self-attention networks (SANs) have been intensively applied for sequential recommenders, but they are limited due to: (1) the quadratic complexity and vulnerability to over-parameterization in self-attention; (2) inaccurate modeling of sequential relations between items due to the implicit position encoding. In this work, we propose the low-rank decomposed self-attention networks (LightSANs) to overcome these problems. Particularly, we introduce the low-rank decomposed self-attention, which projects user's historical items into a small constant number of latent interests and leverages item-to-interest interaction to generate the context-aware representation. It scales linearly w.r.t. the user's historical sequence length in terms of time and space, and is more resilient to over-parameterization. Besides, we design the decoupled position encoding, which models the sequential relations between items more precisely. Extensive experimental studies are carried out on three real-world datasets, where LightSANs outperform the existing SANs-based recommenders in terms of both effectiveness and efficiency.
BibTeX
@inproceedings{fan2021lighter,
  author = {Xinyan Fan and Zheng Liu and Jianxun Lian and Wayne Xin Zhao and Xing Xie and Ji-Rong Wen},
  title = {{Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  doi = {10.1145/3404835.3462978},
}

Loading related content...

Figure