Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation

Chang Liu, Xiaoguang Li, Guohao Cai, Zhenhua Dong, Hong Zhu, Lifeng Shang

TL;DR

Researchers from the University of Hong Kong and Huawei Noah’s Ark Lab have developed NOVA (Non-invasive self-Attention), a novel mechanism for integrating side information (like item categories or price) into sequential recommendation models. By using side information only to guide the attention distribution rather than altering the core item embeddings, NOVA avoids "information overwhelming" and consistently outperforms state-of-the-art BERT-based models with negligible computational overhead.

What Problem Does This Paper Address?

Sequential recommendation is the task of predicting a user's next interaction based on their history. While Transformer-based models like BERT4Rec have set new benchmarks by capturing complex dependencies in behavior sequences, they struggle to effectively incorporate "side information"—additional data like item tags, categories, or timestamps.

The standard approach is invasive fusion: directly adding or concatenating side information embeddings to the item ID embedding. However, the authors found that this often yields diminishing or even negative returns. They hypothesize that mixing these signals "pollutes" the embedding space, making it harder for the model to decode the original item ID and leading to a phenomenon they call information overwhelming.

Methods

The authors propose NOVA-BERT, which utilizes a Non-invasive self-Attention (NOVA) mechanism. The key innovation lies in how the components of the self-attention mechanism—Query ($Q$), Key ($K$), and Value ($V$)—are handled:

  1. Pure vs. Integrated Embeddings: The model maintains two sets of representations. One is the "pure" item ID embedding, and the other is an "integrated" embedding that fuses the ID with side information using a fusion function (like addition, concatenation, or a learnable gating mechanism).
  2. The NOVA Operation: Unlike standard self-attention where $Q, K,$ and $V$ all come from the same fused source, NOVA calculates $Q$ and $K$ from the integrated embeddings but derives $V$ solely from the pure item ID embeddings.
  3. Directional Guidance: In this setup, the side information acts as an auxiliary guide to help the model learn a better attention distribution (the "where to look" part of the equation), while the actual information passed through the network (the "what is being looked at" part) remains undoped.
Figure 3: Comparison between invasive and non-invasive ways of self-attention for feature fusion.

Datasets

The model was evaluated on three datasets:

  • MovieLens-1m & MovieLens-20m: Publicly available movie recommendation datasets. Side information included release year, genres, and user ratings.
  • APP: A private commercial dataset from a real-world app store containing 1.7 million records. Side information included app category, developer, file size, and tags.

The authors notably used a "ranking-all" evaluation metric, ranking the entire vocabulary for every prediction to avoid the biases often introduced by sampling small sets of negative items.

Key Insights and Ideas

  • Maintaining Embedding Consistency: By keeping the Value ($V$) vector space pure, the model ensures that the decoding process (predicting the item ID) remains a homogeneous vector search, which is significantly more stable than searching in a "compounded" space.
  • Attention Locality: Visualization of the attention maps revealed that NOVA-BERT learns a much more structured, "local" attention pattern (concentrated along the diagonal) compared to the scattered patterns of standard BERT4Rec. This suggests that side information helps the model focus on the most relevant historical behaviors.
  • Synergy of Information: The ablation studies showed that combining item-related (static) and behavior-related (dynamic) side information provides a boost greater than the sum of their individual parts, suggesting these features provide complementary signals when processed non-invasively.
Figure 1: An illustration of invasive and non-invasive meth- ods. The invasive methods fuse all kinds of information irre- versibly, and then feed them into sequential models. For the non-invasive method, the side information only participants in the attention matrix calculation, while item information is kept in an independent vector space.

Key Results

  • Superior Accuracy: NOVA-BERT outperformed the baseline BERT4Rec and all invasive fusion variants across all datasets. On the ML-1m dataset, NOVA with a gating fuser improved Hit Ratio (HR@10) by 13.51% over the baseline.
  • Robustness: The non-invasive approach was consistently effective regardless of the specific fusion function used (Add, Concat, or Gating), whereas invasive methods often performed worse than the baseline.
  • Efficiency: The computational cost is nearly identical to standard models. In tests, NOVA-BERT showed almost no extra FLOPs (Floating Point Operations) and maintained the same model size as invasive methods, making it highly suitable for production deployment.

Limitations and Future Work

The authors acknowledge that while NOVA is effective, there are still open questions:

  • Layer-wise Fusion: Currently, the same side information is provided to every NOVA layer. The authors suggest that exploring different fusion strategies at different depths of the network might yield further improvements.
  • Fusion Functions: While the gating mechanism performed well, the search for even "stronger" or more specialized fusion functions continues.
  • Dataset Sensitivity: The scale of improvement was smaller on larger, denser datasets (like ML-20m), suggesting that when the core behavior data is extremely rich, the marginal utility of side information—even when integrated non-invasively—may decrease.

Summary updated

Abstract

Sequential recommender systems aim to model users' evolving interests from their historical behaviors, and hence make customized time-relevant recommendations. Compared with traditional models, deep learning approaches such as CNN and RNN have achieved remarkable advancements in recommendation tasks. Recently, the BERT framework also emerges as a promising method, benefited from its self-attention mechanism in processing sequential data. However, one limitation of the original BERT framework is that it only considers one input source of the natural language tokens. It is still an open question to leverage various types of information under the BERT framework. Nonetheless, it is intuitively appealing to utilize other side information, such as item category or tag, for more comprehensive depictions and better recommendations. In our pilot experiments, we found naive approaches, which directly fuse types of side information into the item embeddings, usually bring very little or even negative effects. Therefore, in this paper, we propose the NOninVasive self-attention mechanism (NOVA) to leverage side information effectively under the BERT framework. NOVA makes use of side information to generate better attention distribution, rather than directly altering the item embedding, which may cause information overwhelming. We validate the NOVA-BERT model on both public and commercial datasets, and our method can stably outperform the state-of-the-art models with negligible computational overheads.
BibTeX
@inproceedings{liu2021non,
  author = {Chang Liu and Xiaoguang Li and Guohao Cai and Zhenhua Dong and Hong Zhu and Lifeng Shang},
  title = {{Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i5.16549},
}
Figure