De-Anchor: Mitigating Attention Polarization for Lifelong User Behavior Modeling in Click-Through Rate Prediction
Lists
Only you see this
Abstract
User lifelong behavior sequences are essential for click-through rate (CTR) prediction tasks in industrial recommender systems. Attention-based module, especially multi-head target attention (MHTA), has been proven to be effective in aggregating behavior features given a certain target item. However, we found a common phenomenon that attention weights in MHTA tend to over-concentrate on merely a small subset of a user's historical behaviors, producing a sparse one-hot distributed attention weights in training gradually, which we callAttention Polarization (AP). These polarized weights on certain behaviors (which we call "\textitattention anchor ") could make the model fail to capture a user's diversified interests, and harm the learning of behavior embedding as the gradients on these features are nearly zero. We introduce two indicators:anchor rate andattention entropy to measure the magnitude of AP, and proposeDe-Anchor, a novel method to alleviate it, which can serve as a stand-alone and parameter-efficient plug-in to existing CTR backbones. De-Anchor contains two modules:Anchor-aware gradient dropout (AGD) forcing the model to capture diversified interest information from behavior sequences by discarding gradients of non-behavior features, andTarget-aware attention anchor (TAA) providing a pseudo behavior to offload excessive weights of MHTA. Extensive offline experiments and industrial online A/B tests demonstrate the efficacy of our method.
BibTeX
@inproceedings{liu2024de,
author = {Hongzun Liu and Kang Yin and Tianyu Sun and Rui Huang and Yunsong Li and Xiao Fang and Zhaojie Liu and Weidong Liu and Guorui Zhou},
title = {{De-Anchor: Mitigating Attention Polarization for Lifelong User Behavior Modeling in Click-Through Rate Prediction}},
booktitle = {The Web Conference},
year = {2024},
doi = {10.1145/3589335.3651486},
}Loading related content...