Denoising Attention for Query-aware User Modeling

Elias Bassani, Pranav Kasela, Gabriella PasiNAACL 2024 · DOI

TL;DR

This paper introduces "Denoising Attention," a novel mechanism for query-aware user modeling in personalized search. It addresses the limitations of standard attention by filtering out irrelevant user history, leading to more robust and effective personalization, as demonstrated by significant improvements in search result ranking metrics.

What Problem Does This Paper Address?

Personalized search aims to tailor search results to individual users' needs and interests. A key challenge is building accurate user models that reflect a user's current information need, especially since users have diverse interests that may not always be relevant to their current query. Previous approaches often used attention mechanisms to weigh user history, but these can be susceptible to "noisy" or irrelevant information, potentially degrading search effectiveness. This paper tackles the problem of how to effectively discern and utilize relevant user history for personalization, preventing irrelevant information from negatively impacting search results.

Methods

The core contribution is the Denoising Attention mechanism. It builds upon the standard attention framework but introduces two key innovations:

  • Filtering Mechanism: Before aggregation, a filtering step is applied. This involves:
  • Thresholding: A threshold t is used to identify and down-weight (or effectively zero out) user-related information that has low alignment with the current query. This is achieved by shifting alignment scores by σ(t) and then applying a ReLU activation.
  • ReLU Activation: The ReLU function is crucial for zeroing out negative alignment scores, effectively discarding noisy or irrelevant user data.
  • Robust Normalization: Instead of the standard Softmax function, Denoising Attention uses a sum-based normalization that is less prone to producing overly narrow or overly smooth attention weights. This helps maintain a balanced representation of user interests.

The paper also proposes a cosine similarity-based alignment model, which is bounded between 0 and 1, to work effectively with the filtering mechanism.

Datasets

The authors evaluated their approach on two datasets:

  • Web Search Dataset: Based on the AOL query log, this dataset captures a broad range of user interests.
  • Academic Search Dataset: A synthetic dataset built from the ArnetMiner Citation Network, simulating an academic search scenario with more focused user interests.

Key Insights and Ideas

  • Attention isn't always better: While attention mechanisms are powerful, their standard formulation can be detrimental to personalization if not carefully managed, especially when dealing with diverse user interests.
  • Noise is a real problem: User history is not always relevant to the current query. Simply weighing all historical data can lead to misinterpretations of user intent and reduced search effectiveness.
  • Filtering is key: A mechanism to actively filter out irrelevant user history is crucial for robust personalization. Denoising Attention's filtering step, combined with ReLU, proves effective in this regard.
  • Softmax limitations: The Softmax function's properties (e.g., always producing positive weights) can lead to personalization even when no user information is relevant, potentially harming search results. Denoising Attention's alternative normalization avoids this.
  • Robustness matters: The Denoising Attention model demonstrates significantly better robustness, meaning it's less likely to decrease search effectiveness compared to other methods when faced with noisy user data.

Key Results

  • Significant Improvements: Denoising Attention achieved improvements of about 15% over the best-performing baselines across key metrics (MAP, MRR, NDCG) on both datasets.
  • Robustness: The Denoising Attention-based user model was substantially less harmful than other models, decreasing BM25 effectiveness for significantly fewer queries (19% vs. 25-33% for other models on the Web Search Dataset).
  • Comparison to Baselines:
    • Standard Attention models showed mixed results, with some even performing worse than a simple "Mean" user model.
  • Zero Attention models were inconsistent.
  • Multi-Head Attention did not provide significant improvements over simpler models despite its complexity.
  • Threshold Impact: The effectiveness of Denoising Attention is sensitive to the threshold parameter σ(t), highlighting the importance of tuning this parameter for different search contexts.

Limitations and Future Work

  • Parameterized Alignment Model: The paper used a cosine similarity-based alignment model. Future work could explore parameterized functions that leverage additional information, such as document timestamps, to better assess relevance.
  • Fixed Threshold: The fixed threshold parameter t might be sub-optimal for different queries. Future research could explore more dynamic or query-dependent thresholding mechanisms.
  • Management of User-Related Information: The authors acknowledge that managing user-related information for personalization is a complex and ongoing challenge, leaving room for further improvements.

Summary updated

BibTeX
@inproceedings{bassani2024denoising,
  author = {Elias Bassani and Pranav Kasela and Gabriella Pasi},
  title = {{Denoising Attention for Query-aware User Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.153},
}
Figure