Abstract
Sequential recommendation (SR) models typically rely on Transformer-based architectures to capture user interests from chronological interaction histories. However, standard self-attention mechanisms often act as low-pass filters, smoothing out high-frequency signals and failing to capture periodic user behaviors. We present FEARec (Frequency Enhanced Hybrid Attention Network), a novel framework that addresses these limitations by shifting behavior analysis into the frequency domain. By integrating a hierarchical frequency ramp structure and an auto-correlation-based attention mechanism, FEARec effectively captures both short-term local interactions and long-term periodic patterns, significantly outperforming state-of-the-art baselines across multiple benchmark datasets.
Introduction
Sequential recommendation aims to predict the next item a user will interact with based on their historical sequence. While models like SASRec have set the standard, they suffer from two primary "blind spots." First, the low-pass filter problem causes standard self-attention to favor global trends while discarding high-frequency signals representing rapid changes in user interest. Second, obscured periodicity—the tendency for users to repeat behaviors at specific intervals—is difficult to isolate in the time domain where signals are often noisy and intertwined.
The FEARec Framework
FEARec overcomes these challenges by processing user sequences in the frequency domain using Fast Fourier Transforms (FFT). The architecture is built around a hybrid attention encoder that balances time-domain correlations with frequency-domain periodicities.
Frequency Domain Attention (FDA)
To identify periodic patterns, FEARec utilizes an auto-correlation mechanism grounded in the Wiener-Khinchin theorem. By comparing a sequence with its time-delayed versions in the frequency domain, the model efficiently identifies relevant "lags" or cycles in user behavior without the computational complexity of exhaustive time-domain comparisons.
Hierarchical Frequency Ramp Structure
A key innovation in FEARec is the Frequency Ramp Structure. Rather than processing the entire signal uniformly across all layers, the model assigns specific frequency bands to different layers. Bottom layers are optimized to capture high-frequency details (local, rapid interactions), while top layers focus on low-frequency signals (global evolution of interests). This hierarchical approach prevents the "oversmoothing" common in deep Transformers.
Multi-Task Learning and Regularization
To ensure robust representation learning, FEARec incorporates a multi-task learning objective. This includes a standard recommendation loss alongside contrastive learning and a unique Frequency Domain Regularization. The latter uses an L1 loss to align the frequency spectrums of different augmented views of the same user profile, ensuring that the model's frequency-based insights remain consistent.
Experimental Evaluation
The authors evaluated FEARec on four benchmark datasets: Amazon Beauty, Clothing, and Sports (sparse, short sequences) and MovieLens-1M (dense, long sequences).
Key Results
- Performance Gains: FEARec consistently outperformed state-of-the-art baselines like SASRec and DuoRec. Notably, it achieved a 10.88% improvement in HR@5 on the Clothing dataset and a 10.77% improvement in NDCG@10 on MovieLens-1M.
- Sparsity Resilience: The model demonstrated particular strength on sparse datasets, suggesting that frequency-domain patterns provide more reliable signals when interaction data is limited.
- Ablation Studies: Removing either the frequency ramp or the auto-correlation module led to significant performance degradation, confirming that both local detail and periodic habit modeling are essential for accuracy.
Conclusion and Future Work
FEARec demonstrates that shifting sequential recommendation into the frequency domain provides a powerful solution to the limitations of standard self-attention. While the model shows high sensitivity to the hybrid ratio between time and frequency domains, its ability to capture periodic and high-frequency signals marks a significant advancement. Future research may explore more sophisticated frequency filters and the application of these techniques to cold-start recommendation scenarios.
Summary updated