Abstract
Sequential Recommendation (SR) is critical for personalized user experiences, yet standard Transformer-based models often overlook the nuanced temporal context of interactions. We propose MOJITO (Mixture Of Joint Item and Temporal cOntext), a novel architecture that utilizes Gaussian mixtures of attention to integrate temporal metadata—such as time of day or seasonality—directly into the sequential modeling process. MOJITO addresses the issue of redundant attention heads by sampling attention matrices from a probabilistic model, allowing for dynamic balancing of item sequences and temporal context. Our evaluation across three large-scale datasets shows that MOJITO achieves state-of-the-art performance, particularly in capturing periodic and context-dependent user preferences.
Introduction
Modern recommender systems rely heavily on Sequential Recommendation to predict a user's next interaction based on their history. While architectures like SASRec and BERT4Rec have set benchmarks, they typically treat time as a simple ordinal sequence. In reality, user behavior is deeply tied to context: music choices vary by time of day, and purchasing habits often follow seasonal cycles.
Existing methods that attempt to incorporate time often rely on simple feature concatenation, which fails to capture the complex, non-linear interactions between "what" was done and "when" it occurred. Furthermore, standard multi-head attention often suffers from "redundant heads," where different parts of the model learn identical patterns, wasting computational capacity. MOJITO addresses these challenges by jointly learning from user-item interactions and time-related metadata through a flexible mixture-of-attention framework.
Methodology: The MOJITO Architecture
MOJITO decomposes the recommendation task into two primary representations: short-term and long-term.
Short-Term Representation (Attention Mixtures)
The core innovation of MOJITO is the use of Attention Mixtures instead of standard multi-head attention. Attention matrices are sampled from a Gaussian mixture probabilistic model, enabling the model to automatically learn the relative importance of item sequences versus temporal context for any given prediction. Temporal features (e.g., "Monday" or "10 PM") are mapped into continuous embedding vectors using a translation-invariant time kernel. This approach ensures that the model can dynamically adjust its focus between the "what" (item history) and the "when" (temporal context), preventing it from being locked into a single signal.
Long-Term Representation (Attentive FISM)
To capture stable user preferences, MOJITO employs an Attentive FISM (Factored Item Similarity Model). Rather than a static user vector, the user is represented as a weighted sum of past interactions, allowing the representation to shift based on the target item. This is particularly effective for users with diverse interests, such as a listener who enjoys both Heavy Metal and Classical music.
Experiments and Results
The model was evaluated on three real-world datasets: MovieLens (1M ratings), Amazon Book (>1M interactions), and LFM-1b (music listening history). MOJITO was compared against eight high-performing baselines, including non-time-aware models (SASRec, BERT4Rec) and time-aware ones (TiSASRec, MEANTIME).
MOJITO achieved the best results on 5 out of 6 primary metrics across the three datasets. On the Amazon Book dataset, it reached a top 83.26% Hit Rate (HR@10) and 59.94% NDCG@10, comfortably beating the next best time-aware model. On the LFM-1b music dataset, it achieved 60.14% NDCG, demonstrating its strength in modeling periodic music listening patterns. While it was a close second in NDCG on MovieLens, it maintained the highest Hit Rate (82.09%).
Conclusion
MOJITO demonstrates that jointly modeling item sequences and temporal context through attention mixtures significantly improves recommendation accuracy. By increasing head diversity and providing a flexible framework for context integration, MOJITO overcomes the limitations of standard Transformers in time-sensitive domains. Future work will explore extending this "Mixture of Context" framework to other metadata types, such as location and device information, to further enhance personalization.
Summary updated