In the world of online shopping and content streaming, sequential recommendation is the engine that predicts what you’ll want next based on your recent history. To do this well, a model needs to understand two things: your broad semantic preferences (e.g., "I like LEGO Ninjago sets") and specific behavioral associations (e.g., "People who bought this specific LEGO set often buy this specific accessory next").
The challenge has always been combining these two types of evidence without creating a bloated, slow model. A new paper, RELAY, introduces an elegant solution: a compact recommender that uses a single query to perform both tasks simultaneously.
TL;DR
RELAY is a sequential recommendation model that reuses a single attention mechanism for two distinct roles: reading semantic preferences and routing behavioral memory. This unified approach achieves state-of-the-art accuracy across six datasets while being 65% smaller and nearly 1.8x faster than previous top-performing models.
What Problem Does This Paper Address?
Modern recommenders often struggle to balance accuracy with efficiency. On one hand, neural sequence models are great at capturing high-level user preferences. On the other hand, explicit behavioral evidence—like "item A often follows item B"—is incredibly powerful for pinpointing the exact next item.
Existing methods often treat these as separate components, leading to complex architectures that are expensive to run in production. The authors of RELAY identified a key redundancy: the decision of which past interactions are relevant is central to both semantic modeling and behavioral routing. They set out to eliminate this redundancy to create a "select once, use twice" architecture.
Methods
The core innovation of RELAY is its "One Query, Two Roles" architecture.
- One Shared Selection: A single learned query looks at the user's interaction history and assigns "attention weights" to past items. This determines which parts of the history are most important for the current prediction.
- Role I: Semantic Readout: These weights are used to aggregate historical interactions into a single "preference vector." This vector is then matched against the entire item catalog to find semantically similar candidates.
- Role II: Behavioral-Memory Routing: The exact same weights are used to "route" evidence from a transition memory. If a past item is highly weighted, its known frequent successors (behavioral neighbors) get a score boost.
- Candidate-Conditioned Gates: To ensure these two sources of evidence play well together, RELAY uses small, candidate-specific "gates" to calibrate how much weight to give the memory evidence for each potential recommendation.
Crucially, the entire model is trained using a single catalog-wide likelihood, meaning the ranking objective itself teaches the model exactly how to balance semantic "vibes" with hard behavioral data.
Datasets
The researchers evaluated RELAY on six diverse datasets from Amazon Reviews, spanning two different timeframes:
- Amazon 2014: Sports, Beauty, and Toys.
- Amazon 2023: Scientific, Instruments, and Games.
These datasets represent a wide range of user behaviors and item popularities, providing a robust testbed for the model's generalizability.
Key Insights and Ideas
- The Power of Redundancy Elimination: By forcing the model to use the same attention for two tasks, the authors didn't just save parameters—they created a tighter coupling between preference representation and the evidence used for ranking.
- Sparse Memory is Fast Memory: RELAY only applies behavioral "corrections" to a small subset of the catalog (items in the user's history and their immediate neighbors). This allows it to score the entire catalog in a single, efficient forward pass.
- "Cheating" Prevention: During training, the model could easily "cheat" by looking at the transition it's trying to predict. RELAY includes a clever "Record-Excluded" learning mechanism that removes the current record's contribution from the memory during training to ensure it learns to generalize.
Key Results
RELAY consistently outperformed 16 baseline methods, including heavyweights like SASRec and TIGER.
- Accuracy: It achieved the highest Recall@5 and NDCG@5 on every single dataset, improving over the previous best model (OneChord) by an average of 14.22% in Recall and 17.88% in NDCG.
- Efficiency: Despite the performance gains, RELAY is remarkably lean. It uses only 2.044 million parameters, a 65.12% reduction compared to the previous best predictor.
- Speed: In inference tests, it achieved a 1.75–1.79× speedup over the strongest baseline, making it highly suitable for real-time production environments.
Limitations and Future Work
While RELAY is highly effective, the authors note a few areas for growth:
- Single Query Bottleneck: While one query is efficient, extremely complex user histories might benefit from multiple queries. The authors briefly explored a multi-query extension in the appendix that showed further accuracy gains at a slight cost to efficiency.
- Static Memory: The behavioral transition memory is currently based on fixed counts. Future iterations could explore making this memory more dynamic or learnable.
- Semantic Identifiers: The model relies on a fixed semantic catalog (SIDs). Integrating more flexible, end-to-end learnable item representations could be a promising next step.
Summary updated