Abstract
We study user history modeling via Transformer encoders in deep learning recommendation models (DLRM). Such architectures can significantly improve recommendation quality, but usually incur high latency cost necessitating infrastructure upgrades or very small Transformer models. An important part of user history modeling is early fusion of the candidate item and various methods have been studied. We revisit early fusion and compare concatenation of the candidate to each history item against appending it to the end of the list as a separate item. Using the latter method allows us to reformulate the recently proposed amortized history inference algorithm M-FALCON for the case of DLRM models. We show via experimental results that appending with cross-attention performs on par with concatenation and that amortization significantly reduces inference costs. We conclude with results from deploying this model on the LinkedIn Feed and Ads surfaces, where amortization reduces latency by 30% compared to non-amortized inference.
Introduction
In modern Deep Learning Recommendation Models (DLRMs), understanding a user's past actions—such as clicks, views, and purchases—is vital for providing relevant content. While simple methods like mean-pooling history embeddings are computationally efficient, they fail to capture the rich, sequential context that Transformers can provide.
However, Transformers are computationally expensive. In a typical ranking scenario, a system might need to score hundreds of candidate items for a single user. Standard architectures often re-process the entire user history for every single candidate, leading to massive latency spikes—sometimes up to 24x higher than baseline models. This computational burden often forces engineers to compromise by using smaller, less effective models or investing in expensive GPU infrastructure. This research seeks to achieve Transformer-level performance with the efficiency required for real-time, CPU-based production environments.
Methodology: Early Fusion and Amortization
The core of the research focuses on the "early fusion" stage, where the candidate item is first introduced to the user's history sequence. The authors compared two primary methods:
- Concatenation (The Status Quo): The candidate item's features are merged with every single item in the user's history. This forces the Transformer to process the candidate at every step of the sequence for every candidate item.
- Appending (The Proposed Path): The candidate item is added to the end of the history list as a single new token.
By combining the Appending method with Cross-Attention, the computation can be "amortized." In this setup, history items only attend to themselves, while the candidate item attends to the entire history.
In Amortized Inference, the system processes the user's history sequence once to create a set of "keys" and "values." Then, for all candidate items (e.g., 500+), it only needs to perform a single cross-attention step. This avoids redundant calculations and allows the Transformer to process one large batch efficiently rather than hundreds of individual samples.
Experimental Evaluation
The approach was validated across public benchmarks and massive internal production datasets at LinkedIn:
- Public Benchmarks: MovieLens 20M, Amazon Books, Goodbooks, and Netflix, all focused on rating prediction.
- LinkedIn Feed: A ranking model predicting user contributions such as likes, comments, and shares.
- LinkedIn Ads: A ranking model predicting click-through rates (CTR).
Key Insights and Results
The study yielded several critical findings regarding performance and efficiency:
- Performance Parity: Contrary to previous assumptions that concatenation was superior for accuracy, this research demonstrates that appending with cross-attention performs just as well across various datasets. On public datasets like MovieLens and Amazon Books, the "Append" method slightly outperformed "Concat" in Mean Absolute Error (MAE).
- Efficiency Gains: On the LinkedIn Feed, the amortized approach reduced p90 latency by approximately 30% compared to standard inference. CPU usage for the user history component dropped from a +43% overhead to just +5.5%.
- Live Production Impact: In live LinkedIn Ads ranking, the model achieved a 10% reduction in p99 latency and CPU usage while maintaining high AUC improvements.
- The Latency-Engagement Loop: Interestingly, the amortized model showed higher user engagement in live A/B tests. This is attributed to the "latency effect," where faster response times in web applications generally lead to better user experiences and higher engagement metrics.
Conclusion and Future Work
Amortized inference provides a scalable path for integrating powerful Transformer models into latency-sensitive recommendation systems. While the current study focused on short-term context (20–50 items), future work will explore better ways to incorporate "lifelong" history without sacrificing offline metrics. Additionally, this amortization strategy is compatible with other efficiency techniques like Multi-Query Attention (MQA) or model quantization, offering further potential for optimization.
Summary updated