Abstract
This paper introduces LURM (Full-Life cycle User Representation Model), a framework designed to generate general-purpose user embeddings from a user's entire history—potentially encompassing millions of behaviors—since registration. By integrating a "Bag-of-Interests" (BoI) encoding with a self-supervised multi-anchor encoder network, LURM captures diverse and evolving user preferences more effectively than existing models limited to short-term behavior sequences. Evaluations on massive datasets demonstrate that LURM significantly outperforms state-of-the-art methods in user profiling and preference prediction.
Introduction
In modern recommendation systems, user modeling typically follows two paradigms: task-specific models and general-purpose models. While general-purpose models are more scalable and economical, they face a significant technical hurdle: the long-sequence problem. Standard architectures like RNNs or Transformers struggle to process more than a few hundred behaviors due to memory constraints and the risk of "catastrophic forgetting."
However, a user's true identity and long-term interests are often buried in their full-life cycle data. The authors propose LURM to bridge this gap, enabling the processing of years of behavioral data without losing the fine-grained details of user activity.
Methodology
The LURM framework consists of two cascaded stages designed for efficiency and representational power.
Bag-of-Interests (BoI)
Inspired by the "Bag-of-Words" concept in NLP, this module clusters billions of items into a fixed vocabulary of approximately $10^5$ "interest clusters." A user's behavior within a specific time period is represented as a sparse, high-dimensional vector based on these clusters. This approach allows the model to handle an arbitrary number of behaviors efficiently by abstracting individual items into broader interest categories.
Self-supervised Multi-anchor Encoder Network (SMEN)
The SMEN maps BoI sequences into low-dimensional embeddings using three key components:
- Multi-anchor Module: Instead of compressing a user into a single vector, this module uses multiple "anchors" to capture different facets of a user's personality (e.g., sports, parenting, technology).
- Time Aggregation: Gated Recurrent Units (GRUs) are employed to model the temporal evolution of these interests.
- Multi-scale Aggregation: The system processes data at different granularities (e.g., monthly vs. yearly) to balance short-term trends with long-term stability.
The entire system is trained via contrastive learning, where the model learns to associate different time-slices of behavior belonging to the same user, ensuring the embeddings are robust and representative.
Experimental Evaluation
The authors evaluated LURM on two massive datasets:
- Amazon Dataset: Product reviews and metadata from 1997 to 2017, covering 43 million users.
- Industrial Dataset: Real-world visit and order logs from Alibaba for over 214 million users, spanning 2016 to 2021.
Key Results
- Superiority over SOTA: LURM consistently outperformed state-of-the-art self-supervised models like PTUM and SUMN, achieving an average AUC improvement of ~3.23% on the Amazon dataset.
- Performance vs. Supervised Models: LURM embeddings often outperformed task-specific supervised models (e.g., TextCNN, HAN), likely because LURM could leverage much longer behavior sequences.
- Impact of History Length: User profiling (e.g., age/gender prediction) benefited significantly from longer histories. Moving from 2 months to 5 years of data improved age prediction AUC by nearly 10% on the industrial dataset.
Conclusion and Future Work
LURM demonstrates that "more is better" when it comes to user behavioral data, provided the architecture can handle the scale. The BoI and Multi-anchor approaches provide a scalable way to compress massive histories into useful embeddings. Future work will focus on incorporating multi-modal data (images, video) and testing the framework on a wider variety of downstream recommendation tasks.
Summary updated