TL;DR
The paper introduces LURM (full-Life cycle User Representation Model), a self-supervised framework designed to learn general-purpose user embeddings from extremely long behavior histories spanning a user's entire lifecycle. By combining a novel "Bag-of-Interests" aggregation strategy with a multi-anchor encoder network, LURM successfully processes sequences containing millions of actions, achieving almost lossless dimensionality reduction and outperforming state-of-the-art methods on diverse downstream tasks.
What Problem Does This Paper Address?
In modern web platforms, understanding user preferences is vital for recommendation systems, search engines, and targeted advertising. Traditional approaches rely on task-specific user representations, training a separate model for every single application. This is computationally expensive and scales poorly.
While general-purpose (task-agnostic) user embeddings offer a more scalable alternative, existing models suffer from a major bottleneck: they can only handle short behavior sequences (typically tens or hundreds of actions). In reality, users accumulate thousands or even millions of behaviors over years of platform interaction. Standard sequential architectures like RNNs or Transformers fail at this scale due to memory constraints, high computational overhead, and catastrophic forgetting. This paper addresses the challenge of efficiently encoding a user's full-lifecycle behavior history into a compact, highly informative representation.
Methods
LURM solves the long-sequence bottleneck using a two-stage cascaded architecture:
- Bag-of-Interests (BoI): Instead of treating behaviors as a raw chronological sequence of billions of possible items (which is computationally intractable), BoI clusters items into a vocabulary of roughly $10^5$ distinct "interests." Item embeddings are first learned via a contrastive task, and then a large-scale clustering algorithm (HDSC) groups similar items. A user's behaviors within a specific timeframe are then summarized into a sparse, high-dimensional vector counting their interactions across these interest clusters.
- Self-supervised Multi-anchor Encoder Network (SMEN): This network compresses the high-dimensional BoI sequences into compact user embeddings through three key sub-modules:
- Multi-anchor Module: Unlike standard multi-head attention, this module uses fixed, learnable "anchors" representing different facets of user preferences. This disentangles user interests and enables almost lossless dimensionality reduction.
- Time Aggregation Module: Uses a Gated Recurrent Unit (GRU) to capture how user interests evolve over time.
- Multi-scale Aggregation Module: Processes behavior sequences at multiple granularities simultaneously (e.g., monthly and yearly views) to balance fine-grained short-term details with stable long-term trends.
The entire network is trained in a self-supervised manner using a contrastive loss function, where different sub-sequences from the same user serve as positive pairs.
Datasets
The authors evaluated LURM on two large-scale datasets:
- Amazon Dataset (Public): Contains user product reviews and metadata spanning from 1997 to 2017. The maximum sequence length per user is 13,122 behaviors.
- Industrial Dataset (Proprietary): Collected from Alibaba's visit and order logs between 2016 and 2021. This dataset represents true full-lifecycle scale, featuring hundreds of millions of users and a maximum sequence length of 1,952,546 behaviors per user.
Key Insights and Ideas
- More Data Beats Shorter Windows (Especially for Profiling): Incorporating years of historical data significantly boosts performance on user profiling tasks (like age or gender prediction). Preferences from years ago still hold valuable clues about a user's demographic profile, whereas category preference prediction peaks with 1–2 years of history.
- Anchors Enable Lossless Compression: The difference in downstream performance between using the raw, high-dimensional BoI features and the highly compressed SMEN embeddings is negligible. The multi-anchor mechanism acts as an incredibly efficient bottleneck, preserving multi-faceted user information.
- Multi-Scale Modeling Prevents Forgetting: By feeding both monthly and yearly aggregated behaviors into the model, LURM bypasses the catastrophic forgetting typical of standard recurrent networks processing long sequences.
Key Results
LURM was evaluated on downstream tasks including category preference identification and user profiling, using simple MLPs trained on top of the frozen embeddings.
- Amazon Dataset: LURM consistently outperformed state-of-the-art unsupervised baselines (like PTUM and SUMN), achieving an average improvement of 3.23% in AUC over SUMN. Remarkably, it also outperformed strong supervised baselines like TextCNN and HAN by 8.26% and 3.82% in average AUC, respectively.
- Industrial Dataset: To prove the value of full-lifecycle modeling, the authors compared LURM using the full history against a version restricted to the last two months of data (
LURM(2M)). Full-history LURM achieved an absolute jump in Age Classification AUC from 88.48% to 96.09%, demonstrating the massive information gain from long-term behavior tracking.
Limitations and Future Work
While LURM demonstrates powerful capabilities for text-based behavior logs (such as item titles), the current implementation focuses primarily on textual data. The authors note that real-world user interactions are inherently multi-modal.
Future Directions: * Extending the Bag-of-Interests framework to natively support multi-modal inputs, including images, videos, and audio. * Exploring more dedicated and advanced network architectures for the time-aggregation component. * Testing the learned representations across an even wider variety of downstream industrial tasks.
Summary updated