Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling

Bei Yang, Jie Gu, Ke Liu, Xiaoxiao Xu, Renjun Xu, Qinghui Sun, Hong Liu

TL;DR

The paper introduces LURM (full-Life cycle User Representation Model), a self-supervised framework designed to learn general-purpose user embeddings from extremely long behavior histories spanning a user's entire lifecycle. By combining a novel "Bag-of-Interests" aggregation strategy with a multi-anchor encoder network, LURM successfully processes sequences containing millions of actions, achieving almost lossless dimensionality reduction and outperforming state-of-the-art methods on diverse downstream tasks.

What Problem Does This Paper Address?

In modern web platforms, understanding user preferences is vital for recommendation systems, search engines, and targeted advertising. Traditional approaches rely on task-specific user representations, training a separate model for every single application. This is computationally expensive and scales poorly.

While general-purpose (task-agnostic) user embeddings offer a more scalable alternative, existing models suffer from a major bottleneck: they can only handle short behavior sequences (typically tens or hundreds of actions). In reality, users accumulate thousands or even millions of behaviors over years of platform interaction. Standard sequential architectures like RNNs or Transformers fail at this scale due to memory constraints, high computational overhead, and catastrophic forgetting. This paper addresses the challenge of efficiently encoding a user's full-lifecycle behavior history into a compact, highly informative representation.

Methods

LURM solves the long-sequence bottleneck using a two-stage cascaded architecture:

  1. Bag-of-Interests (BoI): Instead of treating behaviors as a raw chronological sequence of billions of possible items (which is computationally intractable), BoI clusters items into a vocabulary of roughly $10^5$ distinct "interests." Item embeddings are first learned via a contrastive task, and then a large-scale clustering algorithm (HDSC) groups similar items. A user's behaviors within a specific timeframe are then summarized into a sparse, high-dimensional vector counting their interactions across these interest clusters.
  2. Self-supervised Multi-anchor Encoder Network (SMEN): This network compresses the high-dimensional BoI sequences into compact user embeddings through three key sub-modules:
  3. Multi-anchor Module: Unlike standard multi-head attention, this module uses fixed, learnable "anchors" representing different facets of user preferences. This disentangles user interests and enables almost lossless dimensionality reduction.
  4. Time Aggregation Module: Uses a Gated Recurrent Unit (GRU) to capture how user interests evolve over time.
  5. Multi-scale Aggregation Module: Processes behavior sequences at multiple granularities simultaneously (e.g., monthly and yearly views) to balance fine-grained short-term details with stable long-term trends.

The entire network is trained in a self-supervised manner using a contrastive loss function, where different sub-sequences from the same user serve as positive pairs.

Datasets

The authors evaluated LURM on two large-scale datasets:

  • Amazon Dataset (Public): Contains user product reviews and metadata spanning from 1997 to 2017. The maximum sequence length per user is 13,122 behaviors.
  • Industrial Dataset (Proprietary): Collected from Alibaba's visit and order logs between 2016 and 2021. This dataset represents true full-lifecycle scale, featuring hundreds of millions of users and a maximum sequence length of 1,952,546 behaviors per user.

Key Insights and Ideas

  • More Data Beats Shorter Windows (Especially for Profiling): Incorporating years of historical data significantly boosts performance on user profiling tasks (like age or gender prediction). Preferences from years ago still hold valuable clues about a user's demographic profile, whereas category preference prediction peaks with 1–2 years of history.
  • Anchors Enable Lossless Compression: The difference in downstream performance between using the raw, high-dimensional BoI features and the highly compressed SMEN embeddings is negligible. The multi-anchor mechanism acts as an incredibly efficient bottleneck, preserving multi-faceted user information.
  • Multi-Scale Modeling Prevents Forgetting: By feeding both monthly and yearly aggregated behaviors into the model, LURM bypasses the catastrophic forgetting typical of standard recurrent networks processing long sequences.

Key Results

LURM was evaluated on downstream tasks including category preference identification and user profiling, using simple MLPs trained on top of the frozen embeddings.

  • Amazon Dataset: LURM consistently outperformed state-of-the-art unsupervised baselines (like PTUM and SUMN), achieving an average improvement of 3.23% in AUC over SUMN. Remarkably, it also outperformed strong supervised baselines like TextCNN and HAN by 8.26% and 3.82% in average AUC, respectively.
  • Industrial Dataset: To prove the value of full-lifecycle modeling, the authors compared LURM using the full history against a version restricted to the last two months of data (LURM(2M)). Full-history LURM achieved an absolute jump in Age Classification AUC from 88.48% to 96.09%, demonstrating the massive information gain from long-term behavior tracking.

Limitations and Future Work

While LURM demonstrates powerful capabilities for text-based behavior logs (such as item titles), the current implementation focuses primarily on textual data. The authors note that real-world user interactions are inherently multi-modal.

Future Directions: * Extending the Bag-of-Interests framework to natively support multi-modal inputs, including images, videos, and audio. * Exploring more dedicated and advanced network architectures for the time-aggregation component. * Testing the learned representations across an even wider variety of downstream industrial tasks.

Summary updated

Abstract

User Modeling plays an essential role in industry. In this field, task-agnostic approaches, which generate general-purpose representation applicable to diverse downstream user cognition tasks, is a promising direction being more valuable and economical than task-specific representation learning. With the rapid development of Internet service platforms, user behaviors have been accumulated continuously. However, existing general-purpose user representation researches have little ability for full-life cycle modeling on extremely long behavior sequences since user registration. In this study, we propose a novel framework called full- Life cycle User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (I) Bag-of-Interests (BoI) encodes user behaviors in any time period into a sparse vector with super-high dimension (e.g., 10^5); (II) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of BoI features to multiple low-dimensional user representations. Specially, SMEN achieves almost lossless dimensionality reduction, benefiting from a novel multi-anchor module which can learn different aspects of user interests. Experiments on several benchmark datasets show that our approach outperforms state-of-the-art general-purpose representation methods.
BibTeX
@inproceedings{yang2022empowering,
  author = {Bei Yang and Jie Gu and Ke Liu and Xiaoxiao Xu and Renjun Xu and Qinghui Sun and Hong Liu},
  title = {{Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=YTtMaJUN_uc},
}
Figure