Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling

Bei Yang, Jie Gu, Ke Liu, Xiaoxiao Xu, Renjun Xu, Qinghui Sun, Hong Liu

Abstract

This paper introduces LURM (Full-Life cycle User Representation Model), a framework designed to generate general-purpose user embeddings from a user's entire history—potentially encompassing millions of behaviors—since registration. By integrating a "Bag-of-Interests" (BoI) encoding with a self-supervised multi-anchor encoder network, LURM captures diverse and evolving user preferences more effectively than existing models limited to short-term behavior sequences. Evaluations on massive datasets demonstrate that LURM significantly outperforms state-of-the-art methods in user profiling and preference prediction.

Introduction

In modern recommendation systems, user modeling typically follows two paradigms: task-specific models and general-purpose models. While general-purpose models are more scalable and economical, they face a significant technical hurdle: the long-sequence problem. Standard architectures like RNNs or Transformers struggle to process more than a few hundred behaviors due to memory constraints and the risk of "catastrophic forgetting."

However, a user's true identity and long-term interests are often buried in their full-life cycle data. The authors propose LURM to bridge this gap, enabling the processing of years of behavioral data without losing the fine-grained details of user activity.

Methodology

The LURM framework consists of two cascaded stages designed for efficiency and representational power.

Figure 1: Overview of the LURM framework, illustrating the Bag-of-Interests (BoI) encoding and the Self-supervised Multi-anchor Encoder Network (SMEN).

Bag-of-Interests (BoI)

Inspired by the "Bag-of-Words" concept in NLP, this module clusters billions of items into a fixed vocabulary of approximately $10^5$ "interest clusters." A user's behavior within a specific time period is represented as a sparse, high-dimensional vector based on these clusters. This approach allows the model to handle an arbitrary number of behaviors efficiently by abstracting individual items into broader interest categories.

Self-supervised Multi-anchor Encoder Network (SMEN)

The SMEN maps BoI sequences into low-dimensional embeddings using three key components:

  1. Multi-anchor Module: Instead of compressing a user into a single vector, this module uses multiple "anchors" to capture different facets of a user's personality (e.g., sports, parenting, technology).
  2. Time Aggregation: Gated Recurrent Units (GRUs) are employed to model the temporal evolution of these interests.
  3. Multi-scale Aggregation: The system processes data at different granularities (e.g., monthly vs. yearly) to balance short-term trends with long-term stability.
Figure 6: Detailed architecture of the Multi-anchor Module and its attention mechanism, which allows for the extraction of multifaceted user interests from sparse BoI features.

The entire system is trained via contrastive learning, where the model learns to associate different time-slices of behavior belonging to the same user, ensuring the embeddings are robust and representative.

Experimental Evaluation

The authors evaluated LURM on two massive datasets:

  • Amazon Dataset: Product reviews and metadata from 1997 to 2017, covering 43 million users.
  • Industrial Dataset: Real-world visit and order logs from Alibaba for over 214 million users, spanning 2016 to 2021.

Key Results

  • Superiority over SOTA: LURM consistently outperformed state-of-the-art self-supervised models like PTUM and SUMN, achieving an average AUC improvement of ~3.23% on the Amazon dataset.
  • Performance vs. Supervised Models: LURM embeddings often outperformed task-specific supervised models (e.g., TextCNN, HAN), likely because LURM could leverage much longer behavior sequences.
  • Impact of History Length: User profiling (e.g., age/gender prediction) benefited significantly from longer histories. Moving from 2 months to 5 years of data improved age prediction AUC by nearly 10% on the industrial dataset.

Conclusion and Future Work

LURM demonstrates that "more is better" when it comes to user behavioral data, provided the architecture can handle the scale. The BoI and Multi-anchor approaches provide a scalable way to compress massive histories into useful embeddings. Future work will focus on incorporating multi-modal data (images, video) and testing the framework on a wider variety of downstream recommendation tasks.

Summary updated

Abstract

User Modeling plays an essential role in industry. In this field, task-agnostic approaches, which generate general-purpose representation applicable to diverse downstream user cognition tasks, is a promising direction being more valuable and economical than task-specific representation learning. With the rapid development of Internet service platforms, user behaviors have been accumulated continuously. However, existing general-purpose user representation researches have little ability for full-life cycle modeling on extremely long behavior sequences since user registration. In this study, we propose a novel framework called full- Life cycle User Representation Model (LURM) to tackle this challenge. Specifically, LURM consists of two cascaded sub-models: (I) Bag-of-Interests (BoI) encodes user behaviors in any time period into a sparse vector with super-high dimension (e.g., 10^5); (II) Self-supervised Multi-anchor Encoder Network (SMEN) maps sequences of BoI features to multiple low-dimensional user representations. Specially, SMEN achieves almost lossless dimensionality reduction, benefiting from a novel multi-anchor module which can learn different aspects of user interests. Experiments on several benchmark datasets show that our approach outperforms state-of-the-art general-purpose representation methods.
BibTeX
@inproceedings{yang2022empowering,
  author = {Bei Yang and Jie Gu and Ke Liu and Xiaoxiao Xu and Renjun Xu and Qinghui Sun and Hong Liu},
  title = {{Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=YTtMaJUN_uc},
}
Figure