Pre-training General User Representation with Multi-type APP Behaviors

Yuren Zhang, Min Hou, Kai Zhang, Yuqing Yuan, Chao Song, Zhihao Ye, Enhong Chen, Yang Yu

Abstract

In numerous user-centric services on mobile applications (apps), accurately mining user interests and generating effective user representations are paramount. Traditional approaches, which often involve training task-specific user representations, are becoming increasingly impractical due to their high computational costs and limited adaptability. This paper introduces a novel solution to this challenge: the Multi-type App-usage Fusion Network (MAFN). MAFN innovatively pre-trains universal user representations, leveraging multi-type app behaviors to overcome key limitations in existing methods. We address two primary challenges: 1) the varying frequency of user behaviors (ranging from low-frequency actions like (un)installations to high-frequency yet insightful app launches); and 2) the integration of multi-type behaviors to form a cohesive representation. Our approach involves the creation of novel pre-training tasks that harness self-supervised signals from diverse app behaviors, capturing both long-term and short-term user interests. MAFN's unique fusion approach effectively amalgamates these interests into a unified vector space, facilitating the development of a versatile, general-purpose user representation. With a practical workflow, extensive experiments with three typical downstream tasks on real-world datasets verify the effectiveness of our approach.

Introduction

Understanding user interests is critical for personalization in the mobile ecosystem. However, current practices often rely on building isolated, task-specific models for different applications like advertising or content recommendation. This approach is not only computationally expensive but also fails to capture a holistic "digital twin" of the user.

Two major hurdles prevent the creation of a general-purpose user representation:

  1. Frequency Disparity: User behaviors occur at vastly different scales, from daily app launches to infrequent app uninstalls.
  2. Semantic Mismatch: Short-term "bursty" actions and long-term stable preferences reside in different semantic spaces, making simple concatenation ineffective.

To address these, we propose the Multi-type App-usage Fusion Network (MAFN), a framework designed to pre-train universal user embeddings by bridging these gaps.

Methodology: The MAFN Architecture

MAFN employs a "divide and conquer" strategy followed by a sophisticated fusion mechanism to integrate diverse behavioral signals.

The overall architecture of MAFN, illustrating the Pre-training and Fine-tuning phases.

Short-term Interest Module (SIM)

The SIM processes high-frequency app launch data using a BERT-based architecture. To avoid the noise of exact sequence prediction, it utilizes an App Distribution Prediction task, learning to predict the frequency distribution of apps a user is likely to open in the near future.

Long-term Interest Module (LIM)

This module handles low-frequency events like app (un)installations. It uses a Transformer encoder-decoder for sequence reconstruction and a Semantic Contrastive Learning task. This task explicitly models user preferences by positioning "installed" apps closer to neutral apps than to "uninstalled" apps, capturing active dislike.

Interest Fusion Module (IFM)

The IFM is the core component that merges disparate interests. It utilizes Vector Quantization (VQ) with a shared "codebook" of anchor vectors. This acts as a universal translator, projecting both short- and long-term interests into a unified, cohesive vector space.

User Contrastive Learning

A final pre-training step aligns the high-frequency and low-frequency representations of the same user, ensuring the model focuses on the underlying persona rather than raw data noise.

Experimental Evaluation

The framework was evaluated using massive, anonymized industrial data from a global smartphone manufacturer, involving 5.5 million users and over 10,000 apps.

Performance Across Downstream Tasks

MAFN consistently outperformed state-of-the-art baselines like BERT4Rec and AETN across three key tasks:

  • Ad CVR Prediction: Achieved a 1.2% improvement in AUC.
  • User Attribute Prediction: Saw a 2.6% improvement in AUC, indicating effective capture of demographic traits.
  • Game Player Recall: Improved identification of churned players likely to return.

Embedding Quality and Efficiency

The quality of the learned representations is visually evident in t-SNE visualizations, where MAFN shows superior clustering of user interests compared to baselines.

t-SNE visualization comparing user embeddings from AETN (left) and MAFN (right), showing clearer interest clustering in MAFN.

While offline pre-training is intensive (~33 hours), the online serving is highly efficient. Integrating the 64-dimensional MAFN embedding into production models provides a significant performance lift with negligible latency.

Conclusion

MAFN demonstrates that pre-training universal user representations from multi-type app behaviors is both feasible and highly effective. By reconciling frequency disparities and semantic mismatches through a novel fusion network, MAFN provides a robust foundation for various downstream mobile services. Future work will explore even more diverse behavioral signals and parameter-efficient fine-tuning methods to further enhance scalability.

Summary updated

Abstract

In numerous user-centric services on mobile applications (apps), accurately mining user interests and generating effective user representations are paramount. Traditional approaches, which often involve training task-specific user representations, are becoming increasingly impractical due to their high computational costs and limited adaptability. This paper introduces a novel solution to this challenge: the Multi-type App-usage Fusion Network (MAFN). MAFN innovatively pre-trains universal user representations, leveraging multi-type app behaviors to overcome key limitations in existing methods. We address two primary challenges: 1) the varying frequency of user behaviors (ranging from low-frequency actions like (un)installations to high-frequency yet insightful app launches); and 2) the integration of multi-type behaviors to form a cohesive representation. Our approach involves the creation of novel pre-training tasks that harness self-supervised signals from diverse app behaviors, capturing both long-term and short-term user interests. MAFN's unique fusion approach effectively amalgamates these interests into a unified vector space, facilitating the development of a versatile, general-purpose user representation. With a practical workflow, extensive experiments with three typical downstream tasks on real-world datasets verify the effectiveness of our approach.
BibTeX
@inproceedings{zhang2024pre,
  author = {Yuren Zhang and Min Hou and Kai Zhang and Yuqing Yuan and Chao Song and Zhihao Ye and Enhong Chen and Yang Yu},
  title = {{Pre-training General User Representation with Multi-type APP Behaviors}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=0TFRqtCVc4},
}
Figure