Pre-training General User Representation with Multi-type APP Behaviors

Yuren Zhang, Min Hou, Kai Zhang, Yuqing Yuan, Chao Song, Zhihao Ye

Abstract

Developing universal user representations in the mobile ecosystem is challenging due to the frequency disparity and semantic mismatch between different types of user behaviors. High-frequency app launches provide short-term interest signals, while low-frequency app installations reflect long-term demands. We propose the Multi-type App-usage Fusion Network (MAFN), which employs a Short-term Interest Module (SIM) for launch distribution prediction and a Long-term Interest Module (LIM) for semantic contrastive learning of installations. A shared discrete codebook in the Interest Fusion Module (IFM) bridges these behavioral signals. Experimental results on large-scale industrial datasets demonstrate that MAFN significantly improves performance across various downstream tasks and reduces online serving latency.

Introduction

In the mobile ecosystem, understanding user interests is vital for personalized services. However, traditional approaches usually train "task-specific" models (e.g., one model for ads, another for app recommendations), which is computationally expensive and fails to capture a holistic view of the user.

Existing general-purpose user modeling faces two major gaps:

  1. Frequency Disparity: User behaviors vary wildly in scale. App launches happen dozens of times a day (high-frequency), while app installations or uninstalls might happen once every few weeks (low-frequency). Most models struggle to balance these; they either drown out the rare but informative signals or get lost in the noise of repetitive daily launches.
  2. Semantic Mismatch: Short-term interests and long-term demands live in different "semantic spaces." Simply concatenating these vectors often leads to sub-optimal performance because the model doesn't know how to relate them.

The MAFN Framework

The proposed Multi-type App-usage Fusion Network (MAFN) consists of three specialized modules designed to address these challenges.

The MAFN architecture overview, illustrating the Pre-training and Fine-tuning phases.

Short-term Interest Module (SIM)

This module uses a BERT-based architecture to process high-frequency app launch data. Instead of predicting the "next app," which is often noisy, it introduces an App Distribution Prediction task. The model learns to predict the statistical distribution of apps a user will launch in the near future, proving more robust than standard next-item prediction.

Long-term Interest Module (LIM)

Focusing on (un)installation and retention, LIM uses a Transformer encoder-decoder structure. It employs Semantic Contrastive Learning, treating installations as positive signals and uninstalls as "hard negatives" to better understand user dislikes.

Interest Fusion Module (IFM)

The IFM acts as the "glue" of the system. To bridge the gap between SIM and LIM, it uses Vector Quantization (VQ) with a shared "codebook" of learnable prototype vectors. Both short-term and long-term embeddings are projected into this shared discrete space, ensuring they speak the same "language" before being fused into a final user representation.

Experimental Results

The study utilizes massive, anonymized industrial datasets from a leading global smartphone manufacturer, including 5.5 million users for pre-training and various downstream tasks such as Ad CVR Prediction and User Attribute Prediction.

MAFN consistently outperformed state-of-the-art baselines across all tasks, showing relative improvements of 1.2% to 2.6% in AUC. Beyond accuracy, MAFN offers significant efficiency gains.

Comparison of offline training and online serving times for MAFN vs. baselines.

As shown in Figure 2, while pre-training is computationally intensive, the resulting 64-dimensional embeddings are extremely lightweight for online serving. In simulated tests, MAFN nearly halved online serving latency compared to traditional end-to-end sequential models. Qualitative analysis via t-SNE visualizations further confirmed that MAFN creates more distinct user clusters, capturing more nuanced behavioral differences.

Conclusion

MAFN provides a robust framework for fusing disparate behavioral signals into a unified user representation. By treating uninstalls as active negative feedback and using a shared codebook for fusion, it overcomes the limitations of previous general-purpose models. Future work may explore incorporating additional mobile signals, such as system settings or hardware interactions, to further refine these "digital twins" of users.

Summary updated

BibTeX
@inproceedings{zhang2024pre,
  author = {Yuren Zhang and Min Hou and Kai Zhang and Yuqing Yuan and Chao Song and Zhihao Ye},
  title = {{Pre-training General User Representation with Multi-type APP Behaviors}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=0TFRqtCVc4},
}
Figure