Transferable and Forecastable User Targeting Foundation Model

Bin Dou, Baokun Wang, Yun Zhu, Xiaotong Lin, Yike Xu, Xiaorui Huang, Yang Chen, Yun Liu, Shaoshuai Han, Yongchao Liu, Tianyi Zhang, Yu Cheng, Weiqiang Wang, Chuntao Hong

Abstract

User targeting, the process of selecting targeted users from a pool of candidates for non-expert marketers, has garnered substantial attention with the advancements in digital marketing. However, existing user targeting methods encounter two significant challenges: (i) Poor cross-domain and cross-scenario transferability and generalization, and (ii) Insufficient forecastability in real-world applications. These limitations hinder their applicability across diverse industrial scenarios. In this work, we propose FOUND, an industrial-grade, transferable, and forecastable user targeting foundation model. To enhance cross-domain transferability, our framework integrates heterogeneous multi-scenario user data, aligning them with one-sentence targeting demand inputs through contrastive pre-training. For improved forecastability, the text description of each user is derived based on anticipated future behaviors, while user representations are constructed from historical information. Experimental results demonstrate that our approach significantly outperforms existing baselines in cross-domain, real-world user targeting scenarios, showcasing the superior capabilities of FOUND. Moreover, our method has been successfully deployed on the Alipay platform and is widely utilized across various scenarios.

Introduction

In digital marketing, user targeting is essential for identifying the right audience for specific campaigns. Despite its importance, traditional methods face two major bottlenecks. First, poor transferability limits models to narrow silos; a model trained for e-commerce often fails in financial services. Second, a lack of forecastability means models focus on past actions rather than predicting future behavior. While Large Language Models (LLMs) excel at text understanding, they struggle with complex, "long-tail" industrial data like encrypted billing records. FOUND (FOrecastable User targeting fouNDation model) bridges this gap by creating a unified representation that understands both raw industrial data and human language.

The FOUND Framework

The core of FOUND is a sophisticated two-stage pre-training pipeline designed to create robust, multi-modal user representations.

Stage 1: Self-Supervised User Modeling

The model ingests three primary data types: behavioral sequences (e.g., mini-program logs), tabular data (e.g., fund flows), and search text. It utilizes a TabTransformer for tabular data and ALBERT for sequences. To capture the periodic nature of human behavior, the researchers introduced "Cyclic Regularization" and used "Contrastive Predictive Coding" to forecast future states from historical sequences.

Stage 2: User-Text Alignment

This stage aligns fused user embeddings with text descriptions of future behaviors generated by a fine-tuned LLM (Qwen2). By employing a contrastive loss—similar to the CLIP architecture but for users and behavior descriptions—the model learns to associate a user's current profile with their likely future actions.

The two-stage pre-training pipeline of FOUND, involving self-supervised user modeling and user-text alignment.

Inference and Deployment

FOUND supports flexible inference strategies that cater to different marketing needs:

  • Zero-shot Targeting: Marketers can input simple natural language demands, such as "Find potential 3C buyers," and the model retrieves relevant users directly.
  • Few-shot Prompt Tuning: When a small set of "seed users" is available, the model uses them to tune a learnable prompt, significantly increasing accuracy for niche or high-risk tasks.
Inference strategies for FOUND: Zero-shot targeting via natural language and few-shot targeting using seed users.

Experimental Results and Impact

FOUND was trained on a massive dataset from the Alipay platform, comprising approximately 500 million user-text pairs. Evaluation across 13 diverse scenarios (including account security and marketing) showed that FOUND consistently outperforms state-of-the-art baselines:

  • Zero-shot Performance: FOUND achieved a 10% to 15% improvement in Accuracy, Precision, and Recall compared to fine-tuned LLM baselines.
  • Few-shot Performance: With only 10 seed users, FOUND outperformed baselines by 5% to 8%.
  • Real-world Impact: Deployed on Alipay, the model led to a 30% increase in Click-Through Rate (CTR) for "Used Car Selling" and a 13.2% increase for "Video Watching" scenarios.

Key Insights

  • Forecasting via Alignment: By treating targeting as a retrieval task between current features and future behavior descriptions, the model becomes inherently forecastable.
  • Modality Importance: Ablation studies highlighted that user behavioral sequences are the most critical data modality for model performance.
  • Bridging the Gap: A "Query Rewriting" module translates vague, non-expert demands into structured templates, making the model accessible to non-technical marketers.

Conclusion and Future Work

While FOUND represents a significant step forward, challenges remain, including high training costs (requiring 16 A100 GPUs for 160 hours) and platform-specific optimization. Future work will focus on incorporating more modalities like images and knowledge graphs, while reducing computational overhead.

Summary updated

Abstract

User targeting, the process of selecting targeted users from a pool of candidates for non-expert marketers, has garnered substantial attention with the advancements in digital marketing. However, existing user targeting methods encounter two significant challenges: (i) Poor cross-domain and cross-scenario transferability and generalization, and (ii) Insufficient forecastability in real-world applications. These limitations hinder their applicability across diverse industrial scenarios. In this work, we propose FOUND, an industrial-grade, transferable, and forecastable user targeting foundation model. To enhance cross-domain transferability, our framework integrates heterogeneous multi-scenario user data, aligning them with one-sentence targeting demand inputs through contrastive pre-training. For improved forecastability, the text description of each user is derived based on anticipated future behaviors, while user representations are constructed from historical information. Experimental results demonstrate that our approach significantly outperforms existing baselines in cross-domain, real-world user targeting scenarios, showcasing the superior capabilities of FOUND. Moreover, our method has been successfully deployed on the Alipay platform and is widely utilized across various scenarios.
BibTeX
@inproceedings{dou2024transferable,
  author = {Bin Dou and Baokun Wang and Yun Zhu and Xiaotong Lin and Yike Xu and Xiaorui Huang and Yang Chen and Yun Liu and Shaoshuai Han and Yongchao Liu and Tianyi Zhang and Yu Cheng and Weiqiang Wang and Chuntao Hong},
  title = {{Transferable and Forecastable User Targeting Foundation Model}},
  booktitle = {The Web Conference},
  year = {2024},
  doi = {10.1145/3701716.3715266},
}
Figure