TL;DR
This paper introduces an "Interest-oriented Universal User Representation" framework that leverages contrastive self-supervised learning and a novel multi-interest extraction module. This approach aims to create a single, general-purpose user representation that can be effectively applied across various downstream tasks, reducing the need for task-specific model training and improving generalization.
What Problem Does This Paper Address?
In the realm of personalized services like e-commerce recommendations and advertising, accurately representing users is crucial. Traditionally, this involves training specialized models for each specific application, which is computationally expensive and time-consuming, especially when dealing with dozens or hundreds of downstream tasks. The core challenge is to develop a universal user representation – a single, low-dimensional embedding that captures a user's diverse interests and behaviors without being tied to any particular task. This universal representation should generalize well and be directly usable by downstream applications with minimal fine-tuning. Existing methods often struggle with this generalization due to task-specific training or limitations in capturing the full spectrum of user interests.
Methods
The proposed approach has two main components:
- Interest-oriented Contrastive Learning: This is a novel self-supervised learning paradigm designed to guide the representation model. The core idea is to treat representations of the same user from different time periods as positive pairs and representations from different users as negative pairs. By minimizing the distance between positive pairs and maximizing it for negative pairs, the model learns to capture user interests that are consistent over time. This framework is unified, allowing it to learn both short-term and long-term interests by simply adjusting the time span of the input behavior sequences.
- Multi-Interest Extraction Module: To address the challenge of capturing diverse user interests, the authors introduce a novel module that uses a trainable "interest dictionary." This dictionary acts as a set of anchor points representing principal user interests. For each user behavior, the module first identifies the most relevant interest from the dictionary. Then, it aggregates behavior embeddings based on these identified interests to generate multiple, interest-specific user representations. A regularization loss is also introduced to ensure that all interest vectors in the dictionary are sufficiently trained and not dominated by a few.
Datasets
The paper uses two real-world e-commerce datasets for training and evaluation:
- Amazon Dataset: Contains product reviews and metadata.
- Industry Dataset: Consists of search logs from a popular e-commerce platform (Taobao).
These datasets provide large-scale, unlabelled user behavior data for training the universal user representations and annotated data for evaluating performance on downstream tasks. The paper details how data was collected and split into historical, short-term, and long-term behavior sets for training.
Key Insights and Ideas
- Contrastive Learning for User Interests: Applying contrastive learning at the representation level, rather than just at the word or item level, proves effective for capturing nuanced user interests. The temporal aspect of contrastive learning (comparing different time windows) is key to distinguishing short-term from long-term interests.
- Interest Dictionary as a Unified Anchor: The interest dictionary provides a structured way to infer and represent a user's principal interests, overcoming limitations of previous methods that might cluster behaviors independently or lack interpretability.
- Unified Framework for Short/Long-Term Interests: The ability to learn both short-term and long-term user representations within the same contrastive learning framework by simply varying the input time windows is a significant advantage.
- Generalization Beyond Text: The proposed interest dictionary aggregation module can be extended to incorporate structured data (like category IDs, shop IDs) beyond just text, broadening the applicability of the learned representations.
Key Results
The proposed "Ours-ID-ICL" method consistently outperforms state-of-the-art universal user representation methods (like SUMN and PTUM) and even supervised methods (like TextCNN) on various downstream tasks, including preference identification and user profiling. For instance, it achieves 1-3% AUC improvements on preference identification tasks and about 1.5% accuracy improvement on user profiling tasks compared to other unsupervised methods. Ablation studies confirm that both the Interest Dictionary Aggregation Module and the Interest-oriented Contrastive Learning are crucial for these performance gains.
Limitations and Future Work
- Hyperparameter Sensitivity: The performance is sensitive to hyperparameters like the size of the interest dictionary (M) and the number of interests per user (K). The optimal values can vary across datasets.
- Future Work: The authors suggest that the framework's ability to handle structured data can be further explored to improve performance by incorporating more diverse data modalities. They also imply that further research into optimizing hyperparameter selection could be beneficial.
Summary updated