Abstract
In large-scale e-commerce ecosystems, developing specialized models for thousands of concurrent tasks is inefficient and often suffers from data sparsity. We present GUIM (General User and Item embedding with Mixture of representation), a pre-training framework designed to generate universal user and item embeddings. Unlike traditional methods that compress user interests into a single vector, GUIM employs a Mixture of Representation (MoR) strategy, utilizing multiple vectors to capture diverse consumer behaviors. Evaluated on massive real-world datasets from Alibaba, GUIM demonstrates significant improvements in recommendation and profile prediction tasks, establishing a robust foundation for e-commerce intelligence.
Introduction
Modern e-commerce platforms like Alibaba operate at a scale where thousands of machine learning tasks—ranging from personalized recommendations to demographic profiling—run simultaneously. Traditionally, these tasks are solved with "end-to-end" models, which are labor-intensive to build and require vast amounts of task-specific labeled data. This approach leads to redundant computation and creates a "cold start" barrier for newer or smaller business units that lack historical data.
To address these challenges, researchers at Alibaba developed GUIM. Inspired by the success of BERT in Natural Language Processing, GUIM serves as a foundation model for e-commerce, providing general-purpose representations that can be "plugged into" various downstream applications without the need for extensive task-specific training.
Methodology
GUIM adapts the Transformer-based architecture to model user interaction sequences. The model processes sequences where each "token" represents an item, enriched with category IDs, title descriptions (processed via word embeddings), and temporal metadata to capture the timing of purchases.
Mixture of Representation (MoR)
The primary innovation of GUIM is the Mixture of Representation (MoR) layer. Standard embedding models typically represent a user as a single point in a high-dimensional space. However, a single vector often fails to capture the multi-faceted interests of a consumer (e.g., a user who buys both "baby diapers" and "high-end gaming CPUs"). GUIM introduces $C$ distinct [CLS] tokens, resulting in a set of $C$ vectors to represent each user, effectively breaking the mathematical bottleneck of single-vector representations.
Pre-training and Contrastive Learning
The model is trained using two primary objectives:
- Masked Language Modeling (MLM): Predicting masked items within a sequence to learn local item dependencies.
- Matching Task: Predicting future purchases based on the multi-vector user representation using a "max-pooling" score function.
Given the extreme vocabulary size of over 100 million items, calculating a full softmax is computationally impossible. GUIM utilizes the InfoNCE (Information Noise-Contrastive Estimation) loss, which treats the task as a contrastive learning problem by sampling negative examples, making training at Alibaba's scale feasible.
Evaluation and Results
We evaluated GUIM using datasets from Taobao and Tmall, comprising 20 million users and nearly 100 million unique items. Performance was measured across two benchmark suites: Consumer Matching Prediction (CMP) and Consumer Profile Prediction (CPP), which includes tasks like predicting if a user has a child or the age group of their baby.
Impact of Representation Diversity
A key finding is that increasing the number of user vectors ($C$) significantly enhances performance across all matching tasks. In the Consumer Matching Prediction (CMP-L) task, which focuses on long-term interests, increasing the vector set size from 1 to 4 yielded a 5.68% improvement in Recall@100.
As shown in the results, the MoR approach consistently outperforms "Embedding Dimension Increasing" (EDI) baselines. A GUIM model with 4 vectors is more effective than a single-vector model with four times the dimensionality, suggesting that architectural diversity is more parameter-efficient than simple scaling.
Conclusion
GUIM demonstrates that BERT-like pre-training can be effectively scaled to the e-commerce domain. By moving beyond single-vector representations, the Mixture of Representation approach captures the inherent complexity of human shopping behavior. Future work will explore incorporating multi-modal data, such as item images, and extending the temporal scope of interaction sequences to capture multi-year seasonal dynamics and long-term user evolution.
Summary updated