TL;DR
This paper introduces the Bagging-Expert Network (BEnet), a novel approach to tackle the "polarization" problem in Multi-gate Mixture-of-Experts (MMoE) models used for multi-task learning. By incorporating a bagging layer and an attention mechanism, BEnet encourages experts to specialize in distinct feature subsets, leading to improved performance and robustness in recommendation systems.
What Problem Does This Paper Address?
Multi-task learning (MTL) is a powerful technique for improving model performance by training on multiple related tasks simultaneously. Recommendation systems, for instance, often use MTL to predict user engagement metrics like click-through rate (CTR) and conversion rate (CVR). The Multi-gate Mixture-of-Experts (MMoE) is a popular MTL architecture that uses multiple "experts" (sub-networks) and "gates" (routing mechanisms) to learn from different tasks. However, MMoE suffers from a phenomenon called "polarization" during training. This means that the weights assigned to some experts by the gates tend to converge to zero, effectively rendering those experts useless. This leads to underutilization of model capacity, potential performance degradation, and wasted computational resources. The paper aims to solve this polarization problem to make MMoE models more effective and efficient.
Methods
The core innovation of BEnet lies in its two key components:
- Bagging Layer: Inspired by the "bagging" technique used in random forests, this layer randomly selects a subset of contextual features for each expert and gate. This ensures that different experts receive slightly different views of the data, promoting diversity and preventing them from converging on the same information.
- Attention Mechanism: Within each expert, an attention mechanism is employed to weigh the importance of the bagged contextual features. This allows experts to focus on the most relevant information within their assigned feature subset, further enhancing specialization and filtering out noise.
These components are integrated into the standard MMoE architecture. The bagging layer is applied to contextual features before they are fed into the experts and gates. The attention mechanism then refines the contextual features within each expert.
Datasets
The authors evaluated BEnet on three datasets:
- MultiMNIST: An extended multi-digit classification benchmark.
- AliExpress: Real-world traffic logs from AliExpress's search system (Spanish and French subsets).
- Company-A: A proprietary dataset from an e-commerce company's coupon recommendation campaigns.
Key Insights and Ideas
- Diversity is Key to Avoiding Polarization: The paper strongly suggests that the root cause of polarization is a lack of diversity among experts. When experts are forced to learn from the same data, some naturally become dominant, leading to others being ignored.
- Bagging as a Feature-Level Diversifier: Applying bagging at the feature level is a novel way to introduce this crucial diversity into MMoE, making it more robust than methods like dropout that only randomly disable connections.
- Attention Enhances Expert Focus: The attention mechanism complements bagging by allowing experts to intelligently select the most important features from their assigned subsets, leading to more specialized and effective learning.
- Real-World Validation is Crucial: The paper goes beyond offline experiments to include an online A/B test, demonstrating BEnet's practical effectiveness in a live e-commerce environment.
Key Results
- Improved Performance: BEnet consistently outperformed baseline methods (MMoE, MMoE-dropout, OMoE, PLE, ESMM) across all three datasets in terms of accuracy (ACC) and Area Under Curve (AUC). For example, on the Company-A dataset, BEnet achieved the best performance for CTR, CVR, and TOA tasks.
- Reduced Polarization: Figures 3 and 4 visually demonstrate that BEnet exhibits smaller differences in gate weights between experts compared to original MMoE and MMoE-dropout, indicating a more balanced utilization of experts.
- Online A/B Test Success: In an online A/B test on a coupon recommendation campaign, BEnet achieved a notable improvement of +8.5% in send rate and +9.3% in use rate compared to the online baseline (ESMM).
- Bagging Ratio Sensitivity: Ablation studies (Figure 5) showed that the performance of BEnet is sensitive to the bagging ratios for both expert and gate features, suggesting that careful tuning is important for optimal results.
Limitations and Future Work
- Bagging Ratio Tuning: The paper notes that optimal performance requires careful fine-tuning of the bagging ratio parameters for both experts and gates, which can be a hyperparameter optimization challenge.
- Dataset Dependency: While BEnet generally mitigates polarization, the paper acknowledges that the degree of mitigation can depend on the dataset's nature, with MMoE-dropout showing comparable results in some AliExpress scenarios.
- Future Work: The authors suggest further exploration into optimizing bagging ratios and potentially applying BEnet to other complex MTL scenarios beyond recommendation systems. The successful deployment in industrial settings also points to its practical applicability for future research.
Summary updated