Double-Wing Mixture of Experts for Streaming Recommendations

Yan Zhao, Shoujin Wang, Yan Wang, Hongwei Liu, Weizhe Zhang

Abstract

Streaming recommender systems (SRSs) face the dual challenge of adapting to rapid user preference drift while maintaining long-term memory of stable habits. Furthermore, the inherent heterogeneity of users and items often exceeds the capacity of monolithic neural models. This paper introduces VRS-DWMoE, a framework that integrates Variational and Reservoir-enhanced Sampling (VRS) with a "Double-Wing" Mixture of Experts (DWMoE) architecture. By decoupling user and item modeling into specialized expert sub-networks and employing a time-decayed reservoir for data balancing, VRS-DWMoE significantly outperforms existing state-of-the-art streaming recommenders across multiple real-world datasets.

Introduction

In modern e-commerce and content platforms, user interaction data arrives as a continuous stream. To remain relevant, recommender systems must update in real-time. However, standard approaches often struggle with two critical issues:

  1. Preference Drift vs. Long-Term Memory: Focusing solely on the latest data allows models to capture current trends but leads to "catastrophic forgetting" of stable, long-term user preferences.
  2. Data Heterogeneity: A single model often fails to capture the diverse characteristics of different user segments (e.g., bargain hunters vs. luxury buyers) and item categories simultaneously.

Methods: The VRS-DWMoE Framework

The proposed VRS-DWMoE framework addresses these challenges through two primary modules: Variational and Reservoir-enhanced Sampling and the Double-Wing Mixture of Experts.

The VRS-DWMoE architecture, featuring the Double-Wing Mixture of Experts and Variational Reservoir-enhanced Sampling.

Variational and Reservoir-enhanced Sampling (VRS)

To prevent the loss of long-term memory, VRS maintains a "reservoir" of historical interactions. Instead of training only on new data, the system dynamically calculates a sampling ratio to mix new arrivals with historical samples. A time-decay function ensures that more recent historical data has a higher probability of being selected, effectively balancing the need for immediate adaptation with the preservation of core user profiles.

Double-Wing Mixture of Experts (DWMoE)

The core recommendation engine is split into two independent "wings": the Mixture of User Experts (MoUE) and the Mixture of Item Experts (MoIE).

  • Specialized Experts: Each wing contains multiple sub-networks (experts) that learn to specialize in specific latent types of users or items.
  • Gating Networks: A gating mechanism evaluates incoming user/item IDs and assigns weights to each expert's output, fusing them into a final representation.
  • Load Balancing: To prevent "expert collapse"—where the model relies on a single expert—the authors use a custom regularization term based on the standard deviation of gating weights, forcing the model to utilize the full capacity of the expert pool.

Experimental Evaluation

The framework was evaluated on three benchmark datasets: MovieLens (1M), Netflix (a 5,000-user subset), and Yelp. Explicit ratings were converted to implicit feedback to simulate real-world interaction streams.

Key Results

  • Superior Accuracy: VRS-DWMoE consistently outperformed eight state-of-the-art baselines, including SPMF and iNeuMF. It achieved an average improvement of 8.5% in Hit Ratio (HR@10) and 10.4% in NDCG@10.
  • Sampling Efficacy: The VRS method proved superior to standard strategies like "New Data Only" or simple random reservoir sampling, validating the importance of the time-decay approach.
  • Scalability of Experts: Performance improved as the number of experts increased (from 2 to 8), demonstrating that increased model capacity through specialization directly translates to better recommendations.

Discussion and Future Work

The "Double-Wing" approach effectively handles data heterogeneity by allowing specialized sub-networks to emerge. However, the current implementation uses a symmetric number of experts for both users and items. Future research could explore asymmetric configurations, as the diversity of items in a catalog might differ significantly from the diversity of the user base. Additionally, while the computational overhead of multiple experts is a factor, the gains in recommendation quality suggest that MoE architectures are a promising direction for high-performance streaming systems.

Summary updated

BibTeX
@inproceedings{zhao2020double,
  author = {Yan Zhao and Shoujin Wang and Yan Wang and Hongwei Liu and Weizhe Zhang},
  title = {{Double-Wing Mixture of Experts for Streaming Recommendations}},
  booktitle = {WISE},
  year = {2020},
  url = {https://openreview.net/forum?id=SlYgAya3el},
}
Figure