Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint

Zhuojian Xiao, Yunjiang Jiang, Guoyu Tang, Lin Liu, Sulong Xu, Yun Xiao, Weipeng Yan

Abstract

Product search is the most common way for people to satisfy their shopping needs on e-commerce websites. Products are typically annotated with one of several broad categorical tags, such as "Clothing" or "Electronics", as well as finer-grained categories like "Refrigerator" or "TV", both under "Electronics". These tags are used to construct a hierarchy of query categories. Distributions of features such as price and brand popularity vary wildly across query categories. In addition, feature importance for the purpose of CTR/CVR predictions differs from one category to another. In this work, we leverage the Mixture of Expert (MoE) framework to learn a ranking model that specializes for each query category. In particular, our gate network relies solely on the category ids extracted from the user query. While classical MoE's pick expert towers spontaneously for each input example, we explore two techniques to establish more explicit and transparent connections between the experts and query categories. To help differentiate experts on their domain specialties, we introduce a form of adversarial regularization among the expert outputs, forcing them to disagree with one another. As a result, they tend to approach each prediction problem from different angles, rather than copying one another. This is validated by a much stronger clustering effect of the gate output vectors under different categories. In addition, soft gating constraints based on the categorical hierarchy are imposed to help similar products choose similar gate values and make them more likely to share similar experts. This allows aggregation of training data among smaller sibling categories to overcome data scarcity.

Introduction

In e-commerce search, a "one size fits all" ranking model often fails to capture the nuances of diverse product categories. Feature importance varies significantly; for instance, brand might be critical for electronics, while style is paramount for clothing. While the Mixture of Experts (MoE) framework offers a path toward specialization via gated sub-networks, standard implementations suffer from opaque specialization—where experts learn random data patterns rather than business-meaningful categories—and data scarcity in long-tail categories.

Methodology

To address these challenges, we propose the Adv & HSC-MoE model, which introduces two primary innovations to the standard MoE architecture:

Hierarchical Soft Constraint (HSC)

We utilize the existing business category hierarchy (e.g., Foods -> Cake) to guide expert selection. The model employs two gate networks: an Inference Gate for specific sub-categories and a Constraint Gate for broader top-level categories. The HSC loss function minimizes the L2 difference between these gates, forcing sibling categories to share experts. This mechanism effectively pools training data, allowing niche categories to benefit from the patterns learned in high-volume sibling categories.

Adversarial Mixture of Experts (Adv-MoE)

To ensure experts do not become redundant copies of one another, we introduce an AdvLoss. This adversarial regularization rewards "disagreeing experts"—those not selected for the final prediction—if their outputs differ significantly from the chosen experts. This forces the model to develop a robust ensemble of diverse viewpoints.

The Adv & HSC-MoE model architecture, detailing the interaction between Expert Networks, Gate Networks, and the dual loss functions (HSC and AdvLoss).

Experimental Results

Offline Evaluation

Testing on a massive JD.com dataset (26.6M instances) and the Amazon Review public dataset showed that the combined Adv & HSC-MoE model consistently outperformed standard DNN and vanilla MoE baselines. On the JD dataset, the model achieved a 0.96% improvement in AUC and a 0.99% boost in NDCG.

Clustering and Specialization

Visualization of the gate output vectors using t-SNE confirms the model's ability to learn category-aware representations. Unlike vanilla MoE, our model produces distinct clusters for semantically similar categories, validating that the gating mechanism is successfully specializing based on the category hierarchy.

t-SNE visualization of gate output vectors, demonstrating clear clustering for major categories such as Fashion, Electronics, and Daily Necessities.

Online Impact

In a six-day A/B test on JD.com’s live traffic, the proposed model delivered a 2.18% increase in Conversion Rate (CVR) and a 3.97% increase in Revenue Per Mille (RPM), demonstrating significant real-world business value.

Conclusion

By treating category hierarchy as a structural prior and enforcing expert diversity through adversarial loss, we have developed a ranking framework that excels in multi-category e-commerce environments. Future work will explore integrating product-side category information and applying these constraints to more complex, multi-layered knowledge graphs to further enhance personalization.

Summary updated

Abstract

Product search is the most common way for people to satisfy their shopping needs on e-commerce websites. Products are typically annotated with one of several broad categorical tags, such as "Clothing" or "Electronics", as well as finer-grained categories like "Refrigerator" or "TV", both under "Electronics". These tags are used to construct a hierarchy of query categories. Distributions of features such as price and brand popularity vary wildly across query categories. In addition, feature importance for the purpose of CTR/CVR predictions differs from one category to another. In this work, we leverage the Mixture of Expert (MoE) framework to learn a ranking model that specializes for each query category. In particular, our gate network relies solely on the category ids extracted from the user query. While classical MoE's pick expert towers spontaneously for each input example, we explore two techniques to establish more explicit and transparent connections between the experts and query categories. To help differentiate experts on their domain specialties, we introduce a form of adversarial regularization among the expert outputs, forcing them to disagree with one another. As a result, they tend to approach each prediction problem from different angles, rather than copying one another. This is validated by a much stronger clustering effect of the gate output vectors under different categories. In addition, soft gating constraints based on the categorical hierarchy are imposed to help similar products choose similar gate values. and make them more likely to share similar experts. This allows aggregation of training data among smaller sibling categories to overcome data scarcity.
BibTeX
@misc{xiao2020adversarial,
  author = {Zhuojian Xiao and Yunjiang Jiang and Guoyu Tang and Lin Liu and Sulong Xu and Yun Xiao and Weipeng Yan},
  title = {{Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint}},
  year = {2020},
  eprint = {2007.12349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.12349},
}
Figure