Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication

Xu He, Bo An, Yanghua Li, Haikai Chen, Rundong Wang, Xinrun Wang, Runsheng Yu, Xin Li, Zhirong Wang

Abstract

Modern e-commerce platforms often deploy multiple recommendation modules managed by independent teams, leading to redundant content and sub-optimal user experiences. This paper proposes MASSA (Multi-Agent Soft Signal-Actor), a Multi-Agent Reinforcement Learning framework that achieves global coordination without inter-agent communication during execution. By utilizing a centralized signal network and entropy-regularized exploration, MASSA optimizes the overall page performance. Experiments on real-world Taobao data demonstrate significant improvements in Precision and nDCG over existing MARL baselines.

Introduction

Modern e-commerce web pages are typically composed of several recommendation modules, such as "Flash Sales," "Top Products," or "Daily Hot Deals." In most production environments, these modules are managed by different engineering teams who optimize their ranking strategies independently. This lack of coordination often results in competition for user attention and redundant recommendations. For instance, a user might see the same product repeated across multiple sections of the page, which wastes valuable screen real estate and degrades the overall user experience.

Examples of redundant recommendations across different modules on an e-commerce page.

While standard Multi-Agent Reinforcement Learning (MARL) can facilitate cooperation, existing methods usually require agents to communicate in real-time. In high-concurrency production systems, such inter-module communication is often infeasible due to strict latency requirements. This paper addresses the challenge of achieving global cooperation among isolated recommendation agents without any communication during execution.

The MASSA Framework

The authors introduce the MASSA (Multi-Agent Soft Signal-Actor) algorithm, which treats the multi-module recommendation task as a cooperative multi-agent problem. The framework relies on several key innovations:

The Signal Network

Inspired by the game-theoretic concept of "correlated equilibrium," MASSA utilizes a shared signal network. This network acts like a centralized "traffic light," observing the global user state (including static features and click history) and broadcasting a unique signal vector to each module's agent.

Communication-Free Execution

During deployment, each module (agent) operates independently, using only its local observations and the received signal to perform ranking. This design ensures that no peer-to-peer messaging is required between modules, making it compatible with existing infrastructure constraints.

Entropy-Regularized Exploration

To prevent the system from converging to sub-optimal local minima, the authors incorporate entropy regularization into the signal network's objective. Instead of individual agents exploring randomly, the signal network is incentivized to generate diverse signals, coordinating exploration from a global perspective.

Centralized Training, Decentralized Execution (CTDE)

MASSA follows the CTDE paradigm. During training, a centralized "critic" network observes the actions of all agents to estimate global rewards accurately. During execution, only the "actor" networks are used, guided by the signal network.

Experimental Evaluation

The proposed approach was evaluated using a large-scale dataset from Taobao, featuring approximately 1.5 million training records and 200,000 testing records.

Key Results

MASSA significantly outperformed traditional MARL baselines in offline tests:

  • Precision: MASSA achieved 0.615, compared to 0.452 for QMIX and 0.274 for MADDPG.
  • nDCG: MASSA reached 0.516, demonstrating superior ranking quality.

Ablation studies highlighted the importance of the entropy term; removing it caused Precision to drop to 0.495, confirming that coordinated exploration is vital for performance. Furthermore, in an online simulator with a candidate set of 2,000 items, MASSA consistently maintained its lead over other methods.

Discussion and Insights

A primary insight from this work is that global coordination does not strictly require direct communication if agents can follow a centralized, intelligent signal. The results also revealed an interesting trade-off: to maximize the overall page performance, the system occasionally sacrifices the local performance of secondary modules to significantly boost primary, high-visibility modules. Independent agents, focused solely on local metrics, would be unable to make such strategic sacrifices.

While the reliance on a simulator for "online" testing and the assumption of linear browsing behavior are limitations, MASSA provides a robust and scalable solution for multi-module coordination in real-world e-commerce environments.

Summary updated

BibTeX
@inproceedings{he2020learning,
  author = {Xu He and Bo An and Yanghua Li and Haikai Chen and Rundong Wang and Xinrun Wang and Runsheng Yu and Xin Li and Zhirong Wang},
  title = {{Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication}},
  booktitle = {RecSys},
  year = {2020},
  url = {https://openreview.net/forum?id=J6tnljXenIf},
}
Figure