Intrinsic Motivation for Encouraging Synergistic Behavior

Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, Abhinav Gupta

TL;DR

This paper introduces a novel intrinsic motivation for multi-agent reinforcement learning that specifically rewards synergy. By incentivizing agents to take joint actions that produce effects different from what they could achieve individually, the researchers significantly speed up learning in complex, sparse-reward tasks like bimanual robot manipulation and multi-agent soccer.

The Challenge of Sparse Rewards in Multi-Agent Systems

In reinforcement learning (RL), "sparse rewards" are notoriously difficult to overcome. If a robot only receives a positive signal for successfully completing a complex task (like opening a bottle) and zero for everything else, it might flail around indefinitely without stumbling upon the solution. This challenge is magnified in multi-agent settings because the action space grows exponentially with each added agent.

A common workaround is intrinsic motivation (often called "curiosity"), where agents are rewarded for reaching "surprising" or novel states. However, standard curiosity doesn't distinguish between a complex single-agent action and a true team effort. In tasks where agents must work together to succeed—like two robot arms lifting a heavy bar—standard curiosity might just lead them to explore individual, uncoordinated movements. This paper asks: How can we specifically bias exploration toward synergistic, cooperative behavior?

Methods: Rewarding Synergy

The authors define synergy as a situation where the joint effect of actions is significantly different from the composition of individual effects.

Synergy Reward Architecture

To operationalize this, they propose two ways to calculate a "Synergy Reward":

  1. Compositional Prediction Error ($r_1$): The reward is the difference between the actual next state in the environment and what a model predicts would happen if the agents acted sequentially (one after the other). If the real world looks very different from the "sequential" prediction, the action was likely synergistic.
  2. Prediction Disparity ($r_2$): Instead of relying on the actual environment state, this method compares two learned models: a joint model (trained to predict the outcome of both agents acting together) and the compositional model.

The "Secret Sauce": The second formulation ($r_2$) is analytically differentiable with respect to the agents' actions. This allows the researchers to use the gradient of the reward to directly tell the policy how to adjust its actions to maximize synergy, rather than relying purely on trial and error.

Evaluated Environments

The researchers evaluated their approach on six simulated robotic tasks using the MuJoCo physics engine, focusing on scenarios that demand tight coordination:

Bimanual Bar Pickup Task
  • Bimanual Manipulation (2 Sawyer arms): Tasks include bottle opening, ball pickup, corkscrew rotating, and bar pickup. These require precise coordination, such as one arm holding a bottle base while the other twists the cap.
  • Multi-agent Locomotion: Environments like "Ant push" (two ants pushing a large block) and "Soccer" (two agents must both touch the ball before scoring).
  • Scaling: They also tested 3-agent versions of the Ant push and Soccer environments to demonstrate that the method scales beyond simple pairs.

Key Insights

  • Synergy as a Deviation from Linearity: The core insight is that synergy is essentially the "interaction effect" in a system. If $Effect(A+B) \approx Effect(A) + Effect(B)$, there is no synergy. The proposed reward specifically targets the delta between these two outcomes.
  • Curiosity vs. Synergy: Standard curiosity rewards "surprise." But in a complex environment, many things are surprising but useless. By focusing on the gap between joint and individual effects, the agents ignore "boring" single-agent surprises and focus on "cooperative" ones.
  • Differentiability is King: While the first reward ($r_1$) is more intuitive, the second ($r_2$) is substantially more powerful. Because it provides a clear mathematical direction (gradient) for improvement, it leads to vastly superior sample efficiency.

Key Results

  • Sample Efficiency: In many tasks, the synergistic reward allowed agents to solve the problem with up to 5x fewer samples compared to using sparse rewards alone.
  • Superiority over Baselines: The method consistently outperformed standard "surprise-based" intrinsic motivation. In bottleneck tasks like the "Bar Pickup," standard curiosity often failed to find the solution entirely, while the synergy-biased agents succeeded reliably.
  • Zero-Shot Synergy: Even without any extrinsic task reward ($\lambda = 0$), the agents learned to act synergistically. For example, they learned to move a heavy block together simply because doing so generated high intrinsic synergy reward, even if they weren't explicitly told to move it to a goal.

Limitations and Future Work

  • Pre-training Requirement: The method currently requires pre-training the individual agent models ($f^A$ and $f^B$) on random interactions to establish a baseline of what "acting alone" looks like.
  • Single-Step Focus: The current formulation evaluates synergy within a single timestep. Extending this to "synergistic action sequences" (cooperation over extended time horizons) is a logical next step.
  • State Representation: For more complex, deformable, or asymmetric objects, the simple Euclidean state representations used in these experiments might struggle, likely necessitating more advanced computer vision or graph-based representations in the future.

Summary updated

Abstract

We propose a formulation of intrinsic motivation that is suitable as an exploration bias in synergistic multi-agent tasks, by encouraging agents to affect the world in ways that would not be achieved if they were acting individually.
BibTeX
@inproceedings{chitnis2020intrinsic,
  author = {Rohan Chitnis and Shubham Tulsiani and Saurabh Gupta and Abhinav Gupta},
  title = {{Intrinsic Motivation for Encouraging Synergistic Behavior}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=_MCF4v87DGOV},
}
Figure