Intrinsic Motivation for Encouraging Synergistic Behavior

Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, Abhinav Gupta

Abstract

We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken. We validate our approach in robotic bimanual manipulation and multi-agent locomotion tasks with sparse rewards; we find that our approach yields more efficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior.

Introduction

Reinforcement learning (RL) has achieved remarkable success in single-agent domains, but multi-agent environments with sparse rewards remain a significant challenge. In these settings, agents only receive feedback upon total success (e.g., a "1" for completing a task and "0" otherwise). As the number of agents increases, the joint action space grows exponentially, making it statistically improbable for agents to discover cooperative solutions through random exploration alone.

While "intrinsic motivation"—often referred to as curiosity—has been used to encourage exploration by rewarding agents for encountering "surprising" or unpredictable states, it often fails in cooperative contexts. Standard curiosity does not distinguish between a single agent performing a novel but irrelevant individual action and multiple agents performing a coordinated action necessary for the task. This paper introduces a novel intrinsic reward specifically designed to incentivize synergy.

Defining Synergy as a Prediction Gap

The core hypothesis of this work is that synergy can be quantified as the discrepancy between what agents can achieve together versus what they can achieve individually. To operationalize this, the authors propose a framework based on forward dynamics models.

1. Single-Agent Baselines

The system first utilizes pre-trained forward dynamics models ($f^A$ and $f^B$) that predict the next state of the environment based on the actions of a single agent. These models establish a baseline for what is possible when agents act in isolation.

2. Compositional Prediction

To estimate the outcome of non-synergistic behavior, the authors define a compositional prediction ($f^{composed}$). This simulates a scenario where agents act sequentially rather than simultaneously—for example, Agent A acts, and then Agent B acts upon the resulting state. This represents the "sum of the parts."

3. The Synergy Reward Formulations

The authors propose two distinct methods to calculate the intrinsic synergy reward:

  • Compositional Prediction Error ($r_1$): This reward is granted when the actual state resulting from a joint action differs significantly from the state predicted by the compositional model. It rewards the agents for finding "holes" in the individual models' ability to explain the world.
  • Prediction Disparity ($r_2$): This more advanced version trains a joint dynamics model ($f^{joint}$) alongside the policy. The reward is defined as the difference between the joint model's prediction and the compositional prediction.

One of the most significant advantages of the $r_2$ formulation is that it is analytically differentiable with respect to the actions taken. This allows the policy to receive a rich gradient signal, essentially telling the agents how to adjust their actions to maximize synergy, rather than relying on trial-and-error exploration.

Experimental Evaluation

The approach was evaluated using the MuJoCo physics engine across several complex tasks requiring high degrees of coordination:

  • Bimanual Manipulation: Tasks included opening a bottle (requiring one hand to stabilize and the other to twist), picking up a ball, rotating a corkscrew, and lifting a bar using two Sawyer robot arms.
  • Multi-agent Locomotion: The "Ant Push" task required two agents to coordinate to move a heavy block, while a "Soccer" task required two agents to both touch a ball before a goal could be scored.
  • Scalability: The researchers successfully extended the Ant Push and Soccer environments to three agents, demonstrating the method's potential for larger teams.

Key Insights and Results

Efficiency Gains

The synergistic reward mechanism significantly improved sample efficiency. In several benchmarks, agents reached success thresholds with up to 5x fewer samples than those trained with sparse rewards alone.

Synergy vs. Curiosity

The proposed method consistently outperformed standard surprise-based curiosity. While standard curiosity often led agents to explore irrelevant individual behaviors that were simply hard to predict, the synergy bias forced agents to focus on interactions that required joint effort.

The Power of Differentiability

The $r_2$ model, which utilizes the policy gradient to guide exploration, proved to be the most robust. By turning exploration into a directed search rather than a random walk, the agents were able to solve complex manipulation tasks that were previously intractable for standard RL algorithms.

Limitations and Future Directions

Despite its success, the framework faces challenges in scaling to massive multi-agent systems due to the "ordering problem" in compositional models; as the number of agents grows, the number of possible sequential orderings increases. Additionally, the current model focuses on single-step synergy. Future research may explore rewarding synergistic sequences of actions, where one agent's behavior sets the stage for another's success over a longer time horizon. Finally, transitioning from low-dimensional state representations to raw pixel data remains an open challenge for this architecture.

Summary updated

Abstract

We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken. We validate our approach in robotic bimanual manipulation and multi-agent locomotion tasks with sparse rewards; we find that our approach yields more efficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior. Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.
BibTeX
@inproceedings{chitnis2020intrinsic,
  author = {Rohan Chitnis and Shubham Tulsiani and Saurabh Gupta and Abhinav Gupta},
  title = {{Intrinsic Motivation for Encouraging Synergistic Behavior}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=_MCF4v87DGOV},
}
Figure