Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning

Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, Joel Z. Leibo

TL;DR

This paper introduces "Fruit Market," a novel multi-agent reinforcement learning environment designed to study emergent microeconomic behaviors like production, consumption, and bartering. The authors demonstrate that agents, starting with no domain-specific knowledge, can learn to trade and exhibit behaviors consistent with supply and demand principles, even developing specialized "merchant" roles.

What Problem Does This Paper Address?

The paper tackles the challenge of creating artificial agents that can exhibit complex, emergent social behaviors, moving beyond simple task completion. Specifically, it addresses how fundamental microeconomic concepts like production, consumption, and trade can arise spontaneously from agents learning to maximize their rewards in a simulated environment. The authors highlight the difficulty in reinforcement learning of discovering novel behaviors that aren't frequently present in initial exploration, and propose that rich, socially interactive environments can provide the necessary extrinsic motivation for agents to innovate and learn complex economic strategies. They aim to understand what environmental features are crucial for such behaviors to emerge, moving towards building more human-like artificial general intelligence.

Methods

The core of the paper is the "Fruit Market" environment, a 2D partially observable world where agents (controlled by deep reinforcement learning algorithms, specifically V-MPO) can move, produce, consume, and barter two types of fruit: apples and bananas. Agents are assigned roles (Apple Farmers or Banana Farmers) with differing production efficiencies and consumption preferences, incentivizing trade. The paper uses a multi-agent reinforcement learning (MARL) approach, specifically the independent learning paradigm where each agent learns its own policy. Key to the environment is an "atomic" exchange mechanism that facilitates trading when compatible offers are made within a certain radius. The authors also explore various environmental parameters (e.g., tree spawn rates, penalties, marketplace presence) and agent actions to understand their impact on emergent behaviors.

Datasets

The paper introduces a novel benchmark environment called "Fruit Market." This environment is used for training and evaluating the agents. The authors also mention that Fruit Market will be released as part of the open-source "Melting Pot" package.

Key Insights and Ideas

  • Hunger as a Trade Catalyst: A simple hunger penalty, which incentivizes agents to maintain a reserve of fruit, is surprisingly crucial for the emergence of trade. Without it, agents tend to consume fruit immediately, leaving nothing to trade.
  • Environmental Design Shapes Behavior: Subtle changes in the environment, such as the abundance of resources or the presence of a marketplace, can drastically alter agent behavior, leading to emergent prices, specialized roles (like merchants), and even arbitrage.
  • Trade is Not Trivial: Even with clear incentives, agents struggle to learn basic trading mechanics like "drop and give" actions, highlighting the complexity of coordination and the need for more structured mechanisms like offers and exchanges.
  • Emergent Microeconomics: Agents can learn to produce, consume, and trade in ways that align with basic microeconomic principles like supply and demand, demonstrating the potential of RL for discovering complex economic strategies.
  • Merchant Behavior Emerges from Scarcity and Price Differences: When resources become scarce in certain regions, agents can learn to specialize in transporting goods between regions to exploit price differences, a behavior termed "merchant behavior."

Key Results

  • Agents consistently learn to trade, leading to significantly higher collective rewards compared to scenarios without trade.
  • Varying the spawn rates of apples and bananas influences agent behavior, shifting production, consumption, and prices in directions consistent with supply and demand curves. For instance, increasing apple abundance leads to lower apple prices.
  • The introduction of a marketplace can strongly influence emergent prices, and its location affects how much agents rely on it versus trading amongst themselves.
  • Merchant behavior, involving buying and selling goods across regions to exploit price differences, emerges in specific settings (e.g., "Walls" map with a neutral penalty of ×0.25), with these agents earning substantially more reward.
  • The V-MPO agent architecture significantly outperforms A2C in learning to trade and achieve positive rewards in this environment.

Limitations and Future Work

  • Domain Knowledge Injection: The authors acknowledge that the current environment injects some domain knowledge (e.g., atomic exchanges, compatible offer resolution) to make learning feasible for current RL agents. Future work aims to remove these compromises to allow agents to learn trade from scratch.
  • Simpler Trade Mechanics: The "Drop and Give" actions, while simpler, were not effectively learned by current agents, suggesting a need for more advanced agents or mechanisms to explore these basic interactions.
  • Dynamic Offer Actions: While promising for scalability, dynamic offer actions led to less efficient and rational trading behavior compared to the default offer mechanism. Future work may explore more effective ways to implement these.
  • Non-Stationary Environments: The current experiments use comparative statics, training separate populations for each condition. Future work will explore how agents adapt to gradual or sudden environmental shifts.
  • More Resources and Complex Goods: Extending the environment to include more diverse resources, durable goods, processed goods, or tools could provide further challenges and insights into emergent economic behavior.
  • Emergence of Currency: A significant future direction is to see if agents can learn to establish a common currency for exchanges.

Summary updated

Abstract

Advances in artificial intelligence often stem from the development of new environments that abstract real-world situations into a form where research can be done conveniently. This paper contributes such an environment based on ideas inspired by elementary Microeconomics. Agents learn to produce resources in a spatially complex world, trade them with one another, and consume those that they prefer. We show that the emergent production, consumption, and pricing behaviors respond to environmental conditions in the directions predicted by supply and demand shifts in Microeconomics. We also demonstrate settings where the agents' emergent prices for goods vary over space, reflecting the local abundance of goods. After the price disparities emerge, some agents then discover a niche of transporting goods between regions with different prevailing prices -- a profitable strategy because they can buy goods where they are cheap and sell them where they are expensive. Finally, in a series of ablation experiments, we investigate how choices in the environmental rewards, bartering actions, agent architecture, and ability to consume tradable goods can either aid or inhibit the emergence of this economic behavior. This work is part of the environment development branch of a research program that aims to build human-like artificial general intelligence through multi-agent interactions in simulated societies. By exploring which environment features are needed for the basic phenomena of elementary microeconomics to emerge automatically from learning, we arrive at an environment that differs from those studied in prior multi-agent reinforcement learning work along several dimensions. For example, the model incorporates heterogeneous tastes and physical abilities, and agents negotiate with one another as a grounded form of communication.
BibTeX
@misc{johanson2022emergent,
  author = {Michael Bradley Johanson and Edward Hughes and Finbarr Timbers and Joel Z. Leibo},
  title = {{Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning}},
  year = {2022},
  eprint = {2205.06760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.06760},
}
Figure