Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning

Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, Joel Z. Leibo

TL;DR

DeepMind researchers have introduced "Fruit Market," a multi-agent reinforcement learning (MARL) environment where agents learn to produce, trade, and consume resources based on individual preferences. The study demonstrates that classic microeconomic phenomena—such as supply and demand curves and spatial arbitrage—emerge naturally from agents simply trying to maximize their own rewards.

What Problem Does This Paper Address?

Reinforcement learning (RL) is exceptionally good at refining known behaviors but often struggles to discover entirely novel, coordinated social behaviors in "large worlds" where the target behavior is far from the starting point. While humans innovate through complex social and economic structures, most MARL research has focused on pure competition (zero-sum games) or pure cooperation.

The authors argue that microeconomics provides a "science of incentives" that can drive agents to explore and innovate. The challenge is determining whether standard RL agents, starting with zero knowledge of trade or social norms, can actually discover economic concepts like bartering, price discovery, and middleman behavior (arbitrage) simply through environmental pressure.

Methods

The researchers developed the Fruit Market environment, a 2D grid world where agents act as either "Apple Farmers" or "Banana Farmers." * Specialization: Apple Farmers produce apples efficiently but get higher rewards for consuming bananas (and vice versa). * Trade Mechanism: Instead of using a global exchange, trade is "embodied." Agents must be physically near each other to trade. They use a set of "Offer" actions (e.g., "I'll give 2 apples for 1 banana"). The environment facilitates an atomic swap if two nearby agents make compatible offers. * Agent Architecture: The study uses V-MPO (Value-based Maximum a Posteriori Policy Optimization), a state-of-the-art actor-critic algorithm. Crucially, agents are "independent"—they do not share parameters or have access to each other's internal states. * Comparative Statics: To test economic theories, the researchers performed "parameter sweeps," exogenously changing tree spawn rates (supply) or consumption rewards (demand) to see how the agent population's equilibrium behavior shifted.

Datasets

The paper introduces the Fruit Market environment, which the authors are releasing as part of DeepMind’s Melting Pot suite (a benchmark for multi-agent intelligence). Unlike static datasets, this is a procedural environment where map layouts and resource densities are generated at the start of each episode to ensure agents learn generalized strategies rather than memorizing specific paths.

Key Insights and Ideas

  • The "Hunger Penalty" as a Catalyst: One of the most surprising findings was that trade failed to emerge without a "hunger penalty." If agents don't face a cost for having an empty stomach, they eat whatever they produce immediately. The hunger penalty forces them to carry a "buffer" of inventory, which inadvertently provides the "capital" needed to discover that trading fruit is more lucrative than eating it.
  • Textbook Economics from Scratch: By plotting the quantity of fruit traded against the emergent "price" (the exchange ratio), the authors generated supply and demand curves. These curves sloped exactly as predicted by introductory microeconomics, despite the agents having no programmed concept of "price."
  • Emergent Merchants: In maps with distinct regions (e.g., an apple-rich west and a banana-rich east), local prices diverged. Some agents discovered a "merchant niche"—instead of farming, they spent their time transporting goods between regions to exploit price differences, effectively performing spatial arbitrage.
  • The Importance of Robust Architectures: The researchers compared V-MPO to the older A2C architecture. A2C agents almost entirely failed to learn trade, often collapsing into "spinning in place" behaviors. This suggests that the emergence of complex social coordination is highly sensitive to the underlying learning algorithm.

Key Results

  • Price Convergence: In balanced environments, agents consistently converged to a "fair" 1:1 barter ratio.
  • Response to Scarcity: When the spawn rate of apple trees was doubled, the emergent price of apples dropped significantly, following the law of supply.
  • Arbitrage Profits: In the "Walls" map configuration (where trade is possible across borders but movement is restricted), "merchant" agents who learned to buy low and sell high earned 22% to 32% more reward than agents who stuck to traditional farming.
  • Trade Radius: Agents were able to learn to trade even when the "trade radius" was reduced to 1 (requiring them to be perfectly adjacent), though a larger radius accelerated the learning process.

Limitations and Future Work

The authors acknowledge that their trade mechanism still relies on some "domain knowledge" by having the environment automatically resolve compatible offers. In the real world, trade requires complex norms like private property and trust to prevent theft.

Future directions include: 1. Removing Environmental Assistance: Forcing agents to learn trade using only "Drop" and "Give" actions, which would require them to solve the "problem of trust" and avoid being robbed. 2. Emergent Currency: Moving beyond two-good barter to see if agents eventually nominate one resource as a "numeraire" (money) to facilitate trade. 3. More Complex Goods: Introducing durable goods that don't spoil, or tools (like a stone axe) that require production steps but make future harvesting more efficient.

Summary updated

Abstract

Advances in artificial intelligence often stem from the development of new environments that abstract real-world situations into a form where research can be done conveniently. This paper contributes such an environment based on ideas inspired by elementary Microeconomics. Agents learn to produce resources in a spatially complex world, trade them with one another, and consume those that they prefer. We show that the emergent production, consumption, and pricing behaviors respond to environmental conditions in the directions predicted by supply and demand shifts in Microeconomics. We also demonstrate settings where the agents' emergent prices for goods vary over space, reflecting the local abundance of goods. After the price disparities emerge, some agents then discover a niche of transporting goods between regions with different prevailing prices -- a profitable strategy because they can buy goods where they are cheap and sell them where they are expensive. Finally, in a series of ablation experiments, we investigate how choices in the environmental rewards, bartering actions, agent architecture, and ability to consume tradable goods can either aid or inhibit the emergence of this economic behavior. This work is part of the environment development branch of a research program that aims to build human-like artificial general intelligence through multi-agent interactions in simulated societies. By exploring which environment features are needed for the basic phenomena of elementary microeconomics to emerge automatically from learning, we arrive at an environment that differs from those studied in prior multi-agent reinforcement learning work along several dimensions. For example, the model incorporates heterogeneous tastes and physical abilities, and agents negotiate with one another as a grounded form of communication.
BibTeX
@misc{johanson2022emergent,
  author = {Michael Bradley Johanson and Edward Hughes and Finbarr Timbers and Joel Z. Leibo},
  title = {{Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning}},
  year = {2022},
  eprint = {2205.06760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.06760},
}
Figure