Should I tear down this wall? Optimizing social metrics by evaluating novel actions.

János Kramár, Neil C. Rabinowitz, Tom Eccles, Andrea TacchettiarXiv 2020 · OpenReview

TL;DR

The paper introduces OBOE (Optimize By Observational Extrapolation), a method for selecting physical interventions in multi-agent systems using only observational data. By training models to predict agent outcomes from baseline behavior, researchers can evaluate "what-if" scenarios—like adding walls or moving resources—without needing expensive simulators or risky real-world trials.

What Problem Does This Paper Address?

In complex multi-agent systems, "governors" often want to intervene to improve social outcomes, such as increasing collective wealth or reducing inequality. However, predicting the effect of a novel intervention (e.g., "how will traffic flow change if we add a roundabout here?") is notoriously difficult.

Standard approaches rely on building bespoke, high-fidelity simulators for every possible change, which is prohibitively expensive and often inaccurate. The alternative—trial and error in the real world—is often impossible or unethical. This paper asks: Can we learn to predict the effects of interventions we have never seen, using only data from how the system behaves when we don't intervene?

Methods

The authors propose a two-stage framework called OBOE:

  1. Observational Learning: They collect a large dataset of agents interacting in a baseline environment (no interventions). They use this to train a Forward Return Predictor—a neural network that takes the current state of the environment and predicts the future rewards for every agent.
  2. Counterfactual Evaluation: To evaluate a potential intervention (like building a wall), they manually modify the environment's state representation to reflect that change. They then feed this "counterfactual" state into their pre-trained predictor. By aggregating the predicted individual rewards, they can estimate group-level social metrics (like the Gini index for inequality) for that specific intervention.
The OBOE method workflow: training on observational data to construct a Q-function for interventions.

The researchers tested three neural architectures: a standard Multi-Layer Perceptron (MLP), a Convolutional Neural Network (CNN), and a Relational Forward Model (RFM), which is specifically designed to capture interactions between entities in a scene.

Datasets

The method was evaluated on two socially complex "gridworld" environments that function as Markov social dilemmas:

  • Cleanup: A public goods game where five players must balance collecting apples (reward) with cleaning waste from an aquifer. If no one cleans, apples stop growing.
  • Harvest: A common-pool resource game where apples respawn faster if there are more apples nearby. Players must harvest sustainably or risk depleting the entire stock.

The interventions tested included moving players, moving waste, and—most notably—adding or removing physical walls to steer agent behavior.

Fig. 2. A “move waste” inter- vention for the Cleanup game.

Key Insights and Ideas

  • Causal Inference via Markovianity: The core insight is that if an environment is Markovian (the future depends only on the current state), then the conditional expectation of future outcomes given a state is equivalent to the causal effect of being in that state.
  • Extrapolation through Inductive Bias: The models were never trained on states containing "new" walls. However, because the models (especially the RFM) learn the underlying "physics" and social dynamics of the game, they can generalize their predictions to these novel, out-of-distribution states.
  • Model vs. Simulator: Surprisingly, a model trained on a large, diverse observational dataset can sometimes be more effective at selecting interventions than a simulator that only runs a few stochastic trials, as the latter can suffer from high variance and selection bias.

Key Results

The authors compared OBOE against a "CV" (Cross-Validation) agent that had access to a perfect simulator.

  • High Effectiveness: OBOE agents were, on average, 56% as effective as the ideal simulator-based agent at improving social metrics.
  • Architecture Matters: The Relational Forward Model (RFM) was the most consistent performer, always outperforming a random baseline. In some specific tasks, OBOE even matched the performance of the simulator-based agent.
  • Zero-Shot Generalization: The models successfully identified beneficial interventions (like adding a wall to protect a resource) despite never seeing a wall in that location during training.
Visualizing "before and after" states for interventions in Harvest and Cleanup.

Limitations and Future Work

  • Static Policies: The current model assumes that agents will not change their fundamental strategies in response to the intervention. In reality, agents might adapt to a new wall by learning new paths, which OBOE does not yet account for.
  • State Representation: OBOE requires the environment state to be represented in a structured way that allows for manual "editing" (e.g., changing a 'floor' tile to a 'wall' tile).
  • Single-Step Interventions: The experiments focused on interventions at a single fixed timestep. Future work could explore dynamic, multi-step interventions where a central agent modifies the environment in real-time.

Summary updated

Abstract

One of the fundamental challenges of governance is deciding when and how to intervene in multi-agent systems in order to impact group-wide metrics of success. This is particularly challenging when proposed interventions are novel and expensive. For example, one may wish to modify a building's layout to improve the efficiency of its escape route. Evaluating such interventions would generally require access to an elaborate simulator, which must be constructed ad-hoc for each environment, and can be prohibitively costly or inaccurate. Here we examine a simple alternative: Optimize By Observational Extrapolation (OBOE). The idea is to use observed behavioural trajectories, without any interventions, to learn predictive models mapping environment states to individual agent outcomes, and then use these to evaluate and select changes. We evaluate OBOE in socially complex gridworld environments and consider novel physical interventions that our models were not trained on. We show that neural network models trained to predict agent returns on baseline environments are effective at selecting among the interventions. Thus, OBOE can provide guidance for challenging questions like: "which wall should I tear down in order to minimize the Gini index of this group?"
BibTeX
@misc{kramar2020should,
  author = {János Kramár and Neil C. Rabinowitz and Tom Eccles and Andrea Tacchetti},
  title = {{Should I tear down this wall? Optimizing social metrics by evaluating novel actions.}},
  year = {2020},
  eprint = {2004.07625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.07625},
}
Figure