Zero-shot compositional reasoning in a reinforcement learning setting

Akshay Kumar Jagadish, Marcel Binz, Tankred Saanum, Jane X. Wang, Eric SchulzPsyArXiv 2023 · DOI

Abstract

People can easily evoke previously learned concepts, compose them, and apply the result to solve novel tasks on the first attempt. The aim of this paper is to improve our understanding of how people make such zero-shot compositional inferences in a reinforcement learning setting. To achieve this, we introduce an experimental paradigm where people learn two latent reward functions and need to compose them correctly to solve a novel task. We find that people have the capability to engage in zero-shot compositional reinforcement learning but deviate systematically from optimality. However, their mistakes are structured and can be explained by their performance in the sub-tasks leading up to the composition. Through extensive model-based analyses, we found that a meta-learned neural network model that accounts for limited computational resources best captures participants’ behaviour. Moreover, the amount of computational resources this model identified reliably quantifies how good individual participants are at zero-shot compositional reinforcement learning. Taken together, our work takes a considerable step towards studying compositional reasoning in agents – both natural and artificial – with limited computational resources.

Introduction

In artificial intelligence, reinforcement learning (RL) agents typically require thousands of trials to master a task. In contrast, humans exhibit a remarkable ability to solve novel problems "zero-shot" by composing simpler rules they already know. While compositionality—the ability to build complex entities from simple primitives—is well-studied in linguistics and vision, its role in RL, specifically regarding the composition of latent reward functions, remains less understood. This research investigates whether humans can perform zero-shot compositional reasoning in an RL setting, whether their behavior follows Bayesian logic or is constrained by cognitive limits, and which computational models best capture this "bounded" reasoning.

Methods

The researchers developed a novel experimental paradigm using a structured multi-armed bandit task involving six "slot machines" (arms). The rewards for these arms followed hidden latent functions based on their spatial position.

The experimental paradigm involves learning linear and periodic sub-tasks before facing a composite task.

Participants followed a specific curriculum:

  1. Sub-task 1: Learning a Linear reward function (e.g., rewards increasing spatially).
  2. Sub-task 2: Learning a Periodic reward function (e.g., rewards following a wave pattern).
  3. Composite Task (The Test): Solving a novel task zero-shot based on two composition rules: Additive (sum of functions) or Change-point (spatial split between functions).

Behavioral data was collected from 411 participants via Prolific. To model this behavior, traditional Bayesian models were compared against a meta-learning model called RR-RL² (Resource-Rational RL²). This recurrent neural network includes a penalty for "description length," representing a budget on the computational resources used to represent a strategy.

Results

The study found that humans are indeed capable of zero-shot compositional reasoning. Participants in the curriculum condition chose the optimal arm on the first trial of the final task approximately 38% of the time, significantly outperforming a control group (19%) that did not receive the sub-task training.

Behavioral results demonstrate that curriculum-trained participants achieve lower regret and higher optimal choice probability, though errors remain structured.

Despite this success, human performance was not perfectly optimal. Errors were "structured," meaning they were predictable based on individual performance in the preceding sub-tasks. The RR-RL² model provided the best fit for the data (posterior model frequency ~0.70), suggesting that human reasoning is "resource-rational"—balancing task success against limited computational resources. Furthermore, the model's identified "description length" correlated significantly with individual success, providing a metric for compositional reasoning ability.

Discussion

This work demonstrates that humans can compose latent reward functions to solve novel RL tasks zero-shot, but their performance is constrained by computational limits. While the 6-armed bandit task provides a controlled environment, future research should explore higher-dimensional state spaces and more naturalistic tasks. Additionally, while the RR-RL² model suggests a resource-rational mechanism, further neuroscientific studies are needed to identify the biological substrates of this "compositional engine" in the brain.

Summary updated

Abstract

People can easily evoke previously learned concepts, compose them, and apply the result to solve novel tasks on the first attempt. The aim of this paper is to improve our understanding of how people make such zero-shot compositional inferences in a reinforcement learning setting. To achieve this, we introduce an experimental paradigm where people learn two latent reward functions and need to compose them correctly to solve a novel task. We find that people have the capability to engage in zero-shot compositional reinforcement learning but deviate systematically from optimality. However, their mistakes are structured and can be explained by their performance in the sub-tasks leading up to the composition. Through extensive model-based analyses, we found that a meta-learned neural network model that accounts for limited computational resources best captures participants’ behaviour. Moreover, the amount of computational resources this model identified reliably quantifies how good individual participants are at zero-shot compositional reinforcement learning. Taken together, our work takes a considerable step towards studying compositional reasoning in agents – both natural and artificial – with limited computational resources.
BibTeX
@misc{jagadish2023zero,
  author = {Akshay Kumar Jagadish and Marcel Binz and Tankred Saanum and Jane X. Wang and Eric Schulz},
  title = {{Zero-shot compositional reasoning in a reinforcement learning setting}},
  year = {2023},
  doi = {10.31234/osf.io/ymve5},
  howpublished = {PsyArXiv},
}
Figure