Comparing Human Behavior to an Optimal Policy for Innovation

Bonan Zhao, Natalia Vélez, Thomas L. Griffiths

Abstract

While modern artificial intelligence excels at solving well-defined problems with abundant data, human intelligence is characterized by the ability to set new goals and innovate through the combination of existing knowledge. This paper investigates the mechanisms of human innovation by introducing a "Discovery Game" modeled as a Markov Decision Process (MDP). By comparing human behavior against a mathematically optimal policy, the study explores how success rates and incentives influence the decision to exploit known rewards versus pursuing novel, more rewarding tools. The findings provide insights into the pursuit of innovation and offer a framework for designing more open-ended, human-like AI systems.

Introduction

Modern AI models have achieved remarkable success in environments with clear targets and fixed parameters. However, human intelligence operates on a fundamentally different level: we do not merely solve isolated problems; we actively seek new challenges and invent novel concepts. This drive facilitates cumulative culture—the process by which knowledge, skills, and tools are accumulated and refined across generations.

A significant gap exists in how we model this behavior. Traditional "explore-exploit" frameworks, such as the multi-armed bandit problem, typically focus on sampling from a set of unknown but pre-existing options. In contrast, true innovation is often combinatorial, involving the creation of entirely new options from existing ones. This paper addresses how humans navigate the trade-off between harvesting rewards from current tools and investing effort into combining them to discover superior, undiscovered alternatives.

The Discovery Game: Modeling Innovation as an MDP

To analyze the strategic nature of innovation, the authors developed the Discovery Game. This environment mimics the "tech tree" progression found in crafting games, where players must decide between immediate gain and long-term advancement.

The Mechanics

In the game, players possess a set of items. At each turn, they face a choice:

  1. Exploit: Collect a known reward from an existing item.
  2. Innovate: Attempt to combine existing items to discover a novel, more valuable item.

Because discovery is not guaranteed, players must weigh the probability of a successful combination against the potential reward of the new tool.

The Mathematical Framework

The researchers formalized this environment as a finite-horizon Markov Decision Process (MDP). This allowed them to derive an analytical solution for the optimal policy based on two primary variables:

  • Success Rate ($p$): The probability that a discovery attempt will succeed.
  • Incentive ($r$): The relative increase in reward offered by a newly discovered item compared to existing ones.

One surprising theoretical insight from this model is Opportunity Independence: the optimal level of innovation-seeking behavior is mathematically independent of the total number of opportunities available in the environment.

Experimental Design

The study utilized an online behavioral experiment with $n = 210$ participants to compare human decision-making against the MDP's optimal policy. The experiment manipulated the two key variables—success rate and incentive—across high and low conditions. This design allowed the researchers to observe how changes in environmental parameters shifted human strategies toward or away from the mathematical optimum.

Key Insights and Results

Alignment with Theory

The majority of human decisions aligned with the theoretical predictions of the MDP. Participants were significantly more likely to seek innovation when success rates were high or when the potential incentives were substantial. This suggests that human intuition for innovation is grounded in a rational assessment of risks and rewards.

Subjective Weighting

Despite the general alignment with the optimal policy, the study found that humans do not process the mathematical variables perfectly. Participants appeared to assign unequal subjective weights to the probability of success versus the magnitude of the incentive. This indicates that while humans are strategically rational, their decision-making is influenced by cognitive biases or subjective valuations of risk.

Strategic Diversity

The authors observed a rich variety of strategies across different conditions. These nuances captured how individuals decide exactly when to push into the unknown and when to settle for exploiting their current toolkit, reflecting the complexity of human innovative behavior.

Discussion and Future Work

Limitations

The current study utilizes constant parameters where success rates and incentives remain static. This simplifies the often unpredictable and dynamic nature of real-world innovation. Additionally, as a brief report, this paper focuses on qualitative trends rather than exhaustive quantitative breakdowns of the behavioral data.

Future Directions

Future research aims to introduce Dynamic Environments, where success rates and incentives fluctuate over time to better mimic reality. Another promising avenue is the integration of Semantic Knowledge. Inspired by open-ended LLM agents, researchers propose enriching the game with domain-specific semantics. This would allow for the study of how prior knowledge helps humans guess which combinations are likely to be fruitful.

Finally, the authors intend to explore the Division of Labor and Garden-Pathing. In multi-domain environments, the increasing returns of specialization might lead to "garden-pathing," where expertise in one area blinds an individual to alternative technological paths. Understanding these dynamics could prove vital in designing parallel, distributed artificial learning systems that emulate human-like cultural evolution.

Summary updated

Abstract

Human learning does not stop at solving a single problem. Instead, we seek new challenges, define new goals, and come up with new ideas. Unlike the classic explore-exploit trade-off between known and unknown options, making new tools or generating new ideas is not about collecting data from existing unknown options, but rather about create new options out of what is currently available. We introduce a discovery game designed to study how rational agents make decisions about pursuing innovations, where discovering new ideas is a process of combining existing ideas in an open-ended compositional space. We derive optimal policies of this decision problem formalized as a Markov decision process, and compare people's behaviors to the model predictions in an online behavioral experiment. We found evidence that people both innovate rationally, guided by potential returns in this discovery game, and under- and over-explore systematically in different settings.
BibTeX
@inproceedings{zhao2024comparing,
  author = {Bonan Zhao and Natalia Vélez and Thomas L. Griffiths},
  title = {{Comparing Human Behavior to an Optimal Policy for Innovation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31291},
}
Figure