Meta-control of social learning strategies

Anil Yaman, Nicolas Bredèche, Onur Çaylak, Joel Z. Leibo, Sang Wan Lee

Introduction

Learning is an inherently expensive endeavor. For biological organisms and artificial agents alike, "individual learning" through trial and error demands significant time, energy, and the constant risk of failure. To mitigate these costs, many species employ "social learning"—the practice of copying the behaviors of others to "free-ride" on their exploration. However, social learning introduces a strategic dilemma: in a complex, changing world, who should you copy?

This paper introduces the concept of "meta-social learning," a framework where agents dynamically switch between individual exploration and various social learning strategies based on environmental uncertainty. The researchers demonstrate that while copying successful individuals is effective in stable settings, a "conformist" strategy is far more robust in uncertain environments. By employing a meta-control mechanism to arbitrate between these strategies, agents can achieve near-optimal learning with minimal exploration costs.

The Social Learning Dilemma

The study identifies two primary social learning strategies:

  1. Success-based strategy: Copy the individual currently receiving the highest rewards.
  2. Conformist strategy: Copy the action performed by the majority of the population.
Performance of social learning strategies under varying uncertainty. Success-based learning excels in low uncertainty but falters when noise is high, whereas conformist learning provides a robust filter.

As shown in the dynamics of reward acquisition, the effectiveness of these strategies is highly dependent on environmental uncertainty. In "benign" or low-uncertainty environments, success-based learning is highly efficient. However, its performance collapses in noisy environments because agents frequently end up copying "lucky" individuals who made sub-optimal choices. Conversely, the conformist strategy acts as a robust filter; by copying the majority, agents average out the noise of individual errors, though they are slower to adapt to sudden environmental shifts.

Quantifying Uncertainty and Meta-Control

To address this, the authors developed a novel metric: Optimum Distribution Prediction Uncertainty (ODPU). ODPU quantifies environmental uncertainty as the probability of mistaking a sub-optimal choice for an optimal one due to overlapping reward distributions.

Using agent-based simulations in non-stationary multi-armed bandit tasks, the researchers tested several meta-control models designed to switch strategies based on ODPU and environmental changes:

  • SL-GA: A rule-based controller optimized via Genetic Algorithms.
  • SL-NE: A neural network-based controller trained through neuroevolution.

These models allow agents to use brief periods of individual learning to "sense" the environment's current state before committing to the most appropriate social strategy.

Key Results and Evolutionary Dominance

The research reveals that environmental uncertainty is the primary predictor of strategy performance. Meta-control strategies consistently outperformed pure strategies across diverse conditions, including stable, volatile, and gradually changing environments.

Comparative analysis of strategy performance. Panel (a) shows average rewards at the end of the process vs. after change; panel (b) highlights the performance gap between conformist and success-based strategies in high uncertainty; panel (c) ranks the strategies.

Statistical analysis confirms that the conformist strategy significantly outperforms success-based copying when uncertainty is high. Furthermore, the meta-social learning models (specifically SL-GA and SL-NE) achieved the highest cumulative rewards with the lowest exploration costs—individual learning alone was found to be roughly three times more costly than the best meta-social approach. In evolutionary simulations where strategies competed, these meta-controllers dominated the population, showing higher survival rates and life expectancy.

Conclusion

The study suggests an optimal "lifecycle" for learning: agents should use individual learning immediately following an environmental change to gather fresh data, then transition to success-based social learning if uncertainty is low, or conformist learning if uncertainty is high. This meta-control approach provides a blueprint for more efficient distributed learning in both biological systems and decentralized AI applications, such as swarm robotics.

Summary updated

Abstract

Social learning, copying other's behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others' learning as an external knowledge base.
BibTeX
@article{yaman2022meta,
  author = {Anil Yaman and Nicolas Bredèche and Onur Çaylak and Joel Z. Leibo and Sang Wan Lee},
  title = {{Meta-control of social learning strategies}},
  journal = {PLOS Computational Biology},
  year = {2022},
  doi = {10.1371/journal.pcbi.1009882},
}
Figure