Intrinsic and Extrinsic Motivation in Intelligent Systems

Henry LiebermanIWSSL 2020 · OpenReview

TL;DR

This paper argues that current AI systems, much like human organizations, overemphasize extrinsic motivation (rewards, external goals) at the expense of intrinsic motivation (internal goals, curiosity). By exploring examples from reinforcement learning to automated mathematicians, the authors propose that incorporating intrinsic motivation can lead to more creative, explainable, and cooperative AI systems.

What Problem Does This Paper Address?

The paper addresses the fundamental question of how to imbue intelligent systems, both human and artificial, with motivation. It highlights a perceived imbalance in current AI development, where systems are primarily driven by extrinsic rewards or externally defined goals. This reliance on extrinsic motivation, the authors contend, limits AI's potential for creativity, self-explanation, and genuine problem-solving, mirroring issues seen in human organizations that solely rely on incentives. The core problem is how to design AI systems that can go beyond simply executing instructions and instead exhibit more autonomous, goal-directed, and potentially creative behavior.

Methods

The paper doesn't introduce a single novel algorithm but rather analyzes existing AI paradigms through the lens of intrinsic and extrinsic motivation. It uses a conceptual framework to categorize AI approaches:

  • Extrinsic Motivation: Systems driven by external reward signals or predefined goals, exemplified by standard Reinforcement Learning (RL).
  • Intrinsic Motivation: Systems that generate their own goals or are driven by internal states like curiosity or the desire to explore.
  • Self-Supervised Learning: A hybrid where external goals are interpreted and potentially aligned with internal goals.

The paper then examines several AI examples to illustrate these concepts:

  • Reinforcement Learning (RL): Presented as the quintessential example of extrinsic motivation, where the agent's sole objective is to maximize a reward signal.
  • Intrinsic Reinforcement Learning: An extension of RL that introduces an internal "meta-agent" and "internal environment," allowing for exploration driven by curiosity, akin to intrinsic motivation.
  • AM and Eurisko: Early AI systems designed as "automated mathematicians" that generated their own conjectures and theorems without external reward signals, showcasing intrinsic motivation for discovery.
  • Leela AI: A learning program for a grid-world environment that models child development, using prediction and experimentation to create new "schemas," demonstrating intrinsic motivation through learning and adaptation.
  • Lensing: A mixed-initiative technique that integrates intrinsic (data-driven learning) and extrinsic (user feedback) motivation, aiming to align AI's perspective with human users.

Datasets

Not discussed in the paper. The paper focuses on conceptual analysis and examples of AI programs rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Overemphasis on Extrinsic Motivation: Many current AI systems, particularly in reinforcement learning, are designed with a sole focus on external rewards. This can lead to systems that are good at achieving specific goals but lack explainability or creativity.
  • Intrinsic Motivation as a Source of Creativity and Invention: The paper argues that intrinsic motivation, driven by internal goals, curiosity, or the joy of discovery, is crucial for fostering creativity and innovation, both in humans and AI.
  • Explainability and Intrinsic Motivation: Systems with intrinsic motivation are better positioned to explain their "why" (their goals and plans) rather than just their "what" and "how," which is a significant challenge for many current AI systems.
  • Human-AI Cooperation: Understanding and integrating intrinsic motivation can lead to more natural and effective human-AI collaboration, where AI systems can better align with and understand user goals.
  • The Yin and Yang of Motivation: Neither intrinsic nor extrinsic motivation is inherently superior; they are complementary. The key is to find the right balance and integration for different applications.

Key Results

The paper does not present quantitative results from experiments. Its contribution is primarily conceptual and analytical, offering a framework for understanding and designing AI systems based on motivation. The "results" are the insights derived from analyzing existing AI paradigms and their implications for future AI development.

Limitations and Future Work

  • Conceptual Framework: The paper provides a conceptual framework rather than a prescriptive set of algorithms for implementing intrinsic motivation.
  • Limited Empirical Evidence: While examples are provided, the paper does not include empirical studies to quantitatively demonstrate the benefits of intrinsic motivation in AI systems.
  • Future Work: The paper implicitly suggests future work in developing AI architectures that can more effectively incorporate and leverage intrinsic motivation. This includes exploring how to design systems that can generate their own goals, exhibit curiosity, and provide more meaningful explanations. The Lensing example points towards future research in mixed-initiative systems that blend human and machine motivations.

Summary updated

Abstract

There are two ways that systems, human or machine, can get ”motivated” to take action in problem solving. One, they can be given goals by some external entity. In some instances, they might have no capability other than to work towards the goals provided by that entity. Two, they can have their own, internal goals, and work towards those goals. If given a goal by an outside entity, they can then try to figure out whether, and how, the external goal might align with their internal goals. In that case, the agent might be said to be acting in a ”self-supervised” manner. There are, of course, cases where both intrinsic and extrinsic motivation come into play. This paper will argue that many machine learning systems, as well as human organiza- tions, put too much emphasis on extrinsic motivation, and have not fully taken advantage of the potential of intrinsic motivation. Reinforcement learning systems, for example, have a ”reward signal” that is the sole extrinsic motivating factor. It is no wonder then, that even when such systems work well, they are incapable of explaining themselves, because they cannot express an explanation in terms of their own (or their users’) goals. In hu- man organizations, relying only on extrinsic motivation (= ”incentive”) leads to rigid or dictatorial organizations; engaging internal motivation (at some cost to ”organizational efficiency”) can lead to creativity and invention.
BibTeX
@inproceedings{lieberman2020intrinsic,
  author = {Henry Lieberman},
  title = {{Intrinsic and Extrinsic Motivation in Intelligent Systems}},
  booktitle = {IWSSL},
  year = {2020},
  url = {https://openreview.net/forum?id=FxmixBcB4T},
}
Figure