First Experiments with PowerPlay

Rupesh Kumar Srivastava, Bas R. Steunebrink, Jürgen SchmidhuberarXiv 2012 · arXiv

Introduction

Most contemporary artificial intelligence systems are passive, designed to solve a fixed set of problems predefined by human supervisors. This paper introduces POWERPLAY, a framework for open-ended development that shifts the paradigm from passive learning to active exploration. By autonomously inventing its own increasingly complex yet solvable problems, a POWERPLAY agent mimics the curious behavior of a developing child, building a modular repertoire of abilities without external guidance.

The POWERPLAY Framework

The core of POWERPLAY is a continuous loop designed to expand the agent's capabilities while preventing "catastrophic forgetting" (the loss of previously learned skills). The process follows four key steps:

  1. Task Invention: The system searches for a new task description and a corresponding modification to its current solver.
  2. Greedy Search: It prioritizes the "easiest" tasks to find—those the current solver cannot yet perform, but which a slightly modified version can master quickly.
  3. Non-Forgetting Constraint: A modification is only accepted if the new solver can solve the new task and all previously learned tasks in the library.
  4. Compression and Simplification: The system also invents internal goals to optimize existing solutions, making them faster or more computationally compact.

Architecture: The SLIM RNN

The primary solver utilized in these experiments is the SLIM RNN (Self-Delimiting Recurrent Neural Network). This architecture treats connection weights as programs and includes a specialized "halt" neuron to signal task completion.

To facilitate modularity, the network employs Winner-Take-All Subsets (WITAS). In this scheme, only the most active neurons within a group fire, gating information flow so that the network uses only a fraction of its total weights for any given task. This sparsity is crucial for the non-forgetting constraint, as it allows new skills to be added with minimal disruption to existing ones.

The SLIM RNN activation scheme shows how specific neurons and connections become active at different time steps to solve a task.

Experimental Validation

The authors evaluated POWERPLAY through two distinct environments:

  • Pattern Recognition: A synthetic environment where the system classifies 2D points. Tasks are generated deterministically, providing a vast space of potential challenges.
  • Fovea Control: An agent interacts with a high-resolution image of Lugano, Switzerland. It views the image through a multi-resolution "fovea" window and must learn action sequences to reach specific internal states, which serve as abstract goals.

Emergent Developmental Stages

In the pattern recognition task, a notable phenomenon emerged: the system naturally progressed through developmental stages. It first exhausted simple linear classification boundaries before moving to complex non-linear ones. This suggests that the greedy search for "easy" tasks naturally mirrors human-like skill progression.

Evolution of decision boundaries in Experiment 1, showing the transition from linear to non-linear complexity as more tasks are invented.

Key Results and Insights

  • Efficient Skill Acquisition: In the fovea experiment, the SLIM RNN mastered 67 novel action sequences within 8 hours on a standard PC. Including optimization tasks, the system successfully managed 340 distinct goals.
  • Extreme Modularity: On average, the system only needed to modify 2.9 weights (out of 1120) to learn a new skill. This high degree of self-modularization demonstrates that the system effectively reuses knowledge while isolating new learning.
  • Continuous Self-Optimization: Quantitative results showed that the interaction steps required for specific tasks generally decreased over time as the system "played" with its own code to compress solutions.

Conclusion and Future Work

POWERPLAY demonstrates a viable path toward general-purpose problem solvers that learn through autonomous play. While current experiments focus on pure exploration, future work aims to show how this self-directed learning phase can accelerate the acquisition of externally posed, user-defined tasks through knowledge transfer. Addressing the challenges of noisy environments and internal state interpretability remains a priority for scaling these open-ended systems.

Summary updated

Abstract

Like a scientist or a playing child, PowerPlay not only learns new skills to solve given problems, but also invents new interesting problems by itself. By design, it continually comes up with the fastest to find, initially novel, but eventually solvable tasks. It also continually simplifies or compresses or speeds up solutions to previous tasks. Here we describe first experiments with PowerPlay. A self-delimiting recurrent neural network SLIM RNN is used as a general computational problem solving architecture. Its connection weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences. Our PowerPlay-driven SLIM RNN learns to become an increasingly general solver of self-invented problems, continually adding new problem solving procedures to its growing skill repertoire. Extending a recent conference paper, we identify interesting, emerging, developmental stages of our open-ended system. We also show how it automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.
BibTeX
@misc{srivastava2012first,
  author = {Rupesh Kumar Srivastava and Bas R. Steunebrink and Jürgen Schmidhuber},
  title = {{First Experiments with PowerPlay}},
  year = {2012},
  eprint = {1210.8385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1210.8385},
}
Figure