Abstract
This paper presents "Jean," an AI agent architecture inspired by Piagetian developmental psychology, designed to learn complex behaviors through the iterative refinement of internal models. By employing the Experimental State Splitting (ESS) algorithm, Jean constructs "action schemas" and "gists"—probabilistic finite state machines—that represent tactical strategies. We demonstrate that these learned schemas can be effectively transferred across varying environments in a military simulation, significantly accelerating learning in complex scenarios compared to learning from scratch.
Introduction
A central challenge in artificial intelligence is creating agents that can autonomously discover complex behaviors from a set of simple, innate capabilities. Drawing from developmental psychology, specifically Jean Piaget's concepts of assimilation and accommodation, we propose an agent named Jean. Jean learns by fitting new experiences into existing mental structures (assimilation) and modifying those structures when they fail to predict environmental outcomes (accommodation). The goal is to discover the "decision regions"—the specific conditions under which an agent should switch from one primitive action to another—to achieve high-level objectives.
Methods: Action Schemas and ESS
Jean’s knowledge is organized into action schemas. Each schema consists of three components: a controller (a primitive action like "move-to"), a map (expected trajectories of sensor data over time), and decision regions (spatiotemporal boundaries for state evaluation).
Jean represents high-level plans as "gists," which are essentially Finite State Machines (FSMs). Unlike traditional FSMs, both the states (decision-making) and the transitions (actions) are temporal processes.
The Experimental State Splitting (ESS) algorithm drives learning. Jean begins with a minimal gist (start to goal). As it interacts with the environment, ESS monitors for unpredictability. When sensor variables change simultaneously in ways the current model cannot predict, the algorithm "splits" the state, creating a more granular model of the task. To transfer knowledge, Jean retains the structural topology of a learned FSM but resets the transition probabilities, allowing it to "jump-start" in new environments.
Experimental Evaluation
We tested Jean in ISIS, a 3D real-time strategy military simulator. Jean controlled a squad of soldiers tasked with eliminating an opposing force. We compared two conditions: Condition B (learning from scratch in a difficult scenario) and Condition AB (learning in an easier scenario A first, then transferring to the difficult scenario B). The primary metric was the transfer ratio ($r$), where $r > 1.0$ indicates positive transfer.
Results and Discussion
Our experiments revealed significant positive transfer in most cases:
- Near to Mixed Starts: Learning in a "near" scenario and transferring to a mix of distances yielded $r = 1.591$.
- Open to Mountainous Terrain: Transferring from open terrain to a map with line-of-sight-blocking mountains resulted in $r = 1.531$.
However, we identified a "Far-Start" problem. When the initial scenario was too difficult (e.g., starting far from the enemy), the search space was too large for Jean to discover any useful patterns, resulting in $r = 0.970$ (no transfer). This underscores the necessity of a "curriculum" for effective autonomous learning.
Conclusion
Jean demonstrates that predictability can serve as a powerful intrinsic learning signal. By splitting states to minimize boundary entropy, the agent constructs a reliable model of the world. Future work will focus on moving beyond simple heuristics for state splitting toward an "experimental" agent that actively tests causal hypotheses to refine its action schemas.
Summary updated