Introduction
In the early 1990s, the majority of reinforcement learning (RL) research relied heavily on the Markov property—the assumption that an agent's current sensory input contains all the information necessary to choose the optimal next action. However, real-world environments are rarely so accommodating. Agents frequently face a "non-Markovian interface," where understanding the current situation requires memory of past events. For instance, knowing a car's current position is insufficient for safe driving; one must also know its velocity, which requires recalling previous positions.
Furthermore, traditional RL systems typically relied on a single scalar reward. In contrast, real-world reinforcement is often vector-valued, consisting of multiple distinct types of "pain" and "pleasure" that an agent must learn to balance. Jürgen Schmidhuber’s foundational 1990 paper addresses these limitations by introducing a model-based RL framework that utilizes two interacting recurrent neural networks (RNNs) to navigate complex, memory-dependent tasks.
Methodology: A Dual-Network Architecture
To tackle non-Markovian environments and multi-dimensional reinforcement, the paper proposes an architecture consisting of two fully recurrent, continually running neural networks:
- The Model Network ($M$): This network functions as a "world simulator." By observing the agent's actions and the environment's subsequent reactions, it learns the underlying dynamics of the world. Crucially, it does not merely predict rewards; it attempts to predict the next state of all input units. By building this neural model, the agent effectively makes the world differentiable.
- The Controller Network ($C$): This network is responsible for choosing actions. It receives feedback from the environment via "pain" and "pleasure" units. Because the Model Network makes the environment differentiable, the Controller can "look through" the model to calculate how its actions will affect future rewards, allowing for gradient-based weight updates to minimize future pain.
Key Innovations
- Parallel Learning: Unlike sequential approaches that require training a world model before training a controller, Schmidhuber's algorithm allows both networks to learn concurrently.
- Adaptive Randomness: To prevent the controller from falling into a "deadlock" (repeating the same failing action), the system uses probabilistic outputs. The network adjusts both the mean and the variance of its actions to actively explore the environment.
- Credit Assignment Paths: The recurrent nature of the networks allows the system to build "credit assignment paths" into the past, enabling the agent to trace current "pain" back to an action taken several steps prior.
- Pain as an Input: Rather than treating reward as an external, separate signal, "pain" and "pleasure" are treated as conventional input units with fixed desired values (e.g., the target value for a "pain" unit is always zero).
Experimental Setup and Results
As a foundational paper, the framework was tested on simulated control environments designed to stress-test memory and vector-valued feedback:
- The Reinforcement Flip-Flop: The agent was tasked with mimicking a memory-based electronic flip-flop circuit. It was not given the "correct" answers, only "pain" signals when it output the wrong signal. The system successfully learned the flip-flop behavior from scratch, proving the architecture's viability for memory-dependent tasks.
- Non-Markovian Pole Balancing: In this "blind" version of the classic cart-pole problem, the agent was deliberately deprived of velocity and angular momentum data. It was forced to derive these metrics from sequences of positions using its recurrent memory. The agent successfully achieved balancing times of over 1,000 steps—a highly significant feat for the era.
- Markovian Pole Balancing: This environment was used to benchmark the vector-valued adaptive critic against existing standards. The vector-valued critic reached 30,000 steps of balancing within just 750 failures. This vastly outperformed previous state-of-the-art results (such as Anderson, 1986), which required roughly 7,000 failures to achieve similar performance.
Limitations and Future Directions
While groundbreaking, the paper identifies several limitations inherent to the proposed architecture:
- Spatial Locality: The algorithm is "local in time" but not "local in space." Consequently, the computational complexity grows significantly as the number of network weights increases.
- The Local Minimum Trap: As with all gradient-descent-based methods, the interacting networks are susceptible to getting stuck in sub-optimal local minima.
- The Long Time-Lag Problem: The author notes that while the system handles short-term memory effectively, it struggles to bridge very long time lags between actions and their eventual consequences.
To address these challenges, Schmidhuber pointed toward the development of "adaptive sub-goal generators" and "causality detectors" to handle more complex, hierarchical tasks. These proposed future directions laid the conceptual groundwork for his later, highly influential work on Long Short-Term Memory (LSTM) architectures, which would eventually solve the long time-lag problem in deep learning.
Summary updated