Overview
This paper introduces a connectionist planning method that combines a differentiable world model with an "experience network" to optimize action sequences. By using gradient descent to refine plans through a chain of predicted future states, the system can adapt to new goals at runtime and solve complex temporal credit assignment problems more efficiently than traditional reactive controllers.
The Problem: Limitations of Reactive Controllers
In reinforcement learning, agents often struggle with the temporal credit assignment problem: determining which specific action in a long sequence led to a later reward or penalty.
Most connectionist (neural network) approaches at the time relied on "reactive" controllers that mapped states directly to actions. The authors identify three major flaws in this pure reflex-based approach:
- Training Lag: The controller’s learning often lags behind the world model’s learning.
- Inflexibility: If the objective function changes (e.g., the robot needs to reach a different goal), the entire controller usually requires expensive retraining.
- Many-to-One Mapping: Standard controllers struggle when multiple different actions are equally optimal, often collapsing to just one sub-optimal average.
To overcome these limitations, the authors propose moving away from pure reflex agents toward agents that can plan by looking ahead into the future using a learned, differentiable model of the environment.
Methods: Differentiable Planning
The proposed architecture consists of three main components working in tandem:
- System Identification (The World Model): A neural network is trained to predict the next state and the reinforcement value given the current state and an action. It learns continuously by comparing its predictions to actual interactions with the environment.
- Plan Optimization: To plan $N$ steps ahead, the authors "chain" $N$ copies of the world model. They define a "reinforcement energy" function (the difference between predicted and desired reward). Because the world model is a differentiable neural network, they can use gradient descent to adjust the actions themselves to minimize this energy.
- The Experience Network: To avoid starting the gradient descent from scratch every time, a separate network is trained to predict the outcome of previous successful planning sessions. This network provides a "warm start" or initial plan, which is then further refined by the gradient descent process.
A particularly clever trick introduced is the use of variable plan lengths. If the world model's error is high, the system shortens its look-ahead horizon to avoid making decisions based on unreliable "hallucinations" of the future.
Key Insights
- Gradient Descent in Action Space: The core innovation is treating a plan as a set of parameters that can be optimized via backpropagation. Instead of backpropagating to change network weights, the system backpropagates to change the input actions.
- Goal Flexibility: Because planning happens at runtime through the world model, you can change the robot's behavior (e.g., "approach the ball from above" instead of "just catch it") by simply modifying the energy function. No retraining of the underlying networks is required.
- Smoothing for Binary Rewards: Gradient descent usually fails with binary rewards (0 or 1) because the gradient is zero almost everywhere. The authors solved this by using a second "smoothed" world model (trained with weight decay) specifically for generating gradients, while using the "sharp" model for actual state prediction.
Experimental Results
The authors evaluated their approach on two primary simulations:
- Robotics Task: A simulated 3D robot arm was tasked with intercepting a rolling ball. While a simple reactive controller would just "follow" the ball, the planning agent successfully found "shortcuts," predicting where the ball would be and moving to intercept it. The authors demonstrated that starting with a plan from the experience network and refining it significantly outperformed initial heuristics.
- Pole Balancing: The system was tested on the classic inverted pendulum problem, a "binary reinforcement" task where the only feedback is whether the pole has fallen. With a look-ahead of only $N=4$ steps, the agent was able to balance the pole for more than 20,000 steps. In contrast, a random controller resulted in the pole falling after an average of only 10 steps.
Limitations and Future Work
While powerful, the approach has limitations. The agent is inherently "nearsighted," unable to account for reinforcements that occur beyond its $N$-step planning horizon. Furthermore, because the planning relies on gradient descent, the agent can get stuck in local optima—choosing a "good" sequence of actions while missing a "great" one. Finally, planning via gradient descent is computationally more expensive than a single forward pass through a reactive controller, which poses challenges for high-frequency real-time systems.
Summary updated