Abstract
Standard Reinforcement Learning (RL) often struggles with spatial tasks because it is difficult to design the "perfect" abstraction to simplify continuous space. This paper introduces ReLAI (Reinforcement Learning using Abstraction and Imagery), which utilizes an internal "imagery" system to predict the next state before applying an abstraction. By abstracting the predicted future rather than the current state, agents can learn faster and succeed even with imperfect perceptual tools.
Introduction
In spatial environments, an agent is bombarded with continuous, low-level data, such as pixel coordinates. To learn efficiently, the agent must group similar situations into "abstract states." For example, "near an enemy" is a more useful state than a specific coordinate pair. However, the poverty conjecture posits that a single set of perceptual rules is unlikely to perfectly abstract every possible spatial problem. If an abstraction ignores a critical feature—such as the velocity of an oncoming object—the RL agent may fail to learn optimal behavior. ReLAI proposes using internal mental simulation (imagery) to make RL more robust to these "imperfect" abstractions.
Methodology
The ReLAI framework shifts the focus from traditional state aggregation to state-action $(s, a)$ aggregation. The process involves three main components:
- Imagery System: Instead of abstracting the current state $s$, the agent uses a Euclidean "spatial scene" representation to simulate the effect of an action $a$, "imagining" the resulting future state.
- Prediction-Based Abstraction: The agent applies its abstraction function to this imagined future. The category of the current $(s, a)$ pair is defined by the predicted next abstract state.
- Q-Learning Integration: The agent learns the value of these categories. The authors prove that if imagery is accurate and rewards are captured correctly, this method converges to the optimal policy ($Q^*$).
The system was implemented using the Soar cognitive architecture, specifically leveraging its Spatial Visual System (SVS) to handle the mental imagery of moving objects.
Experimental Evaluation
The approach was evaluated in two primary environments:
- Synthetic Grid World: A simplified environment with $N$ object types used to formally compare ReLAI against standard MDP minimization techniques like bisimulation and homomorphism.
- Frogger II (Atari 2600): A complex environment where the agent navigates a frog through obstacles and water currents. The agent had to navigate to the top of the screen while avoiding moving fish and dealing with environmental noise.
Results and Discussion
Empirical tests in Frogger II revealed a significant performance advantage for ReLAI. The agent learned a successful policy much faster than standard state aggregation. In a final evaluation of 700 games (with exploration turned off), the ReLAI agent achieved a 70% win rate with an average reward of 439. In contrast, the baseline state-aggregation agent (using the same perceptual features but without imagery) won only 45% of games and had a negative average reward of -66.
Key insights from the study include:
- Broadening Abstractions: ReLAI relaxes the strict $Q^*$-irrelevance requirement of standard state aggregation, allowing effective use of simpler abstractions that are effective when paired with one-step-ahead prediction.
- Imagery as a Bridge: Imagery connects low-level Euclidean space (where physics is easy to predict) with high-level symbolic logic (where learning is efficient).
- Robustness to Noise: The agent could predict collisions accurately using concrete coordinates, even when the final abstract symbols seen by the RL algorithm were quite simple.
Conclusion
ReLAI demonstrates that internal imagery can significantly simplify perceptual abstraction in RL. While currently dependent on the accuracy of the imagery system (e.g., assuming linear motion), future work could extend this to non-linear physics and automated discovery of task-relevant abstractions.
Summary updated