Introduction
Reinforcement Learning (RL) has achieved significant milestones, yet it frequently encounters the "curse of dimensionality" when applied to continuous, real-world environments. While Hierarchical Reinforcement Learning (HRL) offers a solution by decomposing complex tasks into simpler sub-tasks, these hierarchies are typically hand-coded by human experts. This paper introduces the Qualitative Learner of Actions and Perception (QLAP), an algorithm designed for developmental robotics that enables agents to autonomously discover action hierarchies from scratch. QLAP allows agents to identify meaningful "milestones" and ignore irrelevant sensory data, effectively bridging the gap between continuous input and discrete symbolic reasoning.
Methodology: The QLAP Framework
The QLAP algorithm operates by discretizing a continuous world into qualitative "landmarks" and modeling transitions using Dynamic Bayesian Networks (DBNs). The process involves four primary stages:
- Qualitative Representation: Instead of raw coordinates, QLAP tracks variables relative to landmarks, identifying whether values are increasing, decreasing, or stable. It specifically flags critical points, such as when the distance between a robot's hand and an object reaches zero.
- Dynamic Bayesian Networks (DBNs): Through initial "motor babbling" (random exploration), the agent learns small predictive models. These DBNs represent the probability that a specific event (antecedent) will lead to another (consequent) within a given context.
- Options Framework: Reliable DBNs are converted into "options"—specialized policies aimed at achieving specific qualitative states. These options utilize state abstraction, focusing only on variables relevant to the sub-task.
- Second-Order DBNs: The hierarchy is refined by tracking the success of these options. If an option fails, the agent incorporates additional contextual variables to improve reliability.
Experimental Setup
The researchers evaluated QLAP within the Breve simulation environment. The setup included a robot torso with a 2-degree-of-freedom arm tasked with interacting with blocks on a table. Crucially, the environment included "distractor" objects—floating spheres that move independently but cannot be interacted with—to test the agent's ability to filter noise.
QLAP was benchmarked against RL-Tile, a standard reinforcement learning approach that uses tile coding to handle continuous state spaces.
Key Results and Insights
The evaluation focused on learning speed, robustness to distractors, and generalization.
- Learning Efficiency: In base comparisons, QLAP reached target performance within approximately 80,000 timesteps. In contrast, RL-Tile struggled due to the high "task diameter"—the sheer number of primitive actions required to reach a goal.
- Robustness to Distractors: QLAP's most significant advantage appeared in the presence of irrelevant floating objects. While RL-Tile's performance plummeted as its state space grew more complex, QLAP remained unaffected. Its learned models correctly identified distractors as having no causal link to the task.
- Temporal and State Abstraction: By creating high-level actions (e.g., "move hand to block"), QLAP reduces the number of steps needed to reach a goal. This intrinsic motivation to build reliable models allows the agent to learn without external rewards.
Conclusion and Future Work
QLAP demonstrates that an agent can autonomously build a functional action hierarchy by seeking to model its environment qualitatively. While the current model assumes variables can be changed independently—a limitation in highly coupled systems—it provides a robust foundation for autonomous learning in continuous spaces. Future research will explore scaling to higher-dimensional environments and refining landmark discovery to capture more subtle environmental dynamics.
Summary updated