Introduction
Reinforcement Learning (RL) agents frequently struggle in "rich" environments—complex worlds characterized by hundreds of state variables, many of which are irrelevant to the immediate task. When an agent attempts to learn a policy based on the entire state space, it encounters the "curse of dimensionality," where the search space grows exponentially with the number of variables.
To address this, researchers have turned to deictic representation, a concept inspired by human selective attention. Rather than tracking absolute coordinates (e.g., "the cup at x,y,z"), humans use "pointers" to focus on task-relevant objects (e.g., "the cup I am holding"). While powerful, deictic pointers typically require manual engineering. This paper introduces Deictic Option Schemas, a framework that enables agents to automatically learn the optimal configuration of these pointers for specific sub-tasks within a hierarchical RL architecture.
Methodology: Deictic Option Schemas
The proposed framework builds upon Relativized Options. In standard RL, an "option" is a sub-routine (e.g., "exit the room"). A relativized option acts as a template that can be applied across different environmental contexts using MDP Homomorphisms—mathematical mappings that project a complex environment into a simplified "image" model.
The core innovation lies in three components:
- Factored Pointers: The agent utilizes a set of pointers to "mark" objects in the environment.
- Bayesian Learning: The agent maintains a weight vector representing beliefs about which pointer configurations are correct. These weights are updated based on how closely observed transitions in the real world align with the expected transitions in the simplified image model.
- Factored Updates: Unlike monolithic approaches that attempt to learn all pointer combinations simultaneously, this method categorizes pointers as independent or dependent. This allows for parallel belief updates, significantly accelerating convergence.
Experimental Evaluation
The researchers evaluated the approach using a simulated gridworld inspired by the classic game Pengi. The environment consists of a room containing a target diamond and eight autonomous adversaries of three types: "Benign" (random movement), "Delayers" (active chasers), and "Retrievers" (behave as delayers only after the diamond is collected).
The primary challenge for the agent is that it does not initially know which robot belongs to which category. It must use its pointers to identify the "Delayer" and "Retriever" to navigate the room safely and escape.
Key Results
The performance of the Factored Deictic Agent was compared against a Composite (Monolithic) Agent, which utilized the same pointers but lacked the factored learning mechanism.
The results demonstrated significant improvements in both speed and efficiency:
- Identification Speed: The deictic agent identified the "Delayer" robot in an average of 52 update steps, compared to 2007 steps for the monolithic agent.
- Computational Efficiency: Training the monolithic agent required 14 hours of CPU time, whereas the deictic agent completed the task in just 4 hours.
- Robustness: The deictic agent exhibited lower performance variability and converged with significantly fewer total updates (85,000 vs. 130,000), even when identifying complex behaviors like those of the "Retriever."
Conclusion
Deictic Option Schemas demonstrate that selective attention, when framed as a learnable abstraction, can drastically simplify complex RL problems. By focusing on "consistency over accuracy"—ensuring the pointed-to representation yields predictable outcomes—agents can ignore vast amounts of irrelevant data. Future work aims to relax the requirement for pre-defined sub-task models and explore more flexible "consistent representations" beyond strict homomorphisms.
Summary updated