Abstract
Learning temporal abstractions—partial solutions to tasks that can be reused for more complex objectives—is a fundamental requirement for agents to plan, learn, and reason across multiple resolutions of perception and time. Just as humans acquire and build upon existing skills to solve increasingly difficult problems, AI agents should develop skills continually and hierarchically. This research addresses how agents can efficiently represent and utilize world knowledge in continual tasks. By extending the options framework, we introduce differentiable interest functions and the concept of temporally extended perception, creating a unified framework for temporal abstractions in both action and perception.
Introduction
In standard Reinforcement Learning (RL), agents typically interact with their environment through fine-grained, step-by-step actions. While Hierarchical Reinforcement Learning (HRL) and the "options framework" allow for higher-level behaviors (e.g., "exiting a room" rather than "moving left"), they often suffer from degeneracy. In many end-to-end architectures, such as the option-critic framework, options frequently collapse into single time steps or become so generalized that they are available in every state, failing to specialize. This research proposes that by constraining where skills can be used and extending how agents perceive time, we can foster the development of specialized, reusable skills essential for lifelong learning.
Specialization via Interest Functions
A primary hurdle in HRL is the assumption that every skill is available in every situation. In contrast, human skills are context-dependent; the "skill" of braking is only relevant when driving. To mirror this, we reintroduce and modernize the concept of an "initiation set" through interest functions ($I_\omega$).
An interest function $I_\omega(s)$ quantifies the "interest" an option $\omega$ has in being active in a specific state $s$. By making these functions differentiable, we allow the agent to learn them automatically via a derived gradient (Theorem 1). This mathematical framework forces options to specialize in specific regions of the state space, preventing the "jack of all trades, master of none" problem common in current HRL models.
Temporally Extended Perception
Most RL agents process the world frame-by-frame, creating a mismatch between high-level actions and low-level perception. We propose that perceptual features should represent multiple time steps, synchronized with the duration of the agent’s options. This "embodied interaction" allows the agent to build internal representations that serve as meaningful "pseudo-rewards" for specific tasks—such as the visual signature of a door opening. By unifying the temporal scale of what an agent sees with what it does, we create a more robust foundation for complex reasoning.
Robustness and Visual Attention
To test the role of attention in these abstractions, we developed the Visually-Attentive UNREAL agent. This model modifies the standard UNREAL architecture to "foveate" on regions of an image that align with human visual saliency.
We evaluated this approach against three levels of environmental perturbations:
- Easy: Gaussian noise.
- Moderate: Consistent random tinting.
- Difficult: Random tinting with varying hues.
The results demonstrated that agents focusing on human-salient regions are significantly more robust to distractors, such as flickering lights or color shifts, compared to baseline agents. This suggests that selective attention is a vital component of stable temporal abstractions.
Conclusion and Future Work
This research provides a theoretical and empirical path toward more efficient RL agents by rethinking temporal abstractions. While the interest function framework is mathematically grounded, and the attention-based models show promise in robustness, several challenges remain. Future work will focus on moving these concepts from the brainstorming and experimental phases into a fully realized lifelong learning context, where agents can incrementally build a library of specialized skills over their entire operational lifespan.
Summary updated