Learning Generalized Temporal Abstractions across Both Action and Perception

Khimya KhetarpalAAAI 2019 · OpenReview

Abstract

Learning temporal abstractions—partial solutions to tasks that can be reused for more complex objectives—is a fundamental requirement for agents to plan, learn, and reason across multiple resolutions of perception and time. Just as humans acquire and build upon existing skills to solve increasingly difficult problems, AI agents should develop skills continually and hierarchically. This research addresses how agents can efficiently represent and utilize world knowledge in continual tasks. By extending the options framework, we introduce differentiable interest functions and the concept of temporally extended perception, creating a unified framework for temporal abstractions in both action and perception.

Introduction

In standard Reinforcement Learning (RL), agents typically interact with their environment through fine-grained, step-by-step actions. While Hierarchical Reinforcement Learning (HRL) and the "options framework" allow for higher-level behaviors (e.g., "exiting a room" rather than "moving left"), they often suffer from degeneracy. In many end-to-end architectures, such as the option-critic framework, options frequently collapse into single time steps or become so generalized that they are available in every state, failing to specialize. This research proposes that by constraining where skills can be used and extending how agents perceive time, we can foster the development of specialized, reusable skills essential for lifelong learning.

Specialization via Interest Functions

A primary hurdle in HRL is the assumption that every skill is available in every situation. In contrast, human skills are context-dependent; the "skill" of braking is only relevant when driving. To mirror this, we reintroduce and modernize the concept of an "initiation set" through interest functions ($I_\omega$).

An interest function $I_\omega(s)$ quantifies the "interest" an option $\omega$ has in being active in a specific state $s$. By making these functions differentiable, we allow the agent to learn them automatically via a derived gradient (Theorem 1). This mathematical framework forces options to specialize in specific regions of the state space, preventing the "jack of all trades, master of none" problem common in current HRL models.

Temporally Extended Perception

Most RL agents process the world frame-by-frame, creating a mismatch between high-level actions and low-level perception. We propose that perceptual features should represent multiple time steps, synchronized with the duration of the agent’s options. This "embodied interaction" allows the agent to build internal representations that serve as meaningful "pseudo-rewards" for specific tasks—such as the visual signature of a door opening. By unifying the temporal scale of what an agent sees with what it does, we create a more robust foundation for complex reasoning.

Robustness and Visual Attention

To test the role of attention in these abstractions, we developed the Visually-Attentive UNREAL agent. This model modifies the standard UNREAL architecture to "foveate" on regions of an image that align with human visual saliency.

We evaluated this approach against three levels of environmental perturbations:

  1. Easy: Gaussian noise.
  2. Moderate: Consistent random tinting.
  3. Difficult: Random tinting with varying hues.

The results demonstrated that agents focusing on human-salient regions are significantly more robust to distractors, such as flickering lights or color shifts, compared to baseline agents. This suggests that selective attention is a vital component of stable temporal abstractions.

Conclusion and Future Work

This research provides a theoretical and empirical path toward more efficient RL agents by rethinking temporal abstractions. While the interest function framework is mathematically grounded, and the attention-based models show promise in robustness, several challenges remain. Future work will focus on moving these concepts from the brainstorming and experimental phases into a fully realized lifelong learning context, where agents can incrementally build a library of specialized skills over their entire operational lifespan.

Summary updated

Abstract

Learning temporal abstractions which are partial solutions to a task and could be reused for other similar or even more complicated tasks is intuitively an ingredient which can help agents to plan, learn and reason efficiently at multiple resolutions of perceptions and time. Just like humans acquire skills and build on top of already existing skills to solve more complicated tasks, AI agents should be able to learn and develop skills continually, hierarchically and incrementally over time. In my research, I aim to answer the following question: How should an agent efficiently represent, learn and use knowledge of the world in continual tasks? My work builds on the options framework, but provides novel extensions driven by this question. We introduce the notion of interest functions. Analogous to temporally extended actions, we propose learning temporally extended perception. The key idea is to learn temporal abstractions unifying both action and perception.
BibTeX
@inproceedings{khetarpal2019learning,
  author = {Khimya Khetarpal},
  title = {{Learning Generalized Temporal Abstractions across Both Action and Perception}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=HI5-_YjOoGq},
}
Figure