Learning Uncertainty-Aware Temporally-Extended Actions

Joongkyu Lee, Seung Joon Park, Yunhao Tang, Min-hwan Oh

Introduction

In reinforcement learning (RL), "temporal abstraction"—the ability to plan and act over multiple time steps—is a fundamental requirement for solving complex, long-horizon tasks. A common and effective approach to temporal abstraction is action repetition, where an agent selects an action and a duration for which to repeat it. While this reduces the search space and prevents "dithering" (ineffective back-and-forth movement), it introduces a significant risk: committing to a sub-optimal action for an extended period can lead to catastrophic failure, especially in safety-critical environments.

This paper introduces the Uncertainty-aware Temporal Extension (UTE) algorithm. UTE addresses the inherent risks of action repetition by explicitly quantifying the uncertainty associated with different repetition lengths. By leveraging an ensemble of value functions, UTE can strategically toggle between aggressive exploration and cautious, uncertainty-averse behavior.

The UTE Framework

UTE decomposes the agent's decision-making process into two distinct components: action selection and duration selection. This decomposition reduces the complexity of the search space from $|A| \times |J|$ to $|A| + |J|$, where $A$ is the action space and $J$ is the set of possible repetition lengths.

Ensemble-Based Duration Selection

The core innovation of UTE is its use of an ensemble method to estimate the value of repeating an action. The agent maintains multiple independent neural network "heads" that provide varying estimates of the value of repeating action $a$ for $j$ steps. By calculating the mean ($\mu$) and standard deviation ($\sigma$) across these estimates, the agent employs an Upper Confidence Bound (UCB)-style selection rule:

$$j = \text{argmax}_{j'} \{ \mu + \lambda \sigma \}$$

In this equation, $\lambda$ serves as a critical uncertainty parameter that dictates the agent's risk profile:

  • Optimism (Positive $\lambda$): Encourages the agent to select long repetitions where the outcome is uncertain, facilitating deep exploration in sparse-reward environments.
  • Pessimism (Negative $\lambda$): Encourages the agent to avoid long repetitions with high uncertainty, prioritizing safety in dangerous environments.

Adaptive Risk Management

Recognizing that the optimal level of risk varies by environment, the authors implement an Adaptive $\lambda$ mechanism. Using a multi-armed bandit approach, the agent automatically tunes its risk-taking behavior based on the rewards received during training. Furthermore, UTE utilizes $n$-step Q-learning, which allows the agent to propagate rewards more efficiently across extended time scales by updating value functions using intermediate transitions observed during a repeated action.

Experimental Evaluation

The researchers evaluated UTE across a diverse set of benchmarks to test its versatility in exploration, safety, and general performance.

Deep Exploration and Safety

In the Chain MDP, a toy environment requiring deep exploration to find a distant reward, UTE's optimistic strategy (high $\lambda$) proved essential for success. Conversely, in Lava Gridworlds (Bridge, ZigZag, Cliff), where a single wrong move leads to episode termination, UTE adopted a pessimistic strategy ($\lambda = -1.5$). This allowed the agent to treat uncertainty as a "safety brake," preferring shorter extensions in dangerous areas where variance in value estimates was high.

Atari 2600 Benchmarking

On a suite of 12 Atari 2600 games, UTE demonstrated significant performance gains over standard baselines. UTE (with $n$-step learning) achieved a DQN-normalized score of 1.81, representing an 81% improvement over vanilla DDQN and a 40% improvement over TempoRL, a leading action-repetition baseline. In games like Road Runner, UTE reached a score of ~12,323, substantially outperforming TempoRL’s ~8,131 within the same 10-million-frame training budget.

Continuous Control

In the Pendulum-v0 task, UTE-DDPG showed superior stability and higher cumulative rewards compared to other temporally-extended methods like FiGAR and t-DDPG. This suggests that UTE's uncertainty-aware approach is robust even in non-discrete settings and scales well as the maximum allowed repetition length increases.

Key Insights

  1. Decomposition Stabilizes Learning: Separating "what" to do from "how long" to do it makes the learning process less prone to overestimation bias and more computationally stable.
  2. Uncertainty as a Safety Mechanism: In risky environments, UTE naturally learns to avoid long, "blind" commitments because the variance of value estimates typically increases with the duration of the extension.
  3. Synergy with n-step Targets: The combination of action repetition and $n$-step learning is particularly effective, as it allows for more frequent updates and better credit assignment across extended actions.

Conclusion and Future Work

UTE provides a robust framework for learning temporally-extended actions by explicitly modeling the uncertainty of those extensions. While the use of an ensemble increases computational overhead per step, the gains in sample efficiency and safety are substantial. Future work may involve integrating UTE with more complex architectures, such as option-critic frameworks, or applying it to high-stakes robotic control tasks where balancing exploration with safety is paramount.

Summary updated

BibTeX
@inproceedings{lee2024learning,
  author = {Joongkyu Lee and Seung Joon Park and Yunhao Tang and Min-hwan Oh},
  title = {{Learning Uncertainty-Aware Temporally-Extended Actions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=pJ5qMsvXbJ},
}
Figure