Abstract
In reinforcement learning, temporal abstraction in the action space, exemplified by action repetition, is a technique to facilitate policy learning through extended actions. However, a primary limitation in previous studies of action repetition is its potential to degrade performance, particularly when sub-optimal actions are repeated. This issue often negates the advantages of action repetition. To address this, we propose a novel algorithm named Uncertainty-aware Temporal Extension (UTE). UTE employs ensemble methods to accurately measure uncertainty during action extension. This feature allows policies to strategically choose between emphasizing exploration or adopting an uncertainty-averse approach, tailored to their specific needs. We demonstrate the effectiveness of UTE through experiments in Gridworld and Atari 2600 environments. Our findings show that UTE outperforms existing action repetition algorithms, effectively mitigating their inherent limitations and significantly enhancing policy learning efficiency.
Introduction
Reinforcement learning (RL) agents often struggle with "long-horizon" tasks where rewards are sparse and far in the future. Temporal abstraction, specifically action repetition, allows agents to explore further by committing to an action for multiple steps (a "skip length"). However, blind commitment can be catastrophic in risky environments. If an agent repeats a sub-optimal action in a dangerous area, such as a platformer with pits or a gridworld with lava, it may fail before it can correct its course. This paper addresses the challenge of balancing deep exploration with safety by introducing Uncertainty-aware Temporal Extension (UTE).
Methods
UTE decomposes decision-making into two distinct parts: an action policy ($\pi_a$) that decides what action to take, and an extension policy ($\pi_e$) that decides how long to repeat it. To manage the risks of temporal abstraction, UTE utilizes an ensemble of Q-function "heads" to estimate uncertainty. The repetition length $j$ is selected using an uncertainty-aware decision rule:
$$j = \text{argmax}_{j'} \{ \text{mean}(Q) + \lambda \cdot \text{std}(Q) \}$$
The parameter $\lambda$ is the critical "uncertainty parameter" that controls the agent's attitude toward risk. A positive $\lambda$ makes the agent "optimistic," encouraging it to repeat actions where the potential payoff is uncertain—ideal for finding hidden rewards in sparse environments. Conversely, a negative $\lambda$ makes the agent "pessimistic" or uncertainty-averse, causing it to shorten repetitions in dangerous or unknown areas.
By separating the action choice from the repetition choice, UTE reduces the search space from a multiplicative one to an additive one, stabilizing the learning process. Furthermore, UTE trains all ensemble heads against the same target, ensuring that uncertainty variance converges toward zero as the agent gains experience and masters the environment.
Results
The effectiveness of UTE was evaluated across several benchmarks, including Chain MDP, risky Gridworlds (Bridge and ZigZag), and Atari 2600 games. In the Chain MDP, UTE achieved a Normalized AUC of 0.919, outperforming the previous state-of-the-art TempoRL (0.904).
In Gridworlds containing "lava" hazards, UTE demonstrated superior risk mitigation. While previous methods like TempoRL often repeated actions into the lava because they lacked commitment awareness, UTE's uncertainty-averse setting ($\lambda < 0$) allowed it to shorten repetitions in dangerous areas, effectively acting as a safety valve. In the Bridge map, UTE maintained an AUC of 0.86, while TempoRL struggled at 0.44.
In Atari 2600 environments, UTE consistently reached higher final rewards and showed faster learning speeds than vanilla DDQN and other action-repetition baselines like FiGAR and DAR. On the continuous Pendulum-v0 task, UTE-DDPG achieved higher rewards (-147.4 vs. -160.1 for t-DDPG) and showed lower variance, indicating more stable learning.
Conclusion
UTE provides a robust framework for temporal abstraction by explicitly accounting for uncertainty. By separating action and extension policies and employing an ensemble-based uncertainty estimate, UTE enables agents to adapt their exploration strategy to the inherent risks of the environment. While ensemble methods increase computational cost, the gains in stability and safety make UTE a powerful tool for complex RL tasks. Future work includes extending UTE to fully continuous temporal abstractions and further optimizing the adaptive mechanism for the uncertainty parameter.
Summary updated