Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Guojian Wang, Faguo Wu, Xiao Zhang, Ning Guo, Zhiming Zheng

TL;DR

This paper introduces a novel "Trajectory-Constrained Exploration" (TACE) strategy for deep reinforcement learning (DRL) that tackles hard-exploration problems. By leveraging imperfect offline demonstrations, TACE guides agents away from suboptimal behaviors and encourages more systematic exploration, leading to improved performance in complex environments.

What Problem Does This Paper Address?

Deep Reinforcement Learning (DRL) has shown remarkable success in various domains, but it struggles with "hard-exploration problems." These arise in environments with sparse rewards (where rewards are infrequent), deceptive rewards (where agents are lured into suboptimal solutions), and large state spaces. In such scenarios, random exploration is inefficient, and agents can easily get stuck in local optima, preventing them from discovering optimal policies. Existing methods often rely on complex architectures for novelty detection or sensitive hyperparameters, leading to instability. This paper aims to provide a more stable and efficient exploration strategy that can be applied to both single- and multi-agent DRL.

Methods

The core of the proposed method is the Adaptive Trajectory-Constrained Exploration (TACE) strategy. It addresses hard-exploration by framing policy optimization as a constrained optimization problem.

  • Leveraging Offline Demonstrations: TACE uses a collection of "incomplete offline demonstrations" (which might lead to suboptimal goals) as a reference. These demonstrations are stored in a replay memory.
  • Trajectory-Distance Metric: The method employs the Maximum Mean Discrepancy (MMD) to measure the disparity between the state-action visitation distributions of the agent's current trajectories and the offline demonstration trajectories. This MMD distance serves as a measure of novelty.
  • Constrained Optimization: The agent's policy is updated to maximize the standard RL objective while being constrained to stay away from the regions represented by the suboptimal demonstrations, as measured by the MMD distance.
  • Adaptive Scaling and Normalization: To improve stability and reduce hyperparameter sensitivity, TACE incorporates an Adaptive Scaling Method (ASM) to dynamically adjust the Lagrange multiplier (which balances the RL objective and the MMD constraint) and a Distance Normalization Method to adapt the constraint boundary based on the current epoch's exploration progress.
  • Policy Gradient Algorithm: The authors propose specific policy-gradient-based algorithms (TCPPO, TCHRL, TCMAE) that integrate these concepts, utilizing adaptively clipped trajectory-distance rewards.

Datasets

The paper evaluates TACE on several benchmark environments:

  • 2D Grid Worlds: Two mazes of different sizes (50x50 and 70x70) with sparse rewards and deceptive goals.
  • MuJoCo Tasks:
    • Deceptive Reacher: A continuous control task with a misleading reward.
  • Hierarchical Control Tasks (Maze 0 and Maze 1): Two MuJoCo mazes with sparse rewards and multiple goals of varying values.
  • Multi-Agent Environments:
    • Discrete Multiple-Particle Environment (MPE): A discrete state-action space environment with cooperative agents and deceptive rewards.
  • SparseAnt Maze: A modified MuJoCo continuous control task with sparse rewards.

No new datasets were introduced; existing benchmarks were used and adapted.

Key Insights and Ideas

  • Imperfect Demonstrations are Valuable: The paper demonstrates that even "incomplete" or "suboptimal" offline demonstrations can be highly effective for guiding exploration, rather than requiring perfect expert trajectories. This makes the approach more practical.
  • MMD as a Novelty Measure: Using MMD to quantify the difference between trajectory distributions provides a principled way to measure exploration novelty and guide the agent away from previously explored (and potentially suboptimal) regions.
  • Constrained Optimization for Stability: Framing exploration as a constrained optimization problem, rather than simply adding intrinsic rewards, helps to stabilize training and prevent the agent from getting stuck in local optima.
  • Adaptive Mechanisms are Crucial: The adaptive scaling and normalization methods are key to making the approach robust and less sensitive to hyperparameter choices, which is a common issue in DRL.
  • Unified Approach for Single and Multi-Agent: The proposed framework is shown to be effective for both single-agent and multi-agent reinforcement learning settings.

Key Results

  • Superior Performance: TACE-based algorithms (TCPPO, TCHRL, TCMAE) consistently outperformed baseline methods across various tasks, including grid worlds, MuJoCo mazes, and multi-agent environments.
  • Faster Learning and Higher Returns: TCPPO achieved higher average returns and success rates, often learning significantly faster than other methods. For instance, in the 70x70 grid world, TCPPO achieved a perfect score (6.00 average reward, 1.00 success rate), while some baselines struggled to reach 3.00 average reward.
  • Effective Exploration: State-visitation count graphs showed that TACE explored a wider region of the state space compared to baseline methods, successfully escaping deceptive reward traps.
  • Reduced Myopic Behavior: The method effectively prevented agents from adopting myopic behaviors, allowing them to discover globally optimal goals rather than settling for easily accessible suboptimal ones.
  • Theoretical Guarantees: The paper provides theoretical analysis, including worst-case approximation error bounds, to support the validity and effectiveness of the TACE approach.

Limitations and Future Work

  • Exploiting Imperfect Trajectories: The authors acknowledge that while TACE uses imperfect offline trajectories as references, further research could explore how to exploit these experiences more directly to accelerate deep exploration, especially in teamwork settings.
  • Hyperparameter Sensitivity (Mitigated): While the adaptive mechanisms reduce sensitivity, some hyperparameters (like the initial value of σ) still require careful selection, though the paper suggests heuristic values that work across environments.
  • Future Research Directions: The paper suggests focusing on diverse exploration in teamwork and leveraging imperfect demonstrations more effectively to indirectly accelerate learning and drive deeper exploration.

Summary updated

Abstract

Deep reinforcement learning (DRL) faces significant challenges in addressing the hard-exploration problems in tasks with sparse or deceptive rewards and large state spaces. These challenges severely limit the practical application of DRL. Most previous exploration methods relied on complex architectures to estimate state novelty or introduced sensitive hyperparameters, resulting in instability. To mitigate these issues, we propose an efficient adaptive trajectory-constrained exploration strategy for DRL. The proposed method guides the policy of the agent away from suboptimal solutions by leveraging incomplete offline demonstrations as references. This approach gradually expands the exploration scope of the agent and strives for optimality in a constrained optimization manner. Additionally, we introduce a novel policy-gradient-based optimization algorithm that utilizes adaptively clipped trajectory-distance rewards for both single- and multi-agent reinforcement learning. We provide a theoretical analysis of our method, including a deduction of the worst-case approximation error bounds, highlighting the validity of our approach for enhancing exploration. To evaluate the effectiveness of the proposed method, we conducted experiments on two large 2D grid world mazes and several MuJoCo tasks. The extensive experimental results demonstrate the significant advantages of our method in achieving temporally extended exploration and avoiding myopic and suboptimal behaviors in both single- and multi-agent settings. Notably, the specific metrics and quantifiable results further support these findings. The code used in the study is available at \url{https://github.com/buaawgj/TACE}.
BibTeX
@inproceedings{wang2023adaptive,
  author = {Guojian Wang and Faguo Wu and Xiao Zhang and Ning Guo and Zhiming Zheng},
  title = {{Adaptive trajectory-constrained exploration strategy for deep reinforcement learning}},
  booktitle = {Knowledge-Based},
  year = {2023},
}
Figure