Decision Theory-Guided Deep Reinforcement Learning for Fast Learning

Zelin Wan, Jin-Hee Cho, Mu Zhu, Ahmed H. Anwar, Charles A. Kamhoua, Munindar P. SingharXiv 2024 · OpenReview

Abstract

This paper introduces a novel approach, Decision Theory-guided Deep Reinforcement Learning (DT-guided DRL), to address the inherent cold start problem in DRL. By integrating decision theory principles, DT-guided DRL enhances agents' initial performance and robustness in complex environments, enabling more efficient and reliable convergence during learning. Our investigation encompasses two primary problem contexts: the cart pole and maze navigation challenges. Experimental results demonstrate that the integration of decision theory not only facilitates effective initial guidance for DRL agents but also promotes a more structured and informed exploration strategy, particularly in environments characterized by large and intricate state spaces. The results of experiment demonstrate that DT-guided DRL can provide significantly higher rewards compared to regular DRL. Specifically, during the initial phase of training, the DT-guided DRL yields up to an 184% increase in accumulated reward. Moreover, even after reaching convergence, it maintains a superior performance, ending with up to 53% more reward than standard DRL in large maze problems. DT-guided DRL represents an advancement in mitigating a fundamental challenge of DRL by leveraging functions informed by human (designer) knowledge, setting a foundation for further research in this promising interdisciplinary domain.

Introduction

Deep Reinforcement Learning (DRL) has achieved remarkable success in various domains, yet it remains plagued by the "cold start" problem. In the early stages of training, an agent typically relies on random exploration. In complex or high-stakes environments, this leads to significant sample inefficiency and potential safety risks, as the agent may spend thousands of steps in unproductive or dangerous states before encountering a reward signal.

While techniques like Transfer Learning (TL) or Imitation Learning (IL) can mitigate these issues, they often require high-quality pre-trained models or expert demonstrations, which are not always available. This paper proposes a hybrid framework that leverages classical Decision Theory (DT) to provide initial guidance through human-designed heuristics, bridging the gap between random exploration and expert-level performance.

Methodology

The core of the proposed approach is the Decision Theory-guided Deep Reinforcement Learning (DT-guided DRL) framework. This architecture runs two processes in parallel: a standard Deep Neural Network (using the Proximal Policy Optimization, or PPO, algorithm) and a Decision Theory agent.

The DT-guided DRL hybrid architecture.

Hybrid Architecture and Integration

As shown in the architecture diagram, the system combines the outputs of both components:

  1. Utility Functions: Instead of learning from scratch, the framework utilizes simple, equation-based utility functions derived from domain knowledge. For instance, in a navigation task, the utility function might favor actions that reduce the Euclidean distance to a goal while penalizing moves toward obstacles.
  2. Logit Combination: The logits from the Neural Network are combined with the probabilities generated by the DT agent.
  3. Dynamic Decay Mechanism: A weight factor $w$ controls the influence of the DT agent. Initially set to $w=1$, the agent relies heavily on human-designed heuristics. As training progresses and the DRL agent gains experience, $w$ decays toward 0, eventually allowing the neural network to take full control of the policy.

Experimental Setup

The researchers evaluated the framework using two benchmarks from the Gymnasium library:

  • Cart Pole: A classic control task requiring the agent to balance a pole on a moving cart.
  • Gym-Maze: A 2D navigation task tested across various grid sizes, from $3 \times 3$ to $8 \times 8$, to assess scalability and performance in increasingly sparse reward environments.

Results and Discussion

The experimental results highlight several key advantages of the DT-guided approach. Most notably, the framework provides a "safety net" that prevents the agent from wasting exploration time in irrelevant parts of the state space.

Performance comparison across maze complexities ($m=3$ to $m=8$).

Key Performance Metrics

  • Early Training Gains: DT-guided DRL achieved up to a 184% increase in accumulated reward during the initial training phase compared to standard PPO.
  • Superior Convergence: In the most complex maze scenarios ($m=8$), the hybrid agent outperformed standard DRL by 53% in final accumulated reward.
  • Robustness: Interestingly, DT-guided DRL often outperformed Transfer Learning (TL) in maze tasks, suggesting that direct mathematical heuristics can be more robust than adapting models across different tasks.
  • Structured Exploration: By assigning zero utility to actions leading to obstacles, the DT component effectively prunes the search space, leading to faster and more reliable convergence.

While the approach significantly improves sample efficiency and performance, it does introduce a higher computational cost per training step due to the dual-process requirement of calculating both the neural network pass and the utility function.

Conclusion and Future Work

DT-guided DRL successfully addresses the cold start problem by integrating human domain knowledge into the reinforcement learning process. This structured exploration leads to faster convergence and higher final rewards, particularly in complex environments with sparse rewards.

Future research will focus on scaling this approach to high-dimensional inputs, such as pixel-based data, where defining simple mathematical utility functions is more challenging. Additionally, exploring more sophisticated, performance-based schedules for the weight decay mechanism could further optimize the transition from heuristic guidance to autonomous learning.

Summary updated

Abstract

This paper introduces a novel approach, Decision Theory-guided Deep Reinforcement Learning (DT-guided DRL), to address the inherent cold start problem in DRL. By integrating decision theory principles, DT-guided DRL enhances agents' initial performance and robustness in complex environments, enabling more efficient and reliable convergence during learning. Our investigation encompasses two primary problem contexts: the cart pole and maze navigation challenges. Experimental results demonstrate that the integration of decision theory not only facilitates effective initial guidance for DRL agents but also promotes a more structured and informed exploration strategy, particularly in environments characterized by large and intricate state spaces. The results of experiment demonstrate that DT-guided DRL can provide significantly higher rewards compared to regular DRL. Specifically, during the initial phase of training, the DT-guided DRL yields up to an 184% increase in accumulated reward. Moreover, even after reaching convergence, it maintains a superior performance, ending with up to 53% more reward than standard DRL in large maze problems. DT-guided DRL represents an advancement in mitigating a fundamental challenge of DRL by leveraging functions informed by human (designer) knowledge, setting a foundation for further research in this promising interdisciplinary domain.
BibTeX
@misc{wan2024decision,
  author = {Zelin Wan and Jin-Hee Cho and Mu Zhu and Ahmed H. Anwar and Charles A. Kamhoua and Munindar P. Singh},
  title = {{Decision Theory-Guided Deep Reinforcement Learning for Fast Learning}},
  year = {2024},
  eprint = {2402.06023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.06023},
}
Figure