Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning

Runhan Yang, Jieao Shi, Mengqi Su, Dongruo ZhouOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

Abstract

In this paper, we introduce an innovative framework that combines Hierarchical Reinforcement Learning (HRL) with Large Language Models (LLMs) to tackle the challenges of complex, sparse-reward environments. A key contribution of our approach is the emphasis on imitation learning during the early training stages, where the LLM plays a crucial role in guiding the agent by providing high-level decision-making strategies. This early-stage imitation learning significantly accelerates the agent's understanding of task structure, reducing the time needed to adapt to new environments. By leveraging the LLM’s ability to generate abstract representations of the environment, the agent can efficiently explore potential strategies, even in tasks with high-dimensional state spaces and delayed rewards. Our method introduces a dynamic annealing strategy in action sampling, balancing the agent's reliance on the LLM’s guidance with its own learned policy as training progresses. Additionally, we implement a novel value function which incorporates the LLM’s predictions to guide decision-making while optimizing token efficiency. This approach reduces computational costs and enhances the agent’s learning process. Experimental results across three environments—MiniGrid, NetHack, and Crafter—demonstrate that our method significantly outperforms baseline HRL algorithms in terms of training speed and success rates. The imitation learning phase proves critical in enabling the agent to adapt quickly and perform efficiently, highlighting the potential of integrating LLMs into HRL for complex tasks.
BibTeX
@misc{yang2025leveraging,
  author = {Runhan Yang and Jieao Shi and Mengqi SU and Dongruo Zhou},
  title = {{Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning}},
  year = {2025},
  url = {https://openreview.net/forum?id=6y00rooi7i},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure