Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning

Jung-Su Ha, Young-Jin Park, Hyeok-Joo Chae, Soon-Seo Park, Han-Lim Choi

TL;DR

The paper introduces DISH, a hierarchical reinforcement learning framework that separates high-level planning from low-level motor control using low-dimensional latent spaces. By distilling a shared low-level control policy and a self-supervised internal dynamics model, a complex 197-DOF humanoid can learn basic walking skills and immediately adapt to unseen navigation tasks—like escaping a bug trap—without any additional training.

What Problem Does This Paper Address?

Deep reinforcement learning (RL) agents often memorize optimal behaviors for specific tasks rather than learning to reason about their actions. When faced with a new task or environment, they typically require extensive retraining. While model-based RL and planning offer a way to adapt by predicting the future, building accurate dynamic models for high-dimensional systems (like a 3D humanoid robot) is notoriously difficult. Errors compound over time, and searching through a massive action space suffers from the "curse of dimensionality."

The authors aim to bridge this gap by mimicking the human motor control system. In humans, a high-level loop plans task-specific intentions in a simplified space, while a low-level, task-agnostic loop (like the spinal cord) handles complex, high-frequency muscle movements and reacts to immediate disturbances. The paper attempts to replicate this biological hierarchy in an RL agent.

Methods

The proposed framework, DISH (DIStilling a Hierarchical policy), alternates between representation learning and reinforcement learning to build its two-tiered system:

  1. Low-level Control (Reinforcement Learning): A shared policy network is trained using Proximal Policy Optimization (PPO). It takes the robot's high-dimensional state and a low-dimensional "command" (a latent intention) to output high-dimensional motor actions. This policy operates at a high frequency (30Hz) to stabilize the robot.
  2. High-level Planning (Self-Supervised Learning): The framework collects trajectories generated by the low-level policy and trains a Latent Variable Model (LVM) using an Adaptive Path-Integral Autoencoder (APIAE). This creates a compact, low-dimensional internal model of the robot's dynamics.
  3. Inference/Planning: To act, the agent uses a particle filter algorithm to simulate multiple future trajectories in the latent space. It evaluates these futures based on expected rewards and selects the optimal low-dimensional command to pass to the low-level controller at a coarser time scale (1Hz or 4Hz).

Datasets

  • CMU Motion Capture (mocap) database: The authors used three locomotion clips (going straight, turning left, turning right) as reference motions for the initial imitation learning phase.
  • Custom Bullet Physics Environments: The learned policies were evaluated in simulated custom trajectory-following tasks and cluttered navigation tasks (including obstacle courses and a "bug trap" scenario).

Key Insights and Ideas

  • Control-Inference Duality: By framing multi-task RL as a probabilistic inference problem, the authors elegantly combine generative model learning (for the world model) and RL (for the control policy) into a single cohesive framework.
  • Latent Space is Enough for Planning: You don't need to plan every joint angle to navigate a room. The framework successfully compressed a 197-dimensional state and 36-dimensional action space into a 3D latent state and a 1D latent command, which proved entirely sufficient for complex navigation.
  • Zero-Shot Generalization: Because the high-level planner operates entirely in the latent space and the low-level controller is robust to disturbances, the agent can solve entirely new tasks (like navigating a cluttered maze) without any additional RL fine-tuning.

Key Results

  • Ablation Superiority: The authors compared DISH to baselines lacking either the hierarchical structure or the latent variable model. Only DISH successfully learned to make the humanoid walk and follow planned trajectories. The baselines either failed to walk entirely (due to the instability of high-dimensional planning) or moved randomly.
  • Quantitative Accuracy: In trajectory following tasks, DISH achieved the lowest RMS errors between the planned and executed trajectories (e.g., an error of 0.0930 for turning left, whereas baselines failed to even record a true trajectory because the agent fell over).
  • Complex Navigation: The DISH-equipped humanoid successfully navigated a cluttered environment and escaped a "bug trap" (a U-shaped obstacle requiring the agent to temporarily move away from the goal to escape). Standard greedy algorithms without planning fail this completely.

Limitations and Future Work

The authors note that for more challenging problems requiring highly complex motions, the current training regime might struggle to find a stable learning landscape. They suggest that a more carefully designed curriculum—gradually increasing task difficulty, using play data, or leveraging intrinsic motivation—would be necessary to ensure the internal model covers a wide enough state space to execute highly complicated high-level commands.

Summary updated

Abstract

We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by representation and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to an optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the planning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments.
BibTeX
@inproceedings{ha2021distilling,
  author = {Jung-Su Ha and Young-Jin Park and Hyeok-Joo Chae and Soon-Seo Park and Han-Lim Choi},
  title = {{Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=ktTP1IBNHYF},
}

Loading related content...

Figure