Abstract
We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by representation and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to an optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the planning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. Video: https://youtu.be/HQsQysUWOhg
Introduction
Deep reinforcement learning (RL) agents often struggle with generalization, typically memorizing optimal behaviors for specific tasks rather than learning to reason about their actions. When faced with new environments, these agents often require extensive retraining. While model-based RL offers a path toward adaptation by predicting future states, building accurate dynamic models for high-dimensional systems—such as a 3D humanoid robot—is notoriously difficult. Errors compound over time, and searching through massive action spaces leads to the "curse of dimensionality."
The DISH (DIStilling a Hierarchical policy) framework addresses these challenges by mimicking the biological hierarchy of human motor control. In this model, a high-level loop plans task-specific intentions in a simplified space, while a low-level, task-agnostic loop handles complex, high-frequency motor actions and reacts to immediate disturbances.
The DISH Framework
DISH alternates between representation learning and reinforcement learning to construct a two-tiered system that separates planning from execution.
- Low-level Control (Reinforcement Learning): A shared policy network is trained using Proximal Policy Optimization (PPO). It processes the robot's high-dimensional state and a low-dimensional "command" (a latent intention) to output high-dimensional motor actions. Operating at 30Hz, this policy ensures the robot remains stable while executing the intended motion.
- High-level Planning (Self-Supervised Learning): The framework collects trajectories from the low-level policy to train a Latent Variable Model (LVM) using an Adaptive Path-Integral Autoencoder (APIAE). This creates a compact, low-dimensional internal model of the robot's dynamics.
- Inference and Planning: To act, the agent employs a particle filter algorithm to simulate multiple future trajectories within the latent space. It evaluates these potential futures based on expected rewards and selects the optimal low-dimensional command for the low-level controller at a coarser time scale (1Hz or 4Hz).
Experimental Results
The authors evaluated DISH using a 197-DOF humanoid model. Initially, the agent learned basic locomotion (walking straight, turning) from the CMU Motion Capture database. Remarkably, the framework compressed the 197-dimensional state and 36-dimensional action space into just a 3D latent state and a 1D latent command.
Quantitative analysis showed that DISH achieved significantly lower RMS errors in trajectory-following tasks compared to baselines. While standard VAE-based models or non-hierarchical planners often resulted in the agent falling over or moving randomly, DISH maintained stability and precision.
Furthermore, DISH demonstrated impressive zero-shot generalization. The humanoid successfully navigated cluttered environments and escaped a "bug trap" (a U-shaped obstacle) without any additional RL fine-tuning. This ability to move away from a goal to eventually reach it highlights the power of planning in a structured latent space.
Conclusion
DISH demonstrates that complex, high-dimensional robotic control can be effectively managed by distilling a hierarchical policy that separates planning from execution. By leveraging low-dimensional latent spaces, the framework achieves both computational efficiency and robust generalization to unseen tasks. Future work may involve more complex training curricula to expand the internal model's coverage for even more sophisticated high-level commands.
Summary updated