Introduction
Bridging the gap between continuous physical dynamics and discrete symbolic planning remains a central challenge in artificial intelligence. While human cognition naturally decomposes complex tasks into discrete sub-goals (e.g., "walk to the door"), standard reinforcement learning (RL) agents often struggle to find such abstractions. Discretization methods typically face the curse of dimensionality, while deep RL models frequently fail in environments with sparse rewards. This paper presents the Hybrid Hierarchical Agent (HHA), a framework that autonomously discovers discrete "modes" within continuous environments to facilitate efficient planning and control.
The Hybrid Hierarchical Agent (HHA) Architecture
The HHA architecture integrates three distinct layers to manage the hierarchy of planning and execution:
- Recurrent Switching Linear Dynamical Systems (rSLDS): The core of the agent's "brain" is an rSLDS that decomposes complex, nonlinear dynamics into a set of simpler linear modes. The recurrence ensures that the agent's current state in the continuous space informs the probability of transitioning between these discrete modes.
- High-Level Discrete Planner: Operating on the discovered modes, a high-level planner utilizes Active Inference to select sub-goals. This framework prioritizes information gain, driving the agent to explore regions where the model's uncertainty is highest.
- Low-Level Control (LQR): Once a target mode is selected, a Linear-Quadratic Regulator (LQR) computes the precise continuous control inputs. Because the dynamics are linearized within each mode, the LQR provides an efficient and robust mechanism for reaching sub-goals.
By decoupling the discrete decision-making process from the continuous execution time, the HHA can perform abstract reasoning while maintaining high-precision control.
Experimental Evaluation: Continuous Mountain Car
The HHA was evaluated on the Continuous Mountain Car task, a classic benchmark characterized by sparse rewards. In this environment, an underpowered car must build momentum by swinging back and forth in a valley to reach a hilltop. The agent receives no reward until the goal is achieved, making it a difficult test for exploration strategies.
The performance of HHA was compared against several baselines, including Actor-Critic (AC), Soft Actor-Critic (SAC), and a model-based DQN (DQN-MBE).
Results and Discussion
The HHA demonstrated significant advantages in both learning speed and exploration efficiency. Unlike standard RL baselines like SAC and AC, which often failed to reach the goal within the first 20 episodes, the HHA consistently converged on a solution much faster.
Key findings include:
- Emergent Abstractions: The agent successfully "discovered" behavioral units, such as specific modes for gaining height versus accelerating toward the goal, without pre-defined maps.
- Curiosity-Driven Exploration: By operating in a small, discrete high-level state space, the agent could use information-theoretic measures to actively seek out unknown dynamics. This resulted in superior state-space coverage compared to random or non-curious agents.
- Robustness via Caching: The LQR solutions for transitions between modes were computed offline, creating a "cache" of control strategies that proved resilient to environmental noise.
Conclusion
The HHA framework successfully bridges the gap between continuous dynamics and discrete planning by leveraging rSLDS for mode discovery. While piecewise linear approximations may trade off some fine-grained optimality, the gains in interpretability and planning efficiency are substantial. Future work will focus on integrating formal safety constraints into the LQR layer and scaling the hierarchical discovery process to higher-dimensional robotic control tasks.
Summary updated