Abstract
This paper develops KL-Ergodic Exploration from Equilibrium ($\text{KL-E}^3$), a method for robotic systems to integrate stability into actively generating informative measurements through ergodic exploration. Ergodic exploration enables robotic systems to indirectly sample from informative spatial distributions globally, avoiding local optima, and without the need to evaluate the derivatives of the distribution against the robot dynamics. Using hybrid systems theory, we derive a controller that allows a robot to exploit equilibrium policies while allowing the robot to explore and generate informative data using an ergodic measure that can extend to high-dimensional states. We show that our method is able to maintain Lyapunov attractiveness with respect to the equilibrium task while actively generating data for learning tasks such as Bayesian optimization, model learning, and off-policy reinforcement learning. In each example, we show that our proposed method is capable of generating an informative distribution of data while synthesizing smooth control signals.
Introduction: The Challenge of Safe Exploration
In robot learning, acquiring high-quality data is a fundamental challenge. To learn effectively, a robot must explore its state space to find informative measurements. However, standard exploration techniques, such as adding random noise to actions, are often dynamically volatile and can easily destabilize the robot, leading to crashes. Conversely, overly conservative safety methods may restrict exploration so severely that the robot fails to collect the diverse data necessary for learning.
Furthermore, traditional "ergodic exploration" methods—which guide a robot to spend time in regions proportional to their information density—rely on computationally expensive Fourier transforms. These methods do not scale well to the high-dimensional state spaces of modern robotics. This paper introduces a solution that enables high-dimensional, intentional exploration while mathematically guaranteeing stability.
Methodology: KL-Ergodic Exploration from Equilibrium
The authors propose KL-Ergodic Exploration from Equilibrium (KL-E$^3$), which integrates three technical pillars to achieve safe, active learning:
- Sample-based KL-Divergence Measure: To overcome the scaling issues of traditional Sobolev distances, the authors utilize a sample-based Kullback-Leibler (KL) divergence. This metric assesses how closely the time-averaged statistics of the robot's trajectory match a target spatial distribution representing informative regions.
- Hybrid Systems Theory: Control signals are generated by calculating the mode insertion gradient, which measures the sensitivity of the KL-divergence to infinitesimal changes in the control signal.
- Exploration from Equilibrium: The algorithm leverages an approximate dynamics model and a baseline "equilibrium policy" (e.g., a stabilizing LQR controller). KL-E$^3$ calculates exploratory signals that minimize the ergodic measure while bounding deviations from the safe policy, ensuring Lyapunov attractiveness.
Experimental Evaluation and Results
The KL-E$^3$ framework was evaluated across several simulated benchmarks, including a Cart Double Pendulum, a Quadcopter, and standard reinforcement learning (RL) tasks like CartPole and HalfCheetah.
Bayesian Optimization
In the Cart Double Pendulum task, the goal was to find a spatial maximum while keeping the pendulum upright. KL-E$^3$ proved to be roughly 6 times more stable during the initial exploration phase compared to a naive LQR-Bayesian optimization baseline. It successfully identified the objective maximum while strictly adhering to the dynamic constraints of the system.
Dynamics Modeling and Reinforcement Learning
For quadcopter dynamics modeling, KL-E$^3$ demonstrated superior energy efficiency, with an average power loss of 0.16 versus 0.97 for standard Ornstein-Uhlenbeck noise. In reinforcement learning (DDPG), the method significantly accelerated convergence. The CartPole inversion task was solved within 50 episodes, and the HalfCheetah began generating running gaits within 250 episodes, significantly outperforming standard random noise exploration which often failed or learned suboptimal behaviors.
Discussion and Future Work
A key insight of this work is that safety and exploration can coexist through Lyapunov attractiveness; absolute stability at every moment is not required as long as the system is guaranteed to return to equilibrium. By treating exploration as an ergodic problem rather than a random one, robots can generate vastly superior training data.
Limitations: The method currently requires an approximate transition model and an existing equilibrium policy. It is not designed for learning entirely from scratch without a baseline safety net. Additionally, the KL-divergence approximation means the robot may lose some consistency in perfectly ergodic movements, though this is generally sufficient for non-stationary targets.
Future Work: The authors intend to extend this framework to multi-agent systems, enabling collaborative teams of robots to explore and learn in shared environments safely.
Summary updated