An Ergodic Measure for Active Learning From Equilibrium

Ian Abraham, Ahalya Prabhakar, Todd D. Murphey

Abstract

This paper develops KL-Ergodic Exploration from Equilibrium ($\text{KL-E}^3$), a method for robotic systems to integrate stability into actively generating informative measurements through ergodic exploration. Ergodic exploration enables robotic systems to indirectly sample from informative spatial distributions globally, avoiding local optima, and without the need to evaluate the derivatives of the distribution against the robot dynamics. Using hybrid systems theory, we derive a controller that allows a robot to exploit equilibrium policies while allowing the robot to explore and generate informative data using an ergodic measure that can extend to high-dimensional states. We show that our method is able to maintain Lyapunov attractiveness with respect to the equilibrium task while actively generating data for learning tasks such as Bayesian optimization, model learning, and off-policy reinforcement learning. In each example, we show that our proposed method is capable of generating an informative distribution of data while synthesizing smooth control signals.

Introduction: The Challenge of Safe Exploration

In robot learning, acquiring high-quality data is a fundamental challenge. To learn effectively, a robot must explore its state space to find informative measurements. However, standard exploration techniques, such as adding random noise to actions, are often dynamically volatile and can easily destabilize the robot, leading to crashes. Conversely, overly conservative safety methods may restrict exploration so severely that the robot fails to collect the diverse data necessary for learning.

Furthermore, traditional "ergodic exploration" methods—which guide a robot to spend time in regions proportional to their information density—rely on computationally expensive Fourier transforms. These methods do not scale well to the high-dimensional state spaces of modern robotics. This paper introduces a solution that enables high-dimensional, intentional exploration while mathematically guaranteeing stability.

Methodology: KL-Ergodic Exploration from Equilibrium

The authors propose KL-Ergodic Exploration from Equilibrium (KL-E$^3$), which integrates three technical pillars to achieve safe, active learning:

  1. Sample-based KL-Divergence Measure: To overcome the scaling issues of traditional Sobolev distances, the authors utilize a sample-based Kullback-Leibler (KL) divergence. This metric assesses how closely the time-averaged statistics of the robot's trajectory match a target spatial distribution representing informative regions.
  2. Hybrid Systems Theory: Control signals are generated by calculating the mode insertion gradient, which measures the sensitivity of the KL-divergence to infinitesimal changes in the control signal.
  3. Exploration from Equilibrium: The algorithm leverages an approximate dynamics model and a baseline "equilibrium policy" (e.g., a stabilizing LQR controller). KL-E$^3$ calculates exploratory signals that minimize the ergodic measure while bounding deviations from the safe policy, ensuring Lyapunov attractiveness.
The KL-E$^3$ framework matches trajectory statistics to a target information distribution.

Experimental Evaluation and Results

The KL-E$^3$ framework was evaluated across several simulated benchmarks, including a Cart Double Pendulum, a Quadcopter, and standard reinforcement learning (RL) tasks like CartPole and HalfCheetah.

Bayesian Optimization

In the Cart Double Pendulum task, the goal was to find a spatial maximum while keeping the pendulum upright. KL-E$^3$ proved to be roughly 6 times more stable during the initial exploration phase compared to a naive LQR-Bayesian optimization baseline. It successfully identified the objective maximum while strictly adhering to the dynamic constraints of the system.

Active learning on a cart double pendulum: the system identifies the objective maximum while maintaining stability over time.

Dynamics Modeling and Reinforcement Learning

For quadcopter dynamics modeling, KL-E$^3$ demonstrated superior energy efficiency, with an average power loss of 0.16 versus 0.97 for standard Ornstein-Uhlenbeck noise. In reinforcement learning (DDPG), the method significantly accelerated convergence. The CartPole inversion task was solved within 50 episodes, and the HalfCheetah began generating running gaits within 250 episodes, significantly outperforming standard random noise exploration which often failed or learned suboptimal behaviors.

Discussion and Future Work

A key insight of this work is that safety and exploration can coexist through Lyapunov attractiveness; absolute stability at every moment is not required as long as the system is guaranteed to return to equilibrium. By treating exploration as an ergodic problem rather than a random one, robots can generate vastly superior training data.

Limitations: The method currently requires an approximate transition model and an existing equilibrium policy. It is not designed for learning entirely from scratch without a baseline safety net. Additionally, the KL-divergence approximation means the robot may lose some consistency in perfectly ergodic movements, though this is generally sufficient for non-stationary targets.

Future Work: The authors intend to extend this framework to multi-agent systems, enabling collaborative teams of robots to explore and learn in shared environments safely.

Summary updated

Abstract

This paper develops KL-Ergodic Exploration from Equilibrium ($\text{KL-E}^3$), a method for robotic systems to integrate stability into actively generating informative measurements through ergodic exploration. Ergodic exploration enables robotic systems to indirectly sample from informative spatial distributions globally, avoiding local optima, and without the need to evaluate the derivatives of the distribution against the robot dynamics. Using hybrid systems theory, we derive a controller that allows a robot to exploit equilibrium policies (i.e., policies that solve a task) while allowing the robot to explore and generate informative data using an ergodic measure that can extend to high-dimensional states. We show that our method is able to maintain Lyapunov attractiveness with respect to the equilibrium task while actively generating data for learning tasks such, as Bayesian optimization, model learning, and off-policy reinforcement learning. In each example, we show that our proposed method is capable of generating an informative distribution of data while synthesizing smooth control signals. We illustrate these examples using simulated systems and provide simplification of our method for real-time online learning in robotic systems.
BibTeX
@inproceedings{abraham2021ergodic,
  author = {Ian Abraham and Ahalya Prabhakar and Todd D. Murphey},
  title = {{An Ergodic Measure for Active Learning From Equilibrium}},
  booktitle = {IEEE Trans Autom. Sci. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=SylB0yuBXm},
}
Figure