An Ergodic Measure for Active Learning From Equilibrium

Ian Abraham, Ahalya Prabhakar, Todd D. Murphey

TL;DR

This paper introduces KL-E$^3$, a novel method that allows robots to actively explore their environments to collect informative data without crashing or destabilizing. By combining a scalable KL-divergence ergodic measure with hybrid systems theory, the algorithm safely perturbs a known "safe" policy to gather high-quality training data, drastically improving the sample efficiency of downstream tasks like Bayesian optimization, dynamics modeling, and reinforcement learning.

What Problem Does This Paper Address?

In robot learning, acquiring good data is half the battle. To learn effectively, a robot needs to explore its state space to find informative measurements. However, standard exploration techniques (like adding random noise to actions) are often dynamically volatile—they can easily destabilize the robot, causing it to fall over or crash. On the flip side, overly safe learning methods restrict exploration so much that the robot fails to collect the diverse data needed to actually learn anything useful.

Furthermore, existing "ergodic exploration" methods—which guide a robot to spend time in regions proportional to how informative they are—rely on computationally heavy Fourier transforms. These traditional methods simply do not scale to the high-dimensional state spaces typical of modern robots. The authors tackle the challenge of enabling high-dimensional, intentional exploration that is mathematically guaranteed to keep the robot safe.

Methods

The authors propose KL-Ergodic Exploration from Equilibrium (KL-E$^3$), which relies on three core technical pillars:

  • Sample-based KL-Divergence Measure: Instead of the traditional Sobolev distance (which scales poorly due to Fourier basis functions), the authors use a sample-based Kullback-Leibler (KL) divergence to measure ergodicity. This metric calculates how well the time-averaged statistics of the robot's trajectory match a target spatial distribution (which represents where the most informative data lives).
  • Hybrid Systems Theory: To generate the actual control signals, the authors calculate the sensitivity of the KL-divergence measure to infinitesimally small changes in the control signal (known as the mode insertion gradient).
  • Exploration from Equilibrium: The algorithm assumes the robot has an approximate dynamics model and a baseline "equilibrium policy" (e.g., a basic LQR controller keeping a drone hovering). KL-E$^3$ calculates an optimal exploratory control signal that minimizes the ergodic measure while strictly bounding how far the robot deviates from the safe policy.

Datasets

Because this paper focuses on active data collection in physical space, it evaluates the method on simulated robotic environments rather than static datasets:

  • Cart Double Pendulum: Used for a constrained Bayesian Optimization task (finding a spatial maximum while keeping the pendulum upright).
  • Quadcopter: Used for learning a stochastic transition (dynamics) model while maintaining a stable hover.
  • CartPole & HalfCheetah: Standard continuous control benchmarks used to test off-policy reinforcement learning (Deep Deterministic Policy Gradient - DDPG).

Key Insights and Ideas

  • Safety and exploration can coexist via Lyapunov attractiveness: You don't need absolute stability at every millisecond to be safe. By allowing the robot to become temporarily unstable but mathematically guaranteeing it will eventually return to an equilibrium state (Lyapunov attractiveness), the robot can harvest highly informative data without catastrophic failure.
  • Ergodicity beats random noise: Standard RL relies on random action noise (like Ornstein-Uhlenbeck) to explore. Treating exploration as an ergodic problem—intentionally spending time in state-space regions proportional to their expected information gain—yields vastly superior training data.
  • Scalability through sampling: Approximating the time-averaged trajectory statistics with a Gaussian distribution and using a sample-based KL-divergence successfully bypasses the curse of dimensionality that plagued older ergodic exploration techniques.

Key Results

  • Bayesian Optimization: On the cart double pendulum, KL-E$^3$ was roughly 6 times more stable at the start of exploration compared to a naive LQR-Bayesian optimization baseline. It successfully found the objective maximum while keeping the pendulum upright.
  • Dynamics Modeling: For the quadcopter, KL-E$^3$ was significantly more energetically efficient. It achieved an average power loss of 0.16 compared to 0.97 for standard Ornstein-Uhlenbeck (OU) noise, while still learning a dynamics model accurate enough for downstream tracking control.
  • Reinforcement Learning: When integrated with DDPG, KL-E$^3$ dramatically accelerated learning. The CartPole inversion task was solved within 50 episodes, and the HalfCheetah began generating running gaits within 250 episodes—tasks where standard DDPG with random noise often failed or learned highly suboptimal behaviors.

Limitations and Future Work

Limitations: The method requires an approximate transition model of the robot and an existing equilibrium policy (like a basic stabilizing controller or a partially learned skill) to function. It cannot learn completely from scratch without this baseline safety net. Additionally, the KL-divergence approximation means the robot loses the ability to generate perfectly consistent ergodic exploratory movements, though the authors note this is usually sufficient for non-stationary targets.

Future Work: The authors plan to extend this framework to multi-agent robotic systems, allowing teams of collaborative robots to actively learn and explore their environments together safely.

Summary updated

Abstract

This article develops KL-ergodic exploration from equilibrium (KL-E 3 ), a method for robotic systems to integrate stability into actively generating informative measurements through ergodic exploration. Ergodic exploration enables robotic systems to indirectly sample from informative spatial distributions globally, avoiding local optima, and without the need to evaluate the derivatives of the distribution against the robot dynamics. Using a hybrid systems theory, we derive a controller that allows a robot to exploit equilibrium policies (i.e., policies that solve a task) while allowing the robot to explore and generate informative data using an ergodic measure that can extend to high-dimensional states. We show that our method is able to maintain Lyapunov attractiveness with respect to the equilibrium task while actively generating data for learning tasks such, as Bayesian optimization, model learning, and off-policy reinforcement learning. In each example, we show that our proposed method is capable of generating an informative distribution of data while synthesizing smooth control signals. We illustrate these examples using simulated systems and provide simplification of our method for real-time online learning in robotic systems. Note to Practitioners —Robotic systems need to adapt to sensor measurements and learn to exploit an understanding of the world around them such that they can truly begin to experiment in the real world. Standard learning methods do not have any restrictions on how the robot can explore and learn, making the robot dynamically volatile. Those that do are often too restrictive in terms of the stability of the robot, resulting in a lack of improved learning due to poor data collection. Applying our method would allow robotic systems to be able to adapt online without the need for human intervention. We show that considering both the dynamics of the robot and the statistics of where the robot has been, we are able to naturally encode where the robot needs to explore and collect measurements for efficient learning that is dynamically safe. With our method, we are able to effectively learn while being energetically efficient compared with state-of-the-art active learning methods. Our approach accomplishes such tasks in a single execution of the robotic system, i.e., the robot does not need human intervention to reset it. Future work will consider multiagent robotic systems that actively learn and explore in a team of collaborative robots.
BibTeX
@inproceedings{abraham2021ergodic,
  author = {Ian Abraham and Ahalya Prabhakar and Todd D. Murphey},
  title = {{An Ergodic Measure for Active Learning From Equilibrium}},
  booktitle = {IEEE Trans Autom. Sci. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=SylB0yuBXm},
}

Loading related content...

Figure