Meta-active Learning in Probabilistically-Safe Optimization

Mariah L. Schrum, Mark J. Connolly, Eric R. Cole, Mihir Ghetiya, Robert Gross, Matthew C. GombolayIEEE RA-L 2020 · DOI · OpenReview

TL;DR

This paper introduces "meta-active learning," a novel approach that combines meta-learning with chance-constrained optimization to safely and efficiently learn the dynamics of complex systems. By meta-learning an acquisition function using an LSTM and integrating it into a mixed-integer linear program (MILP), the system can intelligently balance exploring for information with maintaining safety, achieving state-of-the-art results in both high-dimensional control and medical applications.

What Problem Does This Paper Address?

Controlling safety-critical systems with unknown or changing dynamics, such as deep brain stimulation devices for neurological disorders or autonomous aircraft, presents a significant challenge. These systems require learning their behavior in real-time to optimize performance, but doing so safely is paramount. Traditional active learning methods can be sample-inefficient and struggle with high-dimensional state spaces, while existing safety mechanisms often sacrifice valuable information gathering. The core problem is how to efficiently and safely learn system dynamics and optimal configurations in complex, real-world scenarios where exploration is necessary but failure can have severe consequences.

Methods

The paper proposes a "meta-active learning" framework that addresses this challenge through a novel integration of meta-learning and chance-constrained optimization.

  • Meta-Learning the Acquisition Function: The system meta-learns an "acquisition function" offline. This function's purpose is to predict the expected information gain from querying a particular data point (i.e., taking a specific action in a given state). Crucially, this acquisition function is implemented as a Long-Short Term Memory (LSTM) network. The LSTM is designed to encode the history of state-action pairs, allowing it to learn sophisticated sampling strategies based on past experiences across various tasks.
  • Chance-Constrained Optimization via MILP: The learned acquisition function is then integrated into a Mixed-Integer Linear Program (MILP). This MILP acts as the policy, determining the optimal action trajectory. The integration allows the system to explicitly trade off two objectives:
  • Expected Information Gain: This is provided by the meta-learned LSTM acquisition function.
  • Likelihood of Safe Control: This is enforced through chance constraints, ensuring the system remains within a defined "sphere of safety" with a high probability.
  • Probabilistic Safety: The safety aspect is handled by defining a "sphere of safety" and using probability constraints derived from assuming Gaussian model error. This allows the system to deviate from the safe region for exploration, as long as it can guarantee a high probability of returning to safety within a specified time horizon.
  • Online Adaptation: The dynamics model of the system is retrained online as new state-action pairs are collected, allowing the system to adapt to changing or altered dynamics.

Datasets

The paper evaluates its approach on two primary experimental domains:

  • Deep Brain Stimulation (DBS) for Rat Memory Function: This domain simulates learning optimal parameter settings for a deep brain stimulation device in rats to improve memory function. The data is generated by fitting Gaussian Processes to experimental data, mimicking the variability across individual rat brains. The objective is to maximize discrimination area (a measure of memory function) while avoiding negative side effects (seizures).
  • High-Dimensional Domain (Recovering Dynamics of a Damaged Aircraft): This experiment uses a simulated damaged aircraft in a virtual environment (Flightgear) to test the system's ability to learn altered dynamics and maintain safe flight. The simulation is based on theoretical damage models and requires learning the differences in dynamics to prevent unrecoverable states like a spin or crash.

Key Insights and Ideas

  • Meta-learning for Intelligent Exploration: By meta-learning the acquisition function, the system gains the ability to learn how to learn effectively across a distribution of tasks. This allows it to develop more robust and efficient exploration strategies than traditional, task-specific active learning methods.
  • Bridging Deep Learning and Optimization: The seamless integration of a deep learning-based acquisition function (LSTM) into a MILP framework is a key innovation. This allows the system to leverage the feature learning capabilities of deep networks within a mathematically rigorous optimization and safety guarantee framework.
  • Explicit Safety-Information Trade-off: The chance-constrained MILP formulation provides a principled way to balance the desire for information gain with the critical need for safety. This explicit trade-off mechanism is crucial for safety-critical applications.
  • Generalizability to High Dimensions: The framework demonstrates strong performance in a high-dimensional domain (damaged aircraft), suggesting its potential for complex real-world robotic systems where dimensionality is a significant challenge.

Key Results

The meta-active learning approach demonstrated state-of-the-art performance across both experimental domains:

  • Information Gain:
    • In the high-dimensional aircraft domain, the approach achieved a 46% increase in information gain compared to baselines.
  • In the rat DBS domain, it achieved a 58% increase in information gain over baseline methods.
  • Computation Time:
    • The approach achieved a 20% speedup in computation time compared to active learning baselines in the high-dimensional domain.
  • It was significantly faster than other meta-learning approaches, being 68x faster than LAL and 61x faster than Meta BO in the ADMETS domain.
  • Safety Guarantees:
    • In the aircraft domain, the system achieved an 87% probability of returning to the sphere of safety while greatly increasing information gain.
  • In the rat DBS domain, it achieved a 97% likelihood of terminating in a safe state (avoiding seizures) while only losing 15% of information gain compared to unconstrained learning.

Limitations and Future Work

  • Bias in Acquisition Function: The paper acknowledges that the learned acquisition function is biased by the distribution of the offline training data. Diversifying this training data is suggested as a way to mitigate this bias.
  • Assumption of Gaussian Model Error: The linearization of probability constraints for the MILP relies on the assumption of Gaussian model error. Relaxing this assumption or exploring alternative safety constraint formulations could be future work.
  • Future Directions: The authors suggest that their framework has broad implications for healthcare and robotics, particularly in scenarios requiring learning from limited data and ensuring safety. Future work could explore more complex dynamics, different safety definitions, and applications in other safety-critical domains.

Summary updated

Abstract

Learning to control a safety-critical system with latent dynamics (e.g. for deep brain stimulation) requires taking calculated risks to gain information as efficiently as possible. To address this problem, we present a probabilistically-safe, meta-active learning approach to efficiently learn system dynamics and optimal configurations. We cast this problem as meta-learning an acquisition function, which is represented by a Long-Short Term Memory Network (LSTM) encoding sampling history. This acquisition function is meta-learned offline to learn high quality sampling strategies. We employ a mixed-integer linear program as our policy with the final, linearized layers of our LSTM acquisition function directly encoded into the objective to trade off expected information gain (e.g., improvement in the accuracy of the model of system dynamics) with the likelihood of safe control. We set a new state-of-the-art in active learning for control of a high-dimensional system with altered dynamics (i.e., a damaged aircraft), achieving a 46% increase in information gain and a 20% speedup in computation time over baselines. Furthermore, we demonstrate our system's ability to learn the optimal parameter settings for deep brain stimulation in a rat's brain while avoiding unwanted side effects (i.e., triggering seizures), outperforming prior state-of-the-art approaches with a 58% increase in information gain. Additionally, our algorithm achieves a 97% likelihood of terminating in a safe state while losing only 15% of information gain.
BibTeX
@article{schrum2020meta,
  author = {Mariah L. Schrum and Mark J. Connolly and Eric R. Cole and Mihir Ghetiya and Robert Gross and Matthew C. Gombolay},
  title = {{Meta-active Learning in Probabilistically-Safe Optimization}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2020},
  doi = {10.1109/lra.2022.3193497},
}
Figure