Constrained stochastic optimal control with learned importance sampling: A path integral approach

Jan Carius, René Ranftl, Farbod Farshidian, Marco HutterInt. J. Robotics Res 2022 · DOI

TL;DR

This paper introduces a novel algorithm for controlling complex, high-dimensional robotic systems in uncertain environments. By extending path integral stochastic optimal control (PISOC) with constraint handling and learned importance sampling, the method enables robust, real-time control even when system dynamics are only available through physics simulators.

What Problem Does This Paper Address?

Modern robots are increasingly expected to operate autonomously in dynamic and partially unknown environments. Designing controllers that are both robust to unforeseen situations and capable of safe operation in complex scenarios is a significant challenge. Traditional optimal control (OC) methods often struggle with complex, non-linear dynamics and require accurate, analytical models, which are difficult to obtain. Reinforcement learning (RL) approaches, while more flexible in modeling, can be sample-inefficient and struggle with hard constraints. This paper aims to bridge this gap by combining the strengths of both: the modeling flexibility of sampling-based methods with the constraint satisfaction and reactivity of OC.

Methods

The core of the proposed method is an extension of Path Integral Stochastic Optimal Control (PISOC). Key innovations include:

  • Constraint Handling: The PISOC framework is extended to explicitly handle state-input equality constraints. This is achieved through a projection scheme that reduces the dimensionality of the sampling space and ensures that all sampled trajectories satisfy the constraints.
  • Importance Sampling: To address the high sampling complexity inherent in high-dimensional systems, the authors employ importance sampling. This involves using an "ancillary controller" (which can be a pre-existing controller, a fallback policy, or a learned policy) to guide the sampling distribution towards more informative regions of the state-action space.
  • Learned Importance Sampling Policy: When a suitable ancillary controller is not readily available, the algorithm can learn its own importance sampling policy. This is done using a machine learning approach (specifically, a feed-forward neural network) that is trained online using data from the PISOC rollouts. This learned policy acts as a "smart sampler," guiding the exploration towards promising solutions.
  • Automatic Temperature Tuning: A mechanism to automatically adjust the "temperature" parameter (which controls the balance between exploration and exploitation) is introduced to improve robustness and reduce manual tuning.
  • Receding Horizon Control: The algorithm is implemented within a receding horizon framework, allowing for continuous replanning and adaptation to changing conditions.

Datasets

Not discussed in the paper. The paper relies on simulated dynamics from physics engines (RaiSim) and hardware experiments.

Key Insights and Ideas

  • Constraints as a Powerful Tool for Sampling Efficiency: The paper demonstrates that explicitly incorporating constraints, even simple ones, can drastically reduce the sampling complexity and improve the reliability of the PISOC algorithm, especially for unstable and high-dimensional systems. This is a crucial insight for making sampling-based methods practical.
  • Learned Samplers as a Bridge to Robustness: The ability to learn an importance sampling policy "on the fly" is a significant contribution. This allows the system to adapt to challenging environments (like uneven terrain) where pre-defined controllers might fail due to modeling inaccuracies or lack of adaptability.
  • PISOC as a Flexible Framework: The path integral formulation proves to be a versatile foundation. It naturally accommodates complex dynamics from simulators, allows for the integration of existing controllers as samplers, and can be extended with learned components.
  • Exploration is Key, Even with Learned Policies: Even with learned importance samplers, the inherent stochasticity of the PISOC framework is vital for exploring unknown states and avoiding local minima, particularly in complex scenarios like navigating stepping stones.

Key Results

  • Successful Control of Complex Robots: The algorithm was successfully demonstrated on three robotic systems: a 2D robot arm, a ball-balancing robot (Ballbot), and a quadrupedal robot (ANYmal).
  • Improved Sampling Efficiency: Importance sampling, both with pre-existing controllers and learned policies, significantly improved the effective sample size and reduced the variance of path costs compared to naive sampling.
  • Real-time Performance: The algorithm achieved real-time performance on a quadrupedal robot, with an update rate of approximately 15 Hz, enabling its deployment on physical hardware.
  • Robustness to Unseen Environments: The learned importance sampling policy showed adaptability to unseen terrain variations (e.g., inclined ground for Ballbot), outperforming fixed ancillary controllers.
  • Constraint Satisfaction: The method reliably satisfied state-input equality constraints, as demonstrated in the ANYmal walking experiments.

Limitations and Future Work

  • Simulation-to-Reality Gap: While the paper mentions the challenge of simulation-reality mismatch, it doesn't deeply explore its impact or propose specific solutions beyond the inherent robustness of the stochastic approach.
  • GPU Parallelization: The current implementation relies on CPU parallelization. Transitioning to GPU parallelization is suggested as a future direction to further increase speed and potentially enable more complex simulators.
  • Inequality Constraints: The current work focuses on equality constraints. Extending the method to handle inequality constraints (e.g., joint limits) is mentioned as a future research direction.
  • Perception Integration: The current approach is perception-less. Integrating learned samplers with perception sensors to recognize footholds and adapt to dynamic environments is a promising avenue.
  • Discrete Decisions: Exploring the application of sampling-based methods to discrete decisions (e.g., switching between different gaits) is suggested, though it currently faces challenges with combinatorial explosion.
  • Online Learning Overfitting: The authors acknowledge the potential for overfitting in the learned importance sampling policy when the system remains in a stationary state for extended periods, highlighting the need for replay mechanisms.

Summary updated

BibTeX
@article{carius2022constrained,
  author = {Jan Carius and René Ranftl and Farbod Farshidian and Marco Hutter},
  title = {{Constrained stochastic optimal control with learned importance sampling: A path integral approach}},
  journal = {Int. J. Robotics Res.},
  year = {2022},
  doi = {10.1177/02783649211047890},
}
Figure