Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Takamitsu Matsubara, Vicenç Gómez, Hilbert J. KappenUAI 2014

TL;DR

This paper tackles the challenge of applying efficient "Kullback Leibler (KL) control" to complex, continuous-state robotic systems. They propose a novel approach that embeds the KL control problem within a probabilistic graphical model, using latent discrete variables to represent the system's state. This allows for efficient computation of near-optimal control policies for high-dimensional problems where traditional methods fail.

What Problem Does This Paper Address?

Stochastic optimal control, particularly the "Kullback Leibler (KL) control" framework, offers an efficient way to compute optimal control policies by solving a principal eigenvector problem. This is highly desirable for its computational advantages. However, a major limitation arises when dealing with continuous-state and continuous-action systems, such as those found in robotics. The "curse of dimensionality" makes direct application infeasible due to the combinatorial explosion of states when discretizing. While some methods approximate solutions in continuous spaces, they often involve computationally demanding quadratic programming or require knowledge of the system's dynamics, which is frequently unavailable in real-world scenarios. This paper aims to bridge this gap by enabling efficient KL control for continuous systems without requiring a precise model of their dynamics.

Methods

The core innovation is to combine KL control with probabilistic graphical models. The approach involves three main steps:

  1. Learning a Probabilistic Graphical Model: The system's observed continuous states are mapped to a lower-dimensional, discrete "latent" representation. This is achieved by learning a model (either a standard Hidden Markov Model - HMM, or a Factorial HMM - FHMM for higher dimensions) from data collected during exploratory control.
  2. Solving the KL Control Problem in Latent Space: The KL control problem is then solved in this learned discrete latent space. This leverages the computational efficiency of KL control for discrete systems.
  3. Computing Control in Observed Space: Finally, the computed latent-space control is translated back to control commands for the actual continuous-state system. This involves a filtering step and a mechanism to reduce the "difference" between uncontrolled and optimal behaviors.

Two specific realizations are presented:

  • HMM-based approach: Uses standard HMMs for exact control computation in the latent space, but is limited to low-dimensional systems.
  • FHMM-based approach: Employs Factorial HMMs, which are scalable to higher-dimensional problems by assuming factorized dynamics. This approach offers approximate control computation.

Datasets

The paper uses simulated robot control tasks for evaluation. These include:

  • Pendulum swing-up with limited torque.
  • Robot arm control with an obstacle.
  • Multi-DOF redundant arm reaching task.

No new datasets are introduced; rather, these are standard benchmarks for evaluating control algorithms.

Key Insights and Ideas

  • Latent Space for Efficiency: The key insight is that by learning a discrete latent representation of a continuous system, we can leverage the computational efficiency of KL control designed for discrete states. This bypasses the curse of dimensionality.
  • Model-Free Learning: The approach learns the probabilistic graphical model from data, meaning it doesn't require an explicit, pre-defined model of the robot's dynamics, making it applicable to real-world scenarios.
  • FHMMs for Scalability: The use of Factorial HMMs is crucial for handling high-dimensional systems, as it allows for a more efficient representation and computation compared to standard HMMs when the state space grows.
  • Trade-off between Exactness and Scalability: The paper highlights a trade-off: HMMs provide exact latent control but are limited in dimensionality, while FHMMs offer approximate control but scale to much higher dimensions.

Key Results

  • Feasibility Demonstrated: The proposed method was successfully applied to three benchmark robot control tasks, demonstrating its feasibility.
  • Scalability with FHMMs: The FHMM-based approach showed significant scalability, successfully handling problems with up to 25 degrees of freedom, where exact KL control methods become intractable.
  • Computational Efficiency: For the multi-DOF reaching task, the AVKL (Approximate Variational KL minimization) method demonstrated a roughly linear increase in CPU time with the number of degrees of freedom, contrasting sharply with the exponential scaling of exact KL control.
  • Near-Optimal Performance: The approximate methods (VKL and AVKL) achieved control errors comparable to exact KL minimization where it was feasible.
  • Comparison to Value Iteration: For the robot arm with an obstacle, the proposed HMM-based method was significantly faster (orders of magnitude) and required less discretization than standard value iteration, while achieving optimal energy efficiency.

Limitations and Future Work

  • Model Quality Dependence: The performance of the learned controller is strongly dependent on the quality of the learned probabilistic graphical model. This, in turn, depends on the choice of exploration noise and initialization of model parameters.
  • Learning Efficiency: Current work is focused on developing alternative learning methods that can more efficiently sample interesting regions of the state space and exploit the ergodic nature of the problems.
  • Complex Scenarios: The authors are considering extensions to more complex scenarios, implying that current work is focused on relatively simpler system dynamics.
  • Parameter Tuning: The gain matrix K in the observed space control computation is treated as a free parameter, suggesting that optimal tuning might be an area for further research.

Summary updated

BibTeX
@inproceedings{matsubara2014latent,
  author = {Takamitsu Matsubara and Vicenç Gómez and Hilbert J. Kappen},
  title = {{Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2014},
}
Figure