Inverse Optimal Control with Incomplete Observations

Wanxin Jin, Dana Kulic, Shaoshuai Mou, Sandra Hirche

TL;DR

This paper introduces a novel "recovery matrix" to learn an agent's objective function from incomplete trajectory data, a significant limitation of prior methods. This approach allows for learning even when only segments of an optimal trajectory are observed, paving the way for more robust and practical inverse optimal control (IOC) in real-world scenarios.

What Problem Does This Paper Address?

Inverse Optimal Control (IOC) aims to infer an agent's underlying objective function (e.g., cost or reward) by observing its behavior. A common assumption in IOC is that we have access to a complete trajectory of the agent's states and actions over its entire task horizon. However, in many real-world applications, obtaining such complete data is challenging due to sensor limitations, occlusions, or partial observability. This paper tackles the problem of learning objective functions when only incomplete trajectory observations are available – meaning we only see a segment of the full trajectory. This limitation hinders the application of existing IOC techniques in scenarios where full state-action history isn't recorded.

Methods

The core contribution of this paper is the introduction of the recovery matrix. This matrix establishes a relationship between an observed trajectory segment and the unknown weights of candidate features that define the objective function. The key idea is that the rank of this recovery matrix indicates whether the available data segment is sufficient to uniquely determine the objective function's weights.

The authors derive the recovery matrix based on the optimality conditions of optimal control problems. They show that this matrix can be computed iteratively, and its rank tends to increase with more data, suggesting that additional observations are beneficial. Based on the recovery matrix, they develop an IOC method that can learn from incomplete data and an incremental IOC algorithm. This algorithm automatically determines the minimal required observation length to achieve a successful estimate of the weights.

Datasets

The paper evaluates its proposed method on two systems:

  • Linear Quadratic Regulator (LQR) system: A standard benchmark for optimal control problems.
  • Simulated two-link robot arm: A non-linear system used to test the method's performance on more complex dynamics.

The authors do not introduce new datasets but rather use simulated trajectories generated from these systems.

Key Insights and Ideas

  • The Recovery Matrix as a Sufficiency Indicator: The rank of the recovery matrix is a direct indicator of whether the observed trajectory segment contains enough information to learn the objective function's weights. This is a powerful theoretical tool for understanding data requirements.
  • Handling Unseen Future Information: The recovery matrix implicitly accounts for the "unseen future" information that is missing from incomplete observations, a crucial aspect that allows for successful learning.
  • Incremental Learning for Minimal Observation: The proposed incremental IOC algorithm efficiently finds the shortest observation segment needed for successful learning, reducing data requirements and computational cost.
  • Robustness to Noise and Irrelevant Features: The method demonstrates robustness to noise and the ability to handle candidate feature sets that include irrelevant features, making it more practical.
  • Prior Work as a Special Case: Existing IOC methods that rely on full trajectories are shown to be a special case of the proposed recovery matrix approach.

Key Results

  • Successful Learning from Segments: The proposed method successfully learns objective function weights from incomplete trajectory segments on both LQR and robot arm systems.
  • Minimal Observation Length: The incremental IOC algorithm effectively determines the minimal required observation length, which varies depending on the data informativeness and the chosen features.
  • Comparison with Inverse-KKT: The recovery matrix method outperforms a standard inverse-KKT method when dealing with incomplete data, achieving successful estimates with significantly shorter observation segments.
  • Robustness to Noise: Increasing observation length helps compensate for data noise, leading to improved estimation accuracy.
  • Handling Irrelevant Features: The method can handle candidate feature sets containing irrelevant features, though this generally increases the minimal required observation length.

Limitations and Future Work

  • Assumption of Candidate Feature Set: The method assumes that the candidate feature set includes the true relevant features (Assumption 1). If this assumption is violated, the recovery matrix might not reach the required rank for successful estimation.
  • Near-Optimality of Demonstrations: While the method is shown to be robust to noise, the paper acknowledges that near-optimality of observed trajectories can affect rank evaluation.
  • Choice of Threshold γ: The parameter γ, used for rank evaluation, needs to be chosen appropriately. The paper suggests that a larger γ leads to longer observation lengths but can improve accuracy, and that its choice is not overly sensitive beyond a certain point.

Future work could explore scenarios where the candidate feature set does not necessarily contain all relevant features, or investigate more advanced strategies for automatically selecting relevant features. Further research into handling highly non-linear systems and real-world noisy data would also be valuable.

Summary updated

Abstract

This article develops a methodology that enables learning an objective function of an optimal control system from incomplete trajectory observations. The objective function is assumed to be a weighted sum of features (or basis functions) with unknown weights, and the observed data is a segment of a trajectory of system states and inputs. The proposed technique introduces the concept of the recovery matrix to establish the relationship between any available segment of the trajectory and the weights of given candidate features. The rank of the recovery matrix indicates whether a subset of relevant features can be found among the candidate features and the corresponding weights can be learned from the segment data. The recovery matrix can be obtained iteratively and its rank non-decreasing property shows that additional observations may contribute to the objective learning. Based on the recovery matrix, a method for using incomplete trajectory observations to learn the weights of selected features is established, and an incremental inverse optimal control algorithm is developed by automatically finding the minimal required observation. The effectiveness of the proposed method is demonstrated on a linear quadratic regulator system and a simulated robot manipulator.
BibTeX
@article{jin2021inverse,
  author = {Wanxin Jin and Dana Kulic and Shaoshuai Mou and Sandra Hirche},
  title = {{Inverse Optimal Control with Incomplete Observations}},
  journal = {Int. J. Robotics Res.},
  year = {2021},
  doi = {10.1177/0278364921996384},
}

Loading related content...

Figure