Abstract
Inverse optimal control (IOC) is a powerful approach for learning robotic controllers from demonstration that estimates a cost function which rationalizes demonstrated control trajectories. Unfortunately, its applicability is difficult in settings where optimal control can only be solved approximately. While local IOC approaches have been shown to successfully learn cost functions in such settings, they rely on the availability of good reference trajectories, which might not be available at test time. We address the problem of using IOC in these computationally challenging control tasks by using a graph-based discretization of the trajectory space. Our approach projects continuous demonstrations onto this discrete graph, where a cost function can be tractably learned via IOC. Discrete control trajectories from the graph are then projected back to the original space and locally optimized using the learned cost function. We demonstrate the effectiveness of the approach with experiments conducted on two 7-degree of freedom robotic arms.
Introduction
Teaching a robot to move "naturally" or according to human preferences is a significant challenge in robotics. While simple constraints like obstacle avoidance are easy to program, complex preferences—such as keeping a container upright or avoiding specific sensitive areas like a laptop—are difficult to define mathematically.
Inverse Optimal Control (IOC) addresses this by "reverse engineering" the underlying cost function from human demonstrations. However, for high-dimensional systems like 7-DOF robotic arms, traditional IOC faces a computational bottleneck: global optimization is often intractable, and local optimization is prone to poor local minima without high-quality initializations.
Methodology: The Discrete-to-Continuous Pipeline
To overcome these challenges, we propose a hybrid framework that combines the global search capabilities of discrete graphs with the refinement of local continuous optimization.
1. Graph-Based Discretization
Instead of operating in the infinite continuous configuration space, we generate a sparse graph in the robot's configuration space. Points are sampled more densely in the middle of the trajectory and more sparsely near the endpoints to better capture the variance in human motion. This discretization transforms a complex continuous optimization problem into a tractable graph search.
2. Discrete MaxEnt IOC
Human demonstrations, collected via kinesthetic teaching, are projected onto this graph. We then apply Maximum Entropy (MaxEnt) IOC to learn a cost function. This probabilistic framework is "unbiased," learning a distribution over many feasible paths rather than overfitting to a single demonstration. Features used for learning include distance to obstacles, end-effector orientation (to maintain "uprightness"), and proximity to specific objects like laptops.
3. Local Trajectory Optimization (LTO)
Because graph-based paths are inherently discrete and "jagged," they are unsuitable for direct execution. We use the learned cost function to perform local gradient-descent optimization (inspired by the CHOMP algorithm), smoothing the discrete path into a continuous trajectory that remains faithful to the learned preferences.
Experimental Evaluation
We evaluated our approach on two 7-DOF platforms: the Barrett WAM arm and the PR2 robot. Data consisted of approximately 50 demonstrations per task, provided by four users via kinesthetic teaching.
Task Scenarios
- Move-can: Carrying a can while maintaining a vertical orientation.
- Laptop Task: Carrying a cup upright while avoiding the "no-fly zone" above electronic devices.
Key Results
The proposed method demonstrated superior performance compared to standard baselines:
- Precision in Constraints: In the "Move-can" task, our method achieved an average vertical deviation of 8.6 degrees, outperforming the human demonstrators (9.4 degrees) and standard CHOMP planners (13.0 degrees).
- Preference Learning: In the "Laptop" task, the robot learned to avoid the space above electronics, spending only 1.2% of its trajectory over the laptop, compared to 17.3% for a standard obstacle-avoidance planner that lacked human preference data.
- Generalization: The learned cost functions generalized well to new environments where obstacles or start/goal positions were altered, a feat difficult for simple trajectory-replay methods.
Conclusion and Future Work
By bridging discrete graph search and continuous local optimization, we enable tractable IOC for high-dimensional manipulation. While the current approach requires graph regeneration for new tasks, future work will explore more efficient sampling strategies and real-time implementations to handle dynamic environments.
Summary updated