Q-Learning-Based Model Predictive Variable Impedance Control for Physical Human-Robot Collaboration (Extended Abstract)

Loris Roveda, Andrea Testa, Asad Ali Shahid, Francesco Braghin, Dario PigaIJCAI 2023 · DOI

TL;DR

This paper introduces a novel Q-Learning-based Model Predictive Variable Impedance Control (Q-LMPVIC) system for physical human-robot collaboration (pHRC). It enables robots to adapt their stiffness and damping in real-time to optimize human-robot interaction, making collaboration safer and more efficient.

What Problem Does This Paper Address?

Physical human-robot collaboration (pHRC) is becoming increasingly crucial in manufacturing for tasks requiring flexibility and agility. However, achieving effective collaboration presents several challenges: ensuring safety and adaptive behavior, accurately recognizing human intentions, modeling complex human-robot interaction dynamics, and performing real-time control adaptation and optimization. Existing methods often struggle with simultaneously optimizing impedance parameters, guaranteeing stability, and reliably modeling human-robot dynamics, especially in dynamic industrial settings. This paper aims to address these limitations by proposing a unified control framework that can optimize robot compliance in real-time to enhance human-robot interaction.

Methods

The proposed Q-LMPVIC system operates on two levels:

  • Low-Level Control: A Cartesian impedance controller provides the robot with compliant and decoupled behavior in Cartesian space. This controller's parameters (setpoint and damping) are what the higher-level controller will optimize.
  • High-Level Control: This level is responsible for optimizing the impedance control parameters online to maximize collaboration performance. It comprises:
  • Human-Robot Interaction Dynamics Model: An ensemble of neural networks is used to learn the dynamics of the human-robot interaction, capturing associated uncertainties. This overcomes the need for precise analytical models.
  • Model Predictive Control (MPC): An MPC is designed to optimize the impedance parameters. It's enhanced with stability guarantees through Lyapunov constraints to ensure safety.
  • Q-Learning: The MPC is solved online using a Q-Learning method, specifically an actor-critic algorithm, to approximate the optimal solution. This approach is chosen for its accuracy and computational efficiency, crucial for real-time applications.

The training of the Q-LMPVIC controller occurs during idle time, using a buffer to store interaction data. Once the buffer is full, the controller's weights are updated.

Datasets

Not discussed in the paper. The paper focuses on experimental validation using a Franka EMIKA panda robot in specific tasks.

Key Insights and Ideas

  • Unified Optimization of Impedance Parameters: The core innovation is the online, simultaneous optimization of both the impedance setpoint and damping parameters. This allows the robot to actively adapt its compliance to the specific demands of the collaborative task and the human's actions, rather than relying on pre-set values.
  • Learning Complex Dynamics: By using an ensemble of neural networks to model human-robot interaction dynamics, the system can handle the inherent complexity and uncertainties of human behavior, which are difficult to capture with traditional analytical models.
  • Q-Learning for Efficient MPC: Employing Q-Learning to solve the MPC problem provides a computationally efficient way to achieve near-optimal control, making the system suitable for real-time industrial applications where speed is critical.
  • Guaranteed Stability: The integration of Lyapunov constraints within the MPC framework is a significant step towards ensuring safety in human-robot interactions, a critical concern in collaborative robotics.

Key Results

The paper reports successful experimental validation on a Franka EMIKA panda robot for two complex industrial tasks:

  • Collaborative Assembly Task: Assembling a gear into its shaft.
  • Collaborative Deposition Task: Depositing material along a highlighted path.

The authors state that the proposed controller enables a smooth and reactive robot behavior, effectively reflecting the performance evaluation metrics (though specific quantitative results are not detailed in this extended abstract). The controller is shown to be applicable to real industrial tasks, allowing for complex human-robot collaborative applications.

Limitations and Future Work

  • Control Loop Frequency: The current high-level control loop frequency is limited to 6 Hz. Future work aims to increase this frequency, potentially through parallelization with GPUs, to further enhance pHRC performance.
  • Human Intention Recognition: While the system aims to assist based on detected motion intentions, explicitly capturing more nuanced human intentions could further improve performance.
  • External Sensors: The authors suggest incorporating external sensors, such as Electromyography (EMG) sensors, to better capture human-robot interaction dynamics and collaboration performance.
  • Adaptation to Exoskeletons: The proposed strategy will be adapted (in terms of modeling and control objectives) and applied to an exoskeleton device.
  • Comparison with Prior Work: While the paper mentions comparison with [Roveda et al., 2020] in the full work, specific quantitative comparative results are not detailed in this extended abstract.

Summary updated

BibTeX
@inproceedings{roveda2023q,
  author = {Loris Roveda and Andrea Testa and Asad Ali Shahid and Francesco Braghin and Dario Piga},
  title = {{Q-Learning-Based Model Predictive Variable Impedance Control for Physical Human-Robot Collaboration (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/790},
}
Figure