Prediction-Based Human-Robot Collaboration in Assembly Tasks Using a Learning from Demonstration Model

Zhujun Zhang, Gaoliang Peng, Weitian Wang, Yi Chen, Yunyi Jia, Shaohui LiuSensors 2022 · OpenReview

TL;DR

This paper introduces a novel prediction-based framework for human-robot collaboration in assembly tasks. By learning from human demonstrations, the robot can anticipate human intentions and proactively deliver the correct parts, leading to smoother, more efficient, and safer collaboration.

What Problem Does This Paper Address?

Traditional industrial robots are often programmed for specific, repetitive tasks with little flexibility. However, the rise of Small and Medium-sized Enterprises (SMEs) and the increasing demand for customized products necessitate more adaptable and collaborative robots. The challenge lies in enabling robots to work seamlessly alongside humans, understanding their intentions, and providing assistance without explicit commands. This requires robots to go beyond simple task execution and predict future human actions to facilitate a fluid and efficient collaborative workflow, especially in complex assembly scenarios where human actions can be varied and unpredictable.

Methods

The core of this work is a prediction-based human-robot collaboration framework that leverages Learning from Demonstration (LfD).

  • Learning from Demonstration (LfD): The robot learns task descriptions and human preferences by observing human workers perform assembly tasks. This allows the robot to adapt to different working styles without requiring explicit programming.
  • State-Enhanced Convolutional LSTM (ConvLSTM): A novel ConvLSTM architecture is proposed to extract high-level spatiotemporal features from the shared workspace. This enhanced model incorporates additional linkages to better capture long-term dependencies and reduce information loss, crucial for predicting future actions.
  • Action Prediction: The ConvLSTM framework predicts future human actions, enabling the robot to anticipate what the human worker will need next.
  • Predicted Part Tracking: An Extended Kalman Filter (EKF) is used to track the specific part the human intends to use, based on the predicted actions. This is essential for accurate part delivery.
  • Intention Recognition: A Convolutional Neural Network (CNN) model is employed to recognize the intended assembly part with high accuracy, translating human intentions into actionable robot commands.
  • Proactive Assistance: The robot uses the predicted intentions and recognized parts to proactively pick up and deliver the correct components at the right time, facilitating a smooth task transition.

Datasets

The paper describes the creation of a dataset for training and evaluation.

  • Custom Dataset: The dataset was built from a vehicle seat assembly experiment. It includes video data collected from a vision sensor during both human demonstrations and the robot's prediction sequences.
  • Data Augmentation: The process of collecting images from tracking predicted and demonstration videos, and then cropping them, served as a form of data augmentation.
  • Semi-Automatic Labeling: The dataset was labeled semi-automatically, leveraging the continuous nature of the captured data.

Key Insights and Ideas

  • Proactive Assistance is Key: The most significant insight is that robots can significantly improve collaboration by predicting human needs rather than just reacting. This shifts the robot's role from a passive tool to an active, intelligent partner.
  • LfD for Adaptability: Learning from demonstration is crucial for robots to adapt to diverse human working styles and non-standardized tasks, which is a major challenge in real-world industrial settings.
  • Spatiotemporal Features are Vital: Capturing both spatial (where things are) and temporal (when things happen) information is essential for accurate human action prediction. The state-enhanced ConvLSTM addresses this by improving the model's ability to retain this information over time.
  • Integrated Prediction and Tracking: The combination of action prediction and robust part tracking (using EKF) is critical for translating predicted intentions into concrete robotic actions like part delivery.
  • Reduced Idle Time = Smoother Collaboration: The experimental results strongly suggest that reducing idle time for both humans and robots leads to a more fluid and efficient collaborative process.

Key Results

  • Improved Prediction Accuracy: The proposed state-enhanced ConvLSTM model outperformed a non-enhanced version, demonstrating better learning and prediction capabilities.
  • Higher Recognition Accuracy: The CNN model achieved over 99% accuracy in recognizing intended parts.
  • Reduced Idle Times: The prediction-based HRC approach reduced human idle time by 34.5% and robot idle time by 58% compared to a speech-commanded approach.
  • Faster Task Completion: The total task finish time was reduced by an average of 36.43 seconds using the prediction-based method compared to speech commands.
  • Smoother Collaboration: The framework demonstrated higher smoothness and shorter idle times, leading to a more comfortable and efficient collaborative experience.
  • Adaptability to Working Styles: The model successfully adapted to different assembly preferences, as shown in the experiments with varied seat assembly sequences.

Limitations and Future Work

  • Occlusion Scenarios: The paper acknowledges limitations in predicting human intentions in scenarios with occlusions, particularly when parts are in areas not visible to the camera.
  • Limited Workspace: The view field of the camera restricts the workspace, which can be an issue if humans need to access parts outside this view.
  • Future Work:
    • More Complex Practices: Extending the framework to more complex industrial applications.
  • Wearable Sensors: Incorporating wearable sensors to reduce susceptibility to occlusions and improve data richness.
  • Enhanced Safety and Efficiency: Developing more advanced methods to simultaneously ensure safety and improve efficiency.
  • Personalization: Allowing human coworkers to personalize the robot's assistance further.

Summary updated

Abstract

Most robots are programmed to carry out specific tasks routinely with minor variations. However, more and more applications from SMEs require robots work alongside their counterpart human workers. To smooth the collaboration task flow and improve the collaboration efficiency, a better way is to formulate the robot to surmise what kind of assistance a human coworker needs and naturally take the right action at the right time. This paper proposes a prediction-based human-robot collaboration model for assembly scenarios. An embedded learning from demonstration technique enables the robot to understand various task descriptions and customized working preferences. A state-enhanced convolutional long short-term memory (ConvLSTM)-based framework is formulated for extracting the high-level spatiotemporal features from the shared workspace and predicting the future actions to facilitate the fluent task transition. This model allows the robot to adapt itself to predicted human actions and enables proactive assistance during collaboration. We applied our model to the seats assembly experiment for a scale model vehicle and it can obtain a human worker’s intentions, predict a coworker’s future actions, and provide assembly parts correspondingly. It has been verified that the proposed framework yields higher smoothness and shorter idle times, and meets more working styles, compared to the state-of-the-art methods without prediction awareness.
BibTeX
@inproceedings{zhang2022prediction,
  author = {Zhujun Zhang and Gaoliang Peng and Weitian Wang and Yi Chen and Yunyi Jia and Shaohui Liu},
  title = {{Prediction-Based Human-Robot Collaboration in Assembly Tasks Using a Learning from Demonstration Model}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=25rToY-h-OR},
}
Figure