Cleaning Tasks Knowledge Transfer Between Heterogeneous Robots: a Deep Learning Approach

Jaeseok Kim, Nino Cauli, Pedro Vicente, Bruno D. Damas, Alexandre Bernardino, José Santos-Victor, Filippo CavalloJ. Intell. Robotic Syst 2019

TL;DR

This paper presents a deep learning approach to enable robots to learn cleaning tasks from human demonstrations and transfer this knowledge to a different robot. By using a convolutional neural network (CNN) to interpret visual input and a task-parameterized Gaussian Mixture Model (TP-GMM) to represent learned skills, the system successfully transfers cleaning capabilities between two distinct robotic platforms.

What Problem Does This Paper Address?

Service robots are increasingly expected to operate in dynamic, human-centric environments, performing tasks like cleaning. Unlike highly controlled industrial settings, these robots need to be robust to variations in lighting, object placement, and even the robot's own pose. Traditional control methods struggle with generalization in such unpredictable scenarios. Learning from Demonstration (LfD) offers a promising avenue, but transferring learned skills between robots with different kinematic structures and operating in different environments remains a significant challenge. This paper tackles this challenge by developing a method to transfer cleaning knowledge from one robot (iCub) to another (DoRo) using visual input, aiming for robustness and generalization.

Methods

The core of the proposed approach involves a pipeline that transforms raw camera images into a format suitable for learning and then uses this learned representation to control a robot.

  • Virtual Camera Transformation: To overcome differences in robot camera geometry and pose, input images are transformed into a canonical "bird-eye view" using a homographic transformation. This ensures a consistent viewpoint regardless of the robot's actual camera position.
  • Convolutional Neural Network (CNN): A CNN, based on the AlexNet architecture but with a modified output layer, is used to process the virtual camera images. Instead of directly predicting robot actions, the CNN predicts the positions of three key reference frames (initial, intermediate, and final) for the end-effector.
  • Task-Parameterized Gaussian Mixture Model (TP-GMM): The predicted reference frames from the CNN serve as task parameters for a TP-GMM. This model represents the learned cleaning skill as a probabilistic distribution, allowing for generalization and interpolation of trajectories. Gaussian Mixture Regression (GMR) is then used to generate the actual end-effector trajectory.
  • Data Augmentation: To enhance robustness to variations in lighting and background, several data augmentation techniques are employed:
  • Changes in illumination (random RGB channel adjustments).
  • Pixel and trajectory translation.
  • Substitution of background and table with Perlin noise textures. This is crucial for generalizing to different environments.
  • End-effector Orientation Calculation: Unlike previous work that used a separate CNN for orientation, this paper analytically calculates the 2D rotation matrices for the reference frames based on the predicted positions. This approach is more robust to differences in robot kinematics.

Datasets

  • iCub Dataset: The initial dataset was collected using the iCub robot in Lisbon, Portugal. It consists of 659 kinesthetic demonstrations of sweeping lentils and wiping marker scribbles from a table. The data includes images from the iCub's right eye and the corresponding 2D trajectories of the robot's right hand.
  • Augmented Dataset: The original iCub dataset was augmented 20 times (10 times with illumination/translation changes, and 10 times with Perlin noise) to create a larger, more robust training set.
  • DoRo Dataset (Implicit): While the DoRo robot was used for testing, the paper implies that the trained network from the iCub dataset was directly applied to the DoRo's input images after the virtual camera transformation. No new dataset was collected on the DoRo for training the CNN.

Key Insights and Ideas

  • Virtual Camera as a Generalization Tool: The virtual camera projection is a powerful technique to abstract away differences in robot camera geometry, making the learned model more transferable.
  • Perlin Noise for Environmental Robustness: Using Perlin noise to randomize backgrounds and tables proved essential for transferring the cleaning task to a robot in a visually distinct environment (DoRo vs. iCub). This highlights the importance of domain randomization for cross-robot transfer.
  • Analytical Orientation Calculation: Moving from a learned orientation prediction to an analytical calculation based on predicted positions simplifies the network and improves robustness to differing robot kinematics.
  • Trade-off Between Data and Augmentation: The study demonstrates that with effective data augmentation, a significantly smaller amount of original kinesthetic data can achieve comparable or even better performance than using a larger dataset without augmentation.

Key Results

  • Successful Cross-Robot Transfer: The system trained on the iCub robot was successfully deployed on the DoRo robot, performing cleaning tasks with comparable performance.
  • Marker Scribble Cleaning: On the DoRo robot, the system reduced the dirty area of marker scribbles by 75% within 5 repetitions, with a standard deviation of 20%.
  • Lentil Cleaning: For lentils, the system reduced the distance to the target position by 50% within 5 repetitions, with a standard deviation of 10%.
  • Impact of Data Augmentation: Table 1 shows that using only 20% of the original data with translation and illumination augmentation (T20%) resulted in a validation loss comparable to using 80% of the original data without augmentation (O80%). Perlin noise augmentation (P20%) further improved generalization.
  • Reduced Demonstration Needs: The study suggests that with data augmentation, a significantly smaller number of kinesthetic demonstrations are needed for effective learning, reducing the burden of data collection.

Limitations and Future Work

  • Initial Manual Calibration: The current framework requires an initial manual calibration step to set up the virtual camera on the second robot.
  • Open-Loop Trajectory Generation: The trajectory generation is currently open-loop. This means the robot might repeat the same trajectory if the dirt configuration doesn't change significantly, potentially leading to inefficient cleaning.
  • Lack of Stopping Criteria: The system lacks an explicit criterion to stop cleaning when the table is sufficiently clean.

Future work suggested includes:

  • Implementing a stopping criterion based on detecting a clean table.
  • Exploring deep reinforcement learning approaches, where the robot can learn from trial and error to improve cleaning efficiency and adapt to different dirt configurations.

Summary updated

Abstract

In this paper, a robot is taught to perform two different cleaning tasks over a table, using a learning from demonstration paradigm. Robustness to robot posture and illu- mination changes is achieved using data augmentation techniques and camera images transformation. This robustness allows the transfer of knowledge regarding execution of cleaning tasks be- tween heterogeneous robots operating in different environmental settings. To demonstrate the viability of the proposed approach, a CNN network trained in Lisbon to perform cleaning tasks, using the iCub robot,is successfully employed by the DoRo robot in Peccioli, Italy.
BibTeX
@article{kim2019cleaning,
  author = {Jaeseok Kim and Nino Cauli and Pedro Vicente and Bruno D. Damas and Alexandre Bernardino and José Santos-Victor and Filippo Cavallo},
  title = {{Cleaning Tasks Knowledge Transfer Between Heterogeneous Robots: a Deep Learning Approach}},
  journal = {J. Intell. Robotic Syst},
  year = {2019},
}
Figure