Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

Daniel Seita, Nawid Jamali, Michael Laskey, Ajay Kumar Tanwani, Ron Berenstein, Prakash Baskaran, Soshi Iba, John F. Canny, Ken GoldbergISRR 2019 · DOI

TL;DR

This paper introduces a deep transfer learning approach to automatically identify "pick points" on fabric for robotic manipulation, specifically for the task of bed-making. By training a network on depth images, the system achieves high blanket coverage, comparable to human performance, and generalizes well to different fabrics and robots.

What Problem Does This Paper Address?

Manipulating deformable objects like fabric is a long-standing challenge in robotics due to their infinite configuration spaces and sensitivity to initial conditions. While tasks like folding clothes or robotic surgery have seen progress, achieving robust and generalizable fabric manipulation for everyday tasks remains difficult. The paper focuses on the specific problem of bed-making, a common but physically demanding chore. The core challenge is to enable a robot to effectively grasp and manipulate a blanket to cover a bed, which requires accurately identifying suitable "pick points" on the fabric. Traditional methods often rely on precise object models or specific sensor setups, which are not always practical or generalizable.

Methods

The authors propose a deep transfer learning approach to learn a grasp network ($\pi_\theta$) that maps a depth image of the blanket to a pixel-space pick point $(x, y)$. This network is based on the YOLO (You Only Look Once) object detection architecture, leveraging pre-trained weights from the Pascal VOC 2012 dataset. The depth images are pre-processed by triplicating the single channel to match the input requirements of the RGB-trained YOLO weights. Crucially, only the later convolutional and dense layers of the YOLO network are fine-tuned, while the initial layers are kept fixed. This transfer learning strategy aims to leverage learned features for object recognition while adapting to the specific task of fabric pick point detection.

For the bed-making task, the robot uses the predicted pick point to grasp the blanket and pull it towards the nearest uncovered bed corner. A heuristic is used to determine if additional attempts are needed, with a limit of four attempts per side.

Datasets

The paper introduces a dataset for training and evaluating their pick point detection method. This dataset consists of 2018 depth images collected from two robots (HSR and Fetch) performing the bed-making task with a white blanket. The pick points (ground truth) were automatically labeled using a red marker on the blanket's corner. For evaluation, the dataset was split into 10 folds for cross-validation. The paper also discusses experiments on two novel blankets (multicolored yellow and blue, and teal) to test generalization.

Key Insights and Ideas

  • Depth Images for Invariance: Using depth images as input makes the pick point detection invariant to fabric color and texture, a significant advantage for generalization.
  • Transfer Learning Efficiency: Leveraging pre-trained YOLO weights significantly improves performance and reduces the need for massive amounts of task-specific data. Fixing early layers of the network proved more effective than full fine-tuning.
  • Sim-to-Real Gap Mitigation: While not explicitly a sim-to-real paper, the use of transfer learning from a general object detection task to a specific robotic manipulation task hints at bridging the gap between simulated and real-world performance.
  • Generalization Across Robots: The trained network demonstrates good generalization across two different mobile manipulator robots (HSR and Fetch), suggesting the learned features are robust.
  • Human-Level Performance: The learned pick point selection achieves coverage comparable to human supervisors, indicating the effectiveness of the deep learning approach.

Key Results

  • The trained grasp network achieved an average blanket coverage of 92% on a quarter-scale bed, outperforming an analytic baseline (83%) and closely matching human supervisors (95%).
  • The model demonstrated strong generalization, achieving 93% coverage on two novel blankets it was not explicitly trained on.
  • The Harris Corner Detector, a classical computer vision approach, was found to be insufficient, yielding an average L2 pixel error of 175.0, which is about 6x worse than the learned grasp network.
  • The learned grasp network required fewer grasp attempts on average (4.4 for HSR, 4.3 for Fetch) compared to the analytic baseline (6.2 for HSR, 4.9 for Fetch).
  • The timing analysis shows that the grasp network forward pass is very fast (0.1 ± 0.02 seconds), making it suitable for real-time deployment.

Limitations and Future Work

  • Limited to Specific Task: The current work is focused on bed-making and fabric manipulation on a flat surface. Extending this to more complex fabric manipulation tasks or different object types would be valuable.
  • Number of Attempts: While the robot can make multiple attempts, the paper acknowledges that the task is still challenging, and the robot may not achieve sufficient coverage in a single attempt.
  • Future Work: The authors suggest exploring scalable representations for learning pick points using "Fog robotics" and extending the task to handle a wider variety of blanket configurations. They also plan to investigate applications to other deformable objects like furniture covers and table cloths.

Summary updated

BibTeX
@inproceedings{seita2019deep,
  author = {Daniel Seita and Nawid Jamali and Michael Laskey and Ajay Kumar Tanwani and Ron Berenstein and Prakash Baskaran and Soshi Iba and John F. Canny and Ken Goldberg},
  title = {{Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making}},
  booktitle = {ISRR},
  year = {2019},
  doi = {10.1007/978-3-030-95459-8\_17},
}
Figure