Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting

Mennatullah Siam, Chen Jiang, Steven Weikai Lu, Laura Petrich, Mahmoud Gamal, Mohamed Elhoseiny, Martin Jägersand

TL;DR

This paper introduces a novel "teacher-student" learning approach for video object segmentation, enabling robots to learn new objects through human demonstration without requiring manual segmentation labels. They also present IVOS, a new dataset specifically designed for human-robot interaction (HRI) scenarios involving object manipulation, and demonstrate significant performance improvements on existing benchmarks and their new dataset.

What Problem Does This Paper Address?

Robots operating in real-world, unstructured environments need to be able to learn and segment new objects incrementally. Traditional video object segmentation methods often rely on large, pre-existing datasets or require manual initialization (like a segmentation mask in the first frame). This is impractical for robots that need to adapt to novel objects encountered during interaction. The paper addresses this by proposing a human-robot interaction (HRI) paradigm where a human "teacher" can guide a robot to learn about objects through demonstrations, mimicking how children learn from parents. The key challenge is to enable the robot to segment these newly learned objects in various conditions (static, moving, different poses, occlusions) without explicit manual annotation for each new object.

Methods

The core of the proposed solution is a Teacher-Student Adaptation framework.

  • Teacher Network: This network is designed to understand both motion and appearance cues. It's a two-stream architecture that takes an input video and generates "pseudo-labels" – essentially, educated guesses about which pixels belong to the object of interest. The teacher's ability to leverage motion is crucial for generating these labels.
  • Student Network: This is a more efficient, appearance-only network. Its goal is to learn to segment objects based on the pseudo-labels provided by the teacher.
  • Adaptation Process: The teacher network's output (pseudo-labels) is used to train or adapt the student network. This adaptation happens only once at the beginning of a new object learning task. The paper explores two adaptation strategies:
  • Discrete Labels: Uses "confident positive pixels" identified by the teacher as hard labels (1 for object, 0 for background) after a geometric distance transform. This offers higher accuracy but requires careful parameter tuning.
  • Continuous Labels: Uses the raw probability maps from the teacher network directly. This is more robust to hyperparameter tuning and generalizes better but might yield slightly lower accuracy.

The adaptation process is designed to be "unsupervised" in the sense that it doesn't require manual segmentation masks from the human teacher during the learning phase. The human simply demonstrates the object.

Datasets

The paper introduces a new dataset called (I)nteractive (V)ideo (O)bject (S)egmentation (IVOS).

  • IVOS Dataset: This dataset is specifically designed for HRI scenarios and includes:
  • Human Teaching Objects: Videos where a human demonstrates objects with various transformations (translation, scale, rotation, out-of-plane rotation, non-rigid transformations) and illumination conditions.
  • Manipulation Tasks: Videos featuring objects being manipulated by humans or robots, focusing on activities like drinking, stirring, and pouring. This includes robot trajectories.
  • Existing Datasets: The method is also evaluated on established benchmarks:
  • DAVIS: A popular dataset for unsupervised video object segmentation.
  • FBMS: Another benchmark for motion segmentation.

Key Insights and Ideas

  • Learning by Demonstration in HRI: The paper strongly advocates for using HRI as a natural and effective way to teach robots about the visual world, drawing parallels to how children learn.
  • Teacher-Student for Efficiency: By using a powerful but slower "teacher" network to generate labels and then adapting a faster "student" network, the system achieves efficient inference after the initial learning phase.
  • Motion as a Teacher's Signal: Leveraging motion cues in the teacher network is key to generating useful pseudo-labels without explicit human annotation, making the teaching process more intuitive.
  • Adaptation without Manual Masks: The ability to adapt to new objects without requiring the human to draw segmentation masks is a significant step towards more natural human-robot collaboration.
  • IVOS Dataset's Novelty: The inclusion of manipulation tasks and robot trajectories in IVOS makes it a unique benchmark for evaluating segmentation in practical robotic applications.

Key Results

  • Performance on Existing Datasets:
    • Outperforms state-of-the-art on DAVIS with a 6.8% improvement in F-measure.
  • Outperforms state-of-the-art on FBMS with a 1.2% improvement in F-measure.
  • Performance on IVOS Dataset:
    • Improves over the baseline (without adaptation) by 46.1% and 25.9% in mIoU on Scale/Rotation and Manipulation Tasks, respectively.
  • The discrete adaptation method generally outperforms the continuous method on IVOS.
  • Qualitative Results: Figures demonstrate the effectiveness of motion adaptation in improving segmentation quality, especially in challenging scenarios and for newly learned objects.

Limitations and Future Work

  • Hyperparameter Tuning: The discrete adaptation method, while achieving better results, requires careful tuning of confidence thresholds. The continuous method alleviates this but at a slight performance cost.
  • Human Teacher's Role: While the method reduces the need for manual segmentation, the human teacher still needs to perform demonstrations. Future work could explore more complex teaching signals or further reduce the teacher's burden.
  • Generalization to Highly Diverse Scenarios: While the method shows good generalization, extremely challenging scenarios with severe occlusions, drastic illumination changes, or highly dynamic backgrounds might still pose difficulties.
  • Robot Trajectory Learning: The IVOS dataset includes robot trajectories, suggesting future work could explore how segmentation aids in learning or predicting these trajectories.

Summary updated

Abstract

Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance “teacher” network provides pseudo-labels to adapt an appearance “student” network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Our proposed adaptation method outperforms the state-of-theart on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.
BibTeX
@inproceedings{siam2019video,
  author = {Mennatullah Siam and Chen Jiang and Steven Weikai Lu and Laura Petrich and Mahmoud Gamal and Mohamed Elhoseiny and Martin Jägersand},
  title = {{Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8794254},
}
Figure