Learning to Assess Danger from Movies for Cooperative Escape Planning in Hazardous Environments

Vikram Shree, Sarah Allen, Beatriz A. Asfora, Jacopo Banfi, Mark E. CampbellIROS 2022 · DOI

TL;DR

This paper tackles the challenge of robot navigation in hazardous environments by proposing a novel approach to danger assessment. They leverage movie scenes to create a rich dataset and develop a multi-modal system that fuses visual and human language inputs to estimate danger, ultimately leading to safer escape planning for human-robot teams.

What Problem Does This Paper Address?

Deploying robots in real-world hazardous environments like disaster sites is incredibly difficult. Two major hurdles exist: (1) replicating these dangerous scenarios for training and testing robots is impractical and costly, and (2) current robot perception systems struggle to effectively utilize the complex, multi-modal data present in such environments. This paper aims to bridge this gap by proposing a method to learn danger assessment from readily available visual content (movies) and to fuse this with human input for more robust navigation and escape planning in emergencies.

Methods

The core of the proposed system is a multi-modal danger estimation pipeline. It consists of three main components:

  • Visual Perception: This module uses deep convolutional neural networks (CNNs) to analyze images from the robot's camera and predict a danger level. They evaluated several state-of-the-art CNN architectures (VGGNet, ResNet, DenseNet, EfficientNet) and found VGGNet-13 to be the most effective for their task.
  • Language Perception: This module takes a single word input from a human and predicts a danger level. They experimented with k-Nearest Neighbors (k-NN), Logistic Regression, and Support Vector Machines (SVMs), with SVM (RBF kernel) performing best.
  • Bayesian Fusion: This is the novel contribution. It combines the predictions from the visual and language modules using a Bayesian framework. By assuming conditional independence between the visual and language predictions given the true danger, they can fuse these likelihoods to obtain a more accurate, combined danger estimate. This framework is designed to be extensible to multiple human inputs.
  • Risk-Aware Planning: The fused danger estimate is then used by a risk-aware planner to find the safest escape path. This planner aims to maximize the probability of survival by avoiding areas with intolerable danger levels.

Datasets

The paper introduces a novel hazardous environment dataset created by extracting images from movies and TV shows. This dataset contains:

  • 1002 images depicting various disaster scenarios (fire, flooding, earthquakes).
  • Danger ratings for each image, annotated by 15 human workers on a 5-point scale (1-low to 5-extreme).
  • Associated keywords describing the scene content that contributed to the danger rating.

This dataset is designed to overcome the limitations of real-world data collection by providing a rich, diverse, and ethically sourced source of visual information for hazardous environments.

Key Insights and Ideas

  • Movies as a rich source of training data: Leveraging movie scenes provides a scalable and cost-effective way to generate diverse and realistic data for training robots in hazardous environments, avoiding the ethical and practical challenges of real-world disaster data.
  • Multi-modal fusion is crucial: Combining visual information with human language feedback significantly improves danger estimation compared to using either modality alone. This highlights the power of human-robot collaboration.
  • Language input can be surprisingly informative: Even single words from humans can provide valuable context for danger assessment, especially when combined with visual data.
  • Extreme danger is easier to perceive: Both visual and language models showed higher consensus and accuracy in identifying scenes with very low or very high danger levels, while moderate danger levels were more ambiguous.
  • Improved danger estimation leads to better planning: The risk-aware planner, informed by the fused danger estimate, demonstrated a tangible increase in mission success rates, validating the importance of accurate danger perception.

Key Results

  • Visual Perception: VGGNet-13 achieved a top-1 accuracy of 47.5% and an off-by-1 accuracy of 79.2% on their dataset.
  • Language Perception: SVM (RBF kernel) achieved a top-1 accuracy of 37.6% and an off-by-1 accuracy of 68.5%. Notably, language perception metrics were generally lower than visual perception, attributed to the richer nature of visual data.
  • Multi-modal Fusion: Combining vision with just one word input improved top-1 accuracy to 46.5% and off-by-1 accuracy to 78.2%. With 5 words, top-1 accuracy reached 48.5% and off-by-1 accuracy 83.2%, outperforming single modalities.
  • Planning Performance: In simulations, the multi-modal approach (VL-5) achieved a significantly higher success rate (around 70% at moderate danger tolerance) compared to "no sensor" (around 30%) and even "vision only" (around 55%). In some cases, the multi-modal approach even outperformed a hypothetical "full knowledge" scenario, suggesting that a conservative approach based on estimated danger can be beneficial.

Limitations and Future Work

  • Independence Assumption: The Bayesian fusion relies on the assumption of conditional independence between visual and language inputs. The authors acknowledge that this might not always hold in practice and suggest that more sophisticated models could account for spatial dependencies.
  • Single Word Input: While the framework is extensible, the current language perception module primarily focuses on single-word inputs. Future work could explore processing more complex natural language commands or descriptions.
  • Dataset Limitations: While the movie dataset is rich, it's still a proxy for real-world disaster scenarios. The authors suggest that specialized models for vision-based danger assessment are a direction for future work.
  • Redundant Information: The authors caution against providing too much redundant human feedback, as it could violate the independence assumption and lead to over-reliance on human input.
  • Future Work: The paper suggests developing more specialized models for vision-based danger assessment and exploring more complex natural language processing for richer human feedback.

Summary updated

BibTeX
@inproceedings{shree2022learning,
  author = {Vikram Shree and Sarah Allen and Beatriz A. Asfora and Jacopo Banfi and Mark E. Campbell},
  title = {{Learning to Assess Danger from Movies for Cooperative Escape Planning in Hazardous Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982279},
}
Figure