A Virtual Environment with Multi-Robot Navigation, Analytics, and Decision Support for Critical Incident Investigation

David L. Smyth, James Fennell, Sai Abinesh, Nazli B. Karimi, Frank G. Glavin, Ihsan Ullah, Brett Drury, Michael G. MaddenIJCAI 2018 · DOI

TL;DR

This paper introduces ROCSAFE, an integrated system that uses a virtual environment and multi-robot coordination to aid in the investigation of critical incidents like CBRNE events. By combining AI techniques for scene assessment, object detection, and information retrieval, ROCSAFE aims to reduce the cognitive load on human investigators and improve decision-making in high-risk scenarios.

What Problem Does This Paper Address?

Investigating incidents involving chemical, biological, radiological, nuclear, or explosive (CBRNE) materials is exceptionally rare, high-risk, and cognitively demanding for human responders. Traditional methods struggle with the complexity and danger of these scenarios. The paper addresses the need for advanced tools that can assist investigators by automating tasks, providing real-time analysis, and supporting decision-making in a safe, simulated environment before deployment in real-world situations. This includes planning scene assessment, evaluating risks, controlling autonomous vehicles for data collection, and analyzing retrieved information.

Methods

The ROCSAFE system is built around a virtual environment created using Unreal Engine and Microsoft's AirSim plugin, allowing for realistic physics and photo-realistic visualizations. This environment models a CBRNE scenario involving a train carrying radioactive material.

Key AI techniques employed include:

  • Multi-Robot Navigation and Surveying: A swarm of Micro Unmanned Aerial Vehicles (MUAVs), referred to as Robotic Aerial Vehicles (RAVs), autonomously survey the virtual scene. They divide the work by discretizing the area into grid points and collecting images and metadata. The current route planning uses a greedy algorithm, with plans to integrate hyper-heuristics and Markov Decision Processes for optimization.
  • Image Processing and Scene Analysis: A Deep Neural Network (DNN), specifically Mask R-CNN, is used to detect and identify relevant objects within the images captured by the RAVs. This DNN is fine-tuned on synthetic images from the virtual scene, a technique shown to transfer well to real-world data. It also performs pixel-level semantic annotation for terrain analysis, aiding in route planning for ground robots.
  • Probabilistic Reasoning: A probabilistic model (BLOG) assesses the likelihood of different threat categories (chemical, biological, radiation/nuclear) and specific substances based on incoming information from sensors and human input.
  • Information Retrieval: A module ranks relevant documentation (e.g., standard operating procedures) using TF-IDF and Elastic Search, prioritizing information based on its relevance to the current incident.
  • Communications Protocol: A purpose-built JSON-based protocol facilitates communication between subsystems, designed to be loosely coupled and support integration with real-world systems.

Datasets

The paper does not explicitly mention training on external, publicly available datasets for the core AI modules. Instead, it highlights the use of synthetic images collected from their own virtual scene for fine-tuning the Mask R-CNN object detection model. This approach leverages the controlled environment to generate annotated data for training, which they note has shown good transferability to real-world scenarios in other domains.

Key Insights and Ideas

  • Virtual Environments as a Crucial Testbed: The paper strongly emphasizes the value of realistic virtual environments for developing and evaluating AI tools for high-risk, low-frequency events like CBRNE incidents. This allows for safe experimentation and iteration without the dangers and costs of real-world testing.
  • Integrated AI for Holistic Support: ROCSAFE demonstrates the power of combining multiple AI disciplines—robotics, computer vision, probabilistic reasoning, and information retrieval—into a single, cohesive system to provide comprehensive decision support.
  • Synthetic Data for Real-World Problems: The successful fine-tuning of a state-of-the-art object detection model (Mask R-CNN) on purely synthetic data from their virtual environment is a significant insight, suggesting a viable path for training AI in domains where real-world data is scarce or dangerous to acquire.
  • Extensible Architecture: The design of the system with a loosely coupled communications protocol and the public release of their tools (as indicated in the contributions) aims to foster collaboration and allow other researchers to integrate their own AI modules or test on new scenarios.

Key Results

The paper focuses on the system's architecture and capabilities rather than presenting specific quantitative performance metrics. However, it states that the system:

  • Provides an integrated system for critical incident decision support.
  • Features an extensible virtual environment model of an incident.
  • Implements a communications protocol for information exchange between subsystems.
  • The Image Analysis module uses a DNN to detect and identify relevant objects and perform pixel-level semantic annotation.
  • The Probabilistic Reasoning module assesses the likelihood of different threats.
  • The Information Retrieval module ranks documentation by relevance.

The authors also note that their tools are open-source and being released publicly.

Limitations and Future Work

  • Current Route Planning: The current greedy algorithm for route planning is acknowledged as a limitation. Future work will involve integrating more advanced algorithms like hyper-heuristics and Markov Decision Processes to optimize agent routes, especially in stochastic environments where factors like battery life are uncertain.
  • Real-World Deployment: While the system is designed to support future testing in real-world environments, the current work is confined to the virtual simulation.
  • Enhancing Image Analysis: The authors plan to further enhance the performance of the object detection technique by retraining/fine-tuning the network on other relevant datasets, such as the DOTA dataset for aerial images.
  • Integration of Other AI Modules: The open-source nature of the system is intended to allow other researchers to integrate their own AI modules for image recognition, routing, or probabilistic reasoning.

Summary updated

BibTeX
@inproceedings{smyth2018virtual,
  author = {David L. Smyth and James Fennell and Sai Abinesh and Nazli B. Karimi and Frank G. Glavin and Ihsan Ullah and Brett Drury and Michael G. Madden},
  title = {{A Virtual Environment with Multi-Robot Navigation, Analytics, and Decision Support for Critical Incident Investigation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/863},
}
Figure