Optimal selective floor cleaning using deep learning algorithms and reconfigurable robot hTetro

Balakrishnan Ramalingam, Anh Vu Le, Zhiping Lin, Zhenyu Weng, Mohan Rajesh Elara, Sathian PookkuttathScientific 2022 · DOI

TL;DR

This paper introduces a smart floor cleaning robot, "hTetro," that uses deep learning to identify and clean only dirty areas, significantly reducing cleaning time and energy consumption. By combining human traffic pattern analysis with stain and trash detection, the robot intelligently navigates and cleans, offering a more efficient and autonomous solution for public spaces.

What Problem Does This Paper Address?

Maintaining cleanliness in high-traffic public areas like malls and hospitals is crucial but challenging. Frequent cleaning is necessary, but it leads to increased wear and tear on cleaning robots and higher operational costs. Traditional cleaning robots often operate on a fixed schedule or clean the entire area, regardless of actual dirt levels. This leads to wasted time and energy. Furthermore, existing "selective cleaning" methods often require human supervision, making them impractical for large-scale autonomous operation. This paper aims to bridge this gap by developing an autonomous system that can intelligently identify and target only the areas that need cleaning.

Methods

The proposed framework integrates several key technologies:

  • RGB-D Vision Sensor Network: A network of cameras captures both color (RGB) and depth (D) information of the environment. This allows for a 3D understanding of the space.
  • Deep SORT Human Tracking: This algorithm tracks human movement patterns. The idea is that areas with high human traffic are more likely to accumulate dirt.
  • SSD MobileNet Object Detection: This deep learning model is trained to detect specific types of dirt, including stains, foot stains, and trash on the floor.
  • Selective Area Identification: The system combines the outputs of human tracking and object detection to define "selective cleaning/spot cleaning" regions. This means the robot focuses on areas identified as dirty or high-traffic.
  • Evolutionary-Based Optimization for Path Planning: Once the dirty areas are identified, an evolutionary algorithm (specifically Ant Colony Optimization - ACO) is used to generate an optimal and efficient path for the hTetro robot to navigate to these locations. This is a form of Complete Waypoint Coverage Planning (CWCP).
  • Reconfigurable Robot hTetro: The cleaning robot itself is modular and reconfigurable, allowing it to adapt its shape for efficient navigation in various spaces, including narrow areas.

Datasets

The paper mentions an in-house dataset prepared for training the object detection framework. This dataset consists of images with four classes: Stain, Foot stain, Trash, and Human. Each class has 1200 images, with a mix of in-house captured images and those collected online. The images were captured from an overhead perspective using a Realsense RGBD camera.

Key Insights and Ideas

  • Human Traffic as a Proxy for Dirt: The insight that high human traffic areas are more prone to dirt is a clever way to proactively identify potential cleaning zones, complementing direct dirt detection.
  • Synergy of Deep Learning and Evolutionary Algorithms: The combination of deep learning for perception (identifying dirt and humans) and evolutionary algorithms for optimal path planning creates a robust and efficient system.
  • Reconfigurability for Efficiency: The hTetro robot's ability to change its shape is crucial for navigating complex environments and reaching all identified cleaning spots, enhancing the effectiveness of the selective cleaning approach.
  • Trade-off in Object Detection: The study highlights that while some advanced models like Faster RCNN offer higher accuracy, SSD MobileNet provides a better balance between accuracy and inference time, making it more suitable for real-time robotic applications.

Key Results

  • High Detection Accuracy: The SSD MobileNet algorithm achieved a high confidence score for detecting human traffic patterns (97.0%), stains (93.44%), foot stains (91.69%), and trash (93.0%).
  • Reduced Cleaning Time and Energy: The evolutionary-based optimization path planning scheme reduced navigation time by 15% and energy consumption by 10% compared to conventional methods.
  • Efficient Path Planning: The ACO-based Complete Waypoints Coverage Planning (CWCP) method demonstrated the shortest running time and lowest cost weight for completing cleaning trajectories, outperforming zigzag, spiral, and greedy search methods.
  • Real-time Performance: The system achieved an inference time of 0.05 seconds for processing 640x480 images, indicating its suitability for real-time applications.

Limitations and Future Work

  • Dataset Limitations: While the in-house dataset is described, its diversity and scale compared to large public datasets are not explicitly detailed. The paper notes that stain and foot stain detection confidence levels were relatively lower due to less visibility and complex textures, suggesting room for improvement in these specific detection tasks.
  • Environmental Complexity: The experiments were conducted in three different indoor environments. While the results are promising, the system's performance in highly dynamic or extremely cluttered environments might require further investigation.
  • Future Work: The authors suggest that the proposed framework is suitable for enabling autonomous selective area cleaning/spot cleaning. Future work could focus on further enhancing the detection accuracy for challenging dirt types and exploring more complex environmental mapping and navigation strategies. The paper also implicitly points towards the need for robust sensor fusion and control for the reconfigurable robot in real-world scenarios.

Summary updated

Abstract

Floor cleaning robots are widely used in public places like food courts, hospitals, and malls to perform frequent cleaning tasks. However, frequent cleaning tasks adversely impact the robot's performance and utilize more cleaning accessories (such as brush, scrubber, and mopping pad). This work proposes a novel selective area cleaning/spot cleaning framework for indoor floor cleaning robots using RGB-D vision sensor-based Closed Circuit Television (CCTV) network, deep learning algorithms, and an optimal complete waypoints path planning method. In this scheme, the robot will clean only dirty areas instead of the whole region. The selective area cleaning/spot cleaning region is identified based on the combination of two strategies: tracing the human traffic patterns and detecting stains and trash on the floor. Here, a deep Simple Online and Real-time Tracking (SORT) human tracking algorithm was used to trace the high human traffic region and Single Shot Detector (SSD) MobileNet object detection framework for detecting the dirty region. Further, optimal shortest waypoint coverage path planning using evolutionary-based optimization was incorporated to traverse the robot efficiently to the designated selective area cleaning/spot cleaning regions. The experimental results show that the SSD MobileNet algorithm scored 90% accuracy for stain and trash detection on the floor. Further, compared to conventional methods, the evolutionary-based optimization path planning scheme reduces 15% percent of navigation time and 10% percent of energy consumption.
BibTeX
@inproceedings{ramalingam2022optimal,
  author = {Balakrishnan Ramalingam and Anh Vu Le and Zhiping Lin and Zhenyu Weng and Mohan Rajesh Elara and Sathian Pookkuttath},
  title = {{Optimal selective floor cleaning using deep learning algorithms and reconfigurable robot hTetro}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-19249-7},
}
Figure