Path Planning of Cleaning Robot with Reinforcement Learning

Woohyeon Moon, Bumgeun Park, Sarvar Hussain Nengroo, Taeyoung Kim, Dongsoo Har

TL;DR

Researchers from KAIST have developed an enhanced Reinforcement Learning (RL) framework for cleaning robots that combines Proximal Policy Optimization (PPO) with transfer learning and custom reward mechanisms. By focusing on local observations and "nearest uncleaned tile" detection, their model generalizes across different room layouts and outperforms standard zigzag algorithms in both travel distance and energy efficiency.

What Problem Does This Paper Address?

While cleaning robots are ubiquitous, most rely on simplistic "straight-until-obstacle" or "zigzag" path planning. These methods are often energy-inefficient and fail to adapt to complex, dynamic home environments.

Reinforcement Learning offers a potential solution, but standard RL models face a major hurdle: generalization. Typically, if an RL agent is trained in a specific 5x5 room, its performance collapses when moved to a 20x20 room because the "input state" (the map) has changed size. This paper aims to create a model that can be trained once and applied to various environments without retraining, while minimizing the number of turns and total distance traveled to save battery life.

Methods

The authors utilize Proximal Policy Optimization (PPO) as their base algorithm due to its stability and efficiency in high-dimensional spaces. To overcome the limitations of standard RL, they introduce four supplementary techniques:

  1. Transfer Learning (TL) via Local Windows: Instead of feeding the entire map into the neural network, the agent only "sees" the 8 tiles immediately surrounding it. This keeps the input size constant regardless of the room's actual dimensions, allowing the model to function in any size environment.
  2. Detection of Nearest Uncleaned Tile (DNUT): To prevent the agent from getting "stuck" in a local area that is already clean, this technique provides the agent with the relative coordinates of the closest uncleaned tile, acting as a global compass.
  3. Reward Shaping (RS) with Stacked Values: To encourage continuous cleaning, the authors implemented an exponential reward system. If an agent cleans multiple tiles in a row, its reward increases exponentially ($1.5^n$). Conversely, hitting a wall or a cleaned tile resets this "streak."
  4. Elite Set (ES): This is a data-filtering technique. If an episode takes more than 500 steps (indicating poor performance), the data is discarded. This ensures the agent only learns from high-quality, efficient cleaning trajectories.

Datasets

The study does not use external datasets. Instead, the authors developed a custom simulation environment using Tkinter (a Python GUI toolkit).

  • Training Environment: A 5x5 grid (25 tiles).
  • Testing/Evaluation Environment: A 20x20 grid (400 tiles), representing a typical $100m^2$ living room area.

Key Insights and Ideas

  • Local vs. Global Information: The most significant insight is that a cleaning robot doesn't need to "see" the whole house to clean efficiently. By combining a local view (for immediate obstacles) with a global hint (the DNUT "compass"), the agent achieves better generalization than models that try to process the entire map at once.
  • The Cost of Turning: The authors explicitly penalize rotations (-0.5 reward per 45-degree turn). This forces the RL agent to prioritize long, straight paths, which translates directly to lower electricity consumption in real-world hardware.
  • Ablation Success: The ablation study proved that the "original" PPO struggled significantly. Only when all four techniques (TL, DNUT, RS, and ES) were combined did the agent achieve rapid convergence and high cleaning coverage.

Key Results

The proposed method, referred to as ALL(PPO), was compared against "Random" and "Zigzag" (the industry standard) movements:

  • Travel Distance: The ALL(PPO) model was 3.61% more efficient than the Zigzag method and a staggering 17,438% better than Random movement.
  • Rotation Efficiency: The model reduced unnecessary turns by 6.17% compared to the Zigzag method.
  • Generalization: A model trained on a tiny 5x5 grid was able to successfully clean a 20x20 grid with high efficiency, proving the effectiveness of the transfer learning approach.

Limitations and Future Work

  • Grid-Based Simplification: The simulation operates on a discrete grid. Real-world environments are continuous, and robots must deal with non-grid-aligned obstacles and sensor noise (e.g., LiDAR or camera errors).
  • Static Obstacles: The current simulation uses fixed walls and obstacles. Future work could involve dynamic obstacles (like moving pets or humans) to test the agent's real-time reactivity.
  • Hardware Validation: While the simulation results are promising for energy efficiency, the paper does not include tests on physical robotic hardware to measure actual battery draw.

Summary updated

Abstract

Recently, as the demand for cleaning robots has steadily increased, therefore household electricity consumption is also increasing. To solve this electricity consumption issue, the problem of efficient path planning for cleaning robot has become important and many studies have been conducted. However, most of them are about moving along a simple path segment, not about the whole path to clean all places. As the emerging deep learning technique, reinforcement learning (RL) has been adopted for cleaning robot. However, the models for RL operate only in a specific cleaning environment, not the various cleaning environment. The problem is that the models have to retrain whenever the cleaning environment changes. To solve this problem, the proximal policy optimization (PPO) algorithm is combined with an efficient path planning that operates in various cleaning environments, using transfer learning (TL), detection nearest cleaned tile, reward shaping, and making elite set methods. The proposed method is validated with an ablation study and comparison with conventional methods such as random and zigzag. The experimental results demonstrate that the proposed method achieves improved training performance and increased convergence speed over the original PPO. And it also demonstrates that this proposed method is better performance than conventional methods (random, zigzag).
BibTeX
@inproceedings{moon2022path,
  author = {Woohyeon Moon and Bumgeun Park and Sarvar Hussain Nengroo and Taeyoung Kim and Dongsoo Har},
  title = {{Path Planning of Cleaning Robot with Reinforcement Learning}},
  booktitle = {ROSE},
  year = {2022},
  url = {https://openreview.net/forum?id=P83sgnNaHb},
}

Loading related content...

Figure