Abstract
As the demand for cleaning robots increases, household electricity consumption has become a significant concern. Efficient path planning is essential to mitigate this issue; however, most existing studies focus on simple path segments rather than comprehensive coverage. While Reinforcement Learning (RL) has been proposed as a solution, standard models often fail to generalize, requiring retraining whenever the cleaning environment changes. This paper presents an enhanced path planning framework using the Proximal Policy Optimization (PPO) algorithm combined with Transfer Learning (TL), Detection of Nearest Uncleaned Tile (DNUT), Reward Shaping (RS), and an Elite Set (ES) method. Our experimental results demonstrate that this approach achieves superior training performance and convergence speed compared to standard PPO and outperforms conventional methods like random and zigzag patterns in energy efficiency.
Introduction
Modern cleaning robots often rely on rudimentary pathfinding algorithms, such as "straight-until-obstacle" or basic zigzag patterns. While functional, these methods are frequently inefficient, leading to redundant travel and excessive battery consumption. Reinforcement Learning (RL) offers a more adaptive alternative, yet it faces a critical "generalization gap." Typically, an RL agent trained in a specific room layout cannot transfer its knowledge to a different room size or configuration without extensive retraining. This research addresses the challenge of creating a universal path planning model that maintains efficiency across diverse environments without the need for environment-specific tuning.
Methodology
The proposed framework utilizes Proximal Policy Optimization (PPO) as its core algorithm, chosen for its stability in complex state spaces. To enable generalization and improve efficiency, four specific techniques were integrated:
1. Transfer Learning (TL) via Local Windows
To solve the problem of varying map sizes, the researchers implemented a local observation window. Instead of processing the entire map, the agent only receives data from the 8 tiles immediately surrounding it. This keeps the input state constant, allowing a model trained on a small grid to function seamlessly on much larger maps.
2. Detection of Nearest Uncleaned Tile (DNUT)
Local views alone can lead to an agent getting stuck in already-cleaned areas. DNUT provides the agent with the relative coordinates of the closest uncleaned tile. This acts as a global "compass," guiding the robot toward remaining work areas even when they are outside its immediate local window.
3. Reward Shaping (RS) with Stacked Values
To encourage efficient, continuous cleaning, the authors designed an exponential reward system. When the agent cleans consecutive tiles, the reward increases according to the formula $1.5^n$. To discourage inefficient movement, the reward streak resets if the robot hits a wall or enters a previously cleaned tile. Furthermore, a penalty of -0.5 is applied for every 45-degree turn to minimize energy-intensive rotations.
4. Elite Set (ES)
To accelerate convergence, the training process employs an Elite Set method. Data from episodes that exceed 500 steps—indicating poor pathfinding—are discarded. This ensures the neural network only learns from high-quality, efficient trajectories, significantly improving the learning curve.
Experimental Setup
The researchers developed a custom simulation environment using the Python Tkinter library. The model was trained in a constrained 5x5 grid (25 tiles) to test its ability to learn basic logic quickly. It was then evaluated in a much larger 20x20 grid (400 tiles), simulating a standard $100m^2$ living space, to test its generalization capabilities.
Results and Discussion
The proposed method, termed ALL(PPO), was benchmarked against random movement and the industry-standard zigzag algorithm.
- Efficiency Gains: The ALL(PPO) model demonstrated a 3.61% improvement in travel distance over the zigzag method. Compared to random movement, the efficiency gain was over 17,000%.
- Energy Conservation: By penalizing rotations, the model reduced unnecessary turns by 6.17% compared to the zigzag approach. In physical hardware, fewer rotations translate directly to lower peak current draw and extended battery life.
- Generalization Success: The ablation study confirmed that standard PPO failed to converge effectively in varying environments. Only the combination of TL, DNUT, RS, and ES allowed the agent to successfully navigate the 20x20 environment after being trained only on the 5x5 grid.
Conclusion
This research provides a robust framework for generalized robot path planning. By combining local observations with global hints and specialized reward structures, the model moves beyond the limitations of environment-specific training. While the current study is limited to a discrete grid-based simulation, it lays the groundwork for more energy-efficient, autonomous cleaning systems. Future work will focus on adapting this model to continuous environments and testing its performance against dynamic obstacles and real-world sensor noise.
Summary updated