Introduction
Search-and-rescue (S&R) operations are inherently time-sensitive, demanding rapid deployment and adaptation from robotic agents. While Reinforcement Learning (RL) offers a powerful framework for autonomous decision-making, its high sample complexity often makes it impractical for real-world robotics. Training a robot from scratch for every new environment leads to significant time costs and physical wear. This paper addresses the challenge of Lifelong Learning, exploring how robots can leverage a library of knowledge from previous tasks to accelerate learning in new, related environments—such as transitioning from a smooth floor to a high-friction surface like sand or carpet.
Methodology
The researchers evaluated three RL paradigms on a Turtlebot 2 platform to determine the most effective approach for S&R tasks:
- Q-Learning: A discrete RL algorithm using a simplified state space (relative distance and angle to target) and basic actions (Forward, Left, Right).
- Policy Gradient (PG): A method capable of handling continuous state and action spaces, allowing for more fluid control of the robot's linear and angular velocities.
- PG-ELLA (Policy Gradient Efficient Lifelong Learning Algorithm): The core contribution, which extends PG into a multi-task setting. It maintains a shared latent basis ($L$) that acts as a "knowledge bank." New tasks are modeled as linear combinations of these shared components, facilitating rapid adaptation.
To further enhance training efficiency, the team incorporated Learning from Demonstration. A human operator provides initial "bootstrap" trajectories via teleoperation, which the RL agent then refines.
System Implementation
The experimental setup utilized the Gazebo simulator integrated with ROS and MATLAB. The robot was tasked with reaching a target object across five different environments, each defined by a unique ground friction coefficient ($\mu_t$). This variation simulates the diverse terrains encountered in actual S&R missions. The integration of these disparate software components was a significant technical hurdle, as the MATLAB/ROS interface proved computationally demanding.
Results and Discussion
The study yielded several key insights into the performance and scalability of RL in robotics:
- Continuous Control Advantage: Policy Gradient methods significantly outperformed Q-learning. The PG agent converged to a near-optimal policy in approximately 400 episodes, whereas Q-learning required nearly 1,700 episodes to reach similar performance levels.
- State Space Complexity: The "curse of dimensionality" was evident in Q-learning tests. Increasing the state space from 2D (relative coordinates) to 3D (absolute coordinates) severely degraded performance, with the 3D agent struggling to achieve the same reward efficiency as the 2D model.
- Knowledge Transfer: PG-ELLA successfully demonstrated a functional framework for multi-task learning. By treating robotic skills as modular building blocks, the agent can combine existing knowledge with task-specific parameters to adapt to new friction conditions more efficiently than learning from scratch.
Conclusion and Future Work
While hardware and computational limitations restricted the scale of empirical analysis for PG-ELLA, the researchers successfully deployed a functional lifelong learning framework on a physical robot platform (simulated via Gazebo). Future work will focus on increasing environmental complexity—such as adding obstacles and mazes—and incorporating more sophisticated sensor data like visual perception and GPS to improve the robot's situational awareness in unpredictable S&R environments.
Summary updated