Abstract
This paper investigates the use of Deep Reinforcement Learning (DRL) for the continuous control of flippers on a multi-terrain tracked robot. By employing the Deep Deterministic Policy Gradient (DDPG) algorithm, the study aims to enable a robot to navigate complex obstacles like stairs using end-to-end learning from raw sensor data. While the agent successfully acquires basic locomotion skills, the research highlights the difficulties of high-dimensional physical tasks, including local minima and the limitations of feed-forward architectures in partially observable environments.
Introduction
Robotic locomotion on uneven terrain is a critical capability for search-and-rescue operations and advanced mobility aids, such as the B-Free motorized wheelchair. Tracked robots often utilize "flippers"—movable tracked appendages—to overcome obstacles. However, manual control of these components is cognitively taxing, and traditional rule-based automation often fails in unpredictable real-world scenarios.
This research addresses the gap between Reinforcement Learning (RL) successes in fully observable, discrete domains and the requirements of real-world robotics: continuous control in partially observable environments. The authors propose an end-to-end approach where a robot learns to "sense" its way up stairs by mapping raw depth and inertial data directly to motor actions.
Methodology
The researchers implemented the Deep Deterministic Policy Gradient (DDPG) algorithm, an actor-critic framework suited for continuous action spaces. The training was conducted within the V-Rep simulator using a model based on the B-Free wheelchair.
Sensor Fusion and Input
The state input combines visual and inertial data:
- Depth Maps: Four consecutive 34 × 34 pixel frames from both front and back depth cameras provide temporal context and a sense of motion.
- IMU Data: A 6-axis Inertial Measurement Unit provides acceleration and angular velocity.
Network Architecture
The study compared two neural network topologies to process the temporal depth data: 2D Convolutional Neural Networks (CNNs), which stack frames into input channels, and 3D CNNs, which attempt to preserve the temporal dimension through the layers.
Key Insights and Results
The experimental results yielded several critical observations regarding the training of DRL agents for physical locomotion:
- Architectural Performance: Contrary to expectations, 2D CNNs outperformed 3D CNNs. The 2D models converged faster and more reliably, whereas the 3D models struggled to optimize the policy, resulting in noisy Q-value predictions.
- Temporal Awareness: By feeding the network four sequential frames, the authors effectively "faked" memory in a feed-forward architecture. This allowed the robot to account for its own velocity and orientation changes without the complexity of recurrent layers.
- Safety and Stability: The reward function was designed to penalize excessive tilting and swinging. This successfully trained the policy to prioritize platform stability, a vital requirement for passenger-carrying robots.
Despite these successes, the agent frequently encountered local minima. A common failure mode involved the robot getting "stuck" in a specific stable pose (front flippers at +90°, rear at -90°) that prevented it from completing the climb. This underscores the challenge of exploration in high-dimensional action spaces.
Limitations and Future Directions
While the study demonstrates the feasibility of end-to-end DRL for flipper control, several areas for improvement were identified:
- Recurrent Modeling: The authors suggest that Recurrent Neural Networks (LSTMs) would likely handle the time-series nature of sensor data more effectively than stacked frames.
- Exploration Strategies: To overcome local minima, future work might explore imitation learning—training on human demonstrations—or discrete action spaces to simplify the initial learning phase.
- Real-World Transfer: Moving from simulation to physical hardware remains a significant hurdle, requiring robust handling of sensor noise and mechanical variances.
Summary updated