Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning

Giuseppe Paolo, Lei Tai, Ming LiuarXiv 2017 · arXiv

Abstract

This paper investigates the use of Deep Reinforcement Learning (DRL) for the continuous control of flippers on a multi-terrain tracked robot. By employing the Deep Deterministic Policy Gradient (DDPG) algorithm, the study aims to enable a robot to navigate complex obstacles like stairs using end-to-end learning from raw sensor data. While the agent successfully acquires basic locomotion skills, the research highlights the difficulties of high-dimensional physical tasks, including local minima and the limitations of feed-forward architectures in partially observable environments.

Introduction

Robotic locomotion on uneven terrain is a critical capability for search-and-rescue operations and advanced mobility aids, such as the B-Free motorized wheelchair. Tracked robots often utilize "flippers"—movable tracked appendages—to overcome obstacles. However, manual control of these components is cognitively taxing, and traditional rule-based automation often fails in unpredictable real-world scenarios.

The simulation environment featuring a tracked robot model navigating a flight of stairs.

This research addresses the gap between Reinforcement Learning (RL) successes in fully observable, discrete domains and the requirements of real-world robotics: continuous control in partially observable environments. The authors propose an end-to-end approach where a robot learns to "sense" its way up stairs by mapping raw depth and inertial data directly to motor actions.

Methodology

The researchers implemented the Deep Deterministic Policy Gradient (DDPG) algorithm, an actor-critic framework suited for continuous action spaces. The training was conducted within the V-Rep simulator using a model based on the B-Free wheelchair.

Sensor Fusion and Input

The state input combines visual and inertial data:

  • Depth Maps: Four consecutive 34 × 34 pixel frames from both front and back depth cameras provide temporal context and a sense of motion.
  • IMU Data: A 6-axis Inertial Measurement Unit provides acceleration and angular velocity.

Network Architecture

The study compared two neural network topologies to process the temporal depth data: 2D Convolutional Neural Networks (CNNs), which stack frames into input channels, and 3D CNNs, which attempt to preserve the temporal dimension through the layers.

The Actor network architecture, illustrating the fusion of camera and IMU data into a dense policy layer.

Key Insights and Results

The experimental results yielded several critical observations regarding the training of DRL agents for physical locomotion:

  • Architectural Performance: Contrary to expectations, 2D CNNs outperformed 3D CNNs. The 2D models converged faster and more reliably, whereas the 3D models struggled to optimize the policy, resulting in noisy Q-value predictions.
  • Temporal Awareness: By feeding the network four sequential frames, the authors effectively "faked" memory in a feed-forward architecture. This allowed the robot to account for its own velocity and orientation changes without the complexity of recurrent layers.
  • Safety and Stability: The reward function was designed to penalize excessive tilting and swinging. This successfully trained the policy to prioritize platform stability, a vital requirement for passenger-carrying robots.

Despite these successes, the agent frequently encountered local minima. A common failure mode involved the robot getting "stuck" in a specific stable pose (front flippers at +90°, rear at -90°) that prevented it from completing the climb. This underscores the challenge of exploration in high-dimensional action spaces.

Limitations and Future Directions

While the study demonstrates the feasibility of end-to-end DRL for flipper control, several areas for improvement were identified:

  1. Recurrent Modeling: The authors suggest that Recurrent Neural Networks (LSTMs) would likely handle the time-series nature of sensor data more effectively than stacked frames.
  2. Exploration Strategies: To overcome local minima, future work might explore imitation learning—training on human demonstrations—or discrete action spaces to simplify the initial learning phase.
  3. Real-World Transfer: Moving from simulation to physical hardware remains a significant hurdle, requiring robust handling of sensor noise and mechanical variances.

Summary updated

Abstract

In this paper we focus on developing a control algorithm for multi-terrain tracked robots with flippers using a reinforcement learning (RL) approach. The work is based on the deep deterministic policy gradient (DDPG) algorithm, proven to be very successful in simple simulation environments. The algorithm works in an end-to-end fashion in order to control the continuous position of the flippers. This end-to-end approach makes it easy to apply the controller to a wide array of circumstances, but the huge flexibility comes to the cost of an increased difficulty of solution. The complexity of the task is enlarged even more by the fact that real multi-terrain robots move in partially observable environments. Notwithstanding these complications, being able to smoothly control a multi-terrain robot can produce huge benefits in impaired people daily lives or in search and rescue situations.
BibTeX
@misc{paolo2017towards,
  author = {Giuseppe Paolo and Lei Tai and Ming Liu},
  title = {{Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning}},
  year = {2017},
  eprint = {1709.08430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.08430},
}
Figure