Hybrid of representation learning and reinforcement learning for dynamic and complex robotic motion planning

Chengmin Zhou, Xin Lu, Jiapeng Dai, Xiaoxu Liu, Bingding Huang, Pasi FräntiRobotics Auton. Syst 2023

Abstract

Navigating dense, dynamic environments remains a significant challenge in robotics. While Deep Reinforcement Learning (DRL) offers potential for time-sequential decision-making, standard models often struggle with state representation and generalization. This paper introduces LSA-DSAC, a hybrid algorithm that enhances the Discrete Soft Actor-Critic (DSAC) framework with a sophisticated representation learning encoder. By combining attention mechanisms, skip connections, and LSTM pooling, the proposed method achieves a 99.6% success rate in complex simulations and demonstrates robust sim-to-real transferability.

Introduction

Robotic motion planning in crowded, unpredictable spaces—such as delivery bots in busy lobbies—requires a delicate balance between safety and efficiency. Traditional algorithms like A* are often restricted to static maps, while reactive methods like Optimal Reciprocal Collision Avoidance (ORCA) can be overly conservative or "myopic" at high speeds. DRL has emerged as a powerful alternative, but its performance is heavily dependent on the "expressive power" of the environmental state representation. Poor representations lead to slow convergence and overfitting to specific training scenarios.

Methodology: The LSA-DSAC Framework

The core contribution of this work is the LSA-DSAC algorithm, which optimizes the representation of obstacles through a four-step architectural evolution of the encoder:

  1. Attention-Weighted (AW) Encoding: Instead of treating all obstacles equally, an attention mechanism weighs obstacles based on their relative speed, distance, and direction.
  2. Skip Connections (SA): To prevent the loss of fine-grained environmental details during deep processing, the authors integrated skip connections. This allows the model to concatenate raw input features with high-level attention features, mitigating overfitting.
  3. LSTM Pooling (LSA): Traditional attention models often use simple summation to aggregate obstacle data. However, summation lacks "injectivity"—different obstacle configurations can result in the same sum. By replacing summation with Long Short-Term Memory (LSTM) pooling, the model preserves the structural and statistical nuances of the obstacle field.
  4. Discrete Soft Actor-Critic (DSAC): These enhanced representations feed into a DSAC framework, where the policy and critic use separate attention networks to improve convergence.
Evolution of the encoder architecture from basic Relational Graph (RG) to the final LSTM-SkipCon-Attention (LSA) model.

Experimental Evaluation and Results

The model was rigorously tested in both simulated and physical environments. Training was primarily conducted in a "Circle-crossing" simulator, where obstacles converge toward the center, creating high-density conflict zones.

Performance Metrics

LSA-DSAC demonstrated state-of-the-art performance across several key metrics:

  • Success Rate: The model achieved a 99.6% success rate in 5-obstacle scenarios, outperforming SARL (99.0%) and vastly exceeding classical ORCA (43%).
  • Safety: It recorded the lowest collision rate (0.004) among all tested algorithms.
  • Generalization: In a "Square-crossing" transferability test, LSA-DSAC maintained a 93% success rate, proving its ability to handle unseen environmental geometries.
Overview of the circle-crossing environment, the iterative improvement of the algorithm, and the resulting reward convergence compared to baseline models.

Real-World Validation

The system was deployed on a physical mobile robot equipped with a Jetson Nano and an optical motion capture system. The robot successfully navigated through both static and dynamic human obstacles, confirming that the LSA-DSAC model effectively bridges the "sim-to-real" gap.

Discussion and Conclusion

The success of LSA-DSAC highlights the importance of "injectivity" in feature aggregation and the role of skip connections in maintaining environmental context. While the increased complexity of the LSA encoder leads to slightly slower initial training convergence, the final performance gains in safety and success rate are substantial. Future work will focus on improving the interpretability of the attention weights and exploring Bayesian models to infer hidden obstacle intentions to further refine navigation safety.

Summary updated

BibTeX
@inproceedings{zhou2023hybrid,
  author = {Chengmin Zhou and Xin Lu and Jiapeng Dai and Xiaoxu Liu and Bingding Huang and Pasi Fränti},
  title = {{Hybrid of representation learning and reinforcement learning for dynamic and complex robotic motion planning}},
  booktitle = {Robotics Auton. Syst},
  year = {2023},
}
Figure