Abstract
An effective way to achieve intelligence is to simulate various intelligent behaviors in the human brain. In recent years, bio-inspired learning methods have emerged, and they are different from the classical mathematical programming principle. In the perspective of brain inspiration, reinforcement learning has gained additional interest in solving decision-making tasks as increasing neuroscientific research demonstrates that significant links exist between reinforcement learning and specific neural substrates. Because of the tremendous research that focuses on human brains and reinforcement learning, scientists have investigated how robots can autonomously tackle complex tasks in the form of a self-driving agent control in a human-like way. In this study, we propose an end-to-end architecture using novel deep-Q-network architecture in conjunction with a recurrence to resolve the problem in the field of simulated self-driving. The main contribution of this study is that we trained the driving agent using a brain-inspired trial-and-error technique, which was in line with the real world situation. Besides, there are three innovations in the proposed learning network: raw screen outputs are the only information which the driving agent can rely on, a weighted layer that enhances the differences of the lengthy episode, and a modified replay mechanism that overcomes the problem of sparsity and accelerates learning. The proposed network was trained and tested under a third-partied OpenAI Gym environment. After training for several episodes, the resulting driving agent performed advanced behaviors in the given scene. We hope that in the future, the proposed brain-inspired learning system would inspire practicable self-driving control solutions.
Introduction
Traditional autonomous driving systems often rely on supervised learning, which requires extensive handcrafted datasets of "perfect" driving. However, these systems struggle with "out-of-distribution" scenarios because they lack experience in recovering from suboptimal states, such as drifting off-track. This paper proposes a brain-inspired approach using Deep Recurrent Reinforcement Learning (DRRL) to train a self-driving agent through trial and error, mimicking human learning processes and improving robustness in partially observable environments.
Methodology
The core of the proposed system is a Deep Recurrent Q-Network (DRQN). Unlike standard DQNs, this architecture incorporates temporal context, which is crucial for driving tasks where a single frame cannot convey momentum or hidden obstacles.
Architectural Innovations
- Feature Extraction: A four-layer Convolutional Neural Network (CNN) processes raw 320x240 grayscale screen outputs, ensuring the agent relies solely on visual information.
- Temporal Memory: An LSTM layer with 900 units replaces the standard fully connected layer, allowing the agent to integrate historical observations into its current decision-making process.
- The Weighting Layer: The network splits into two collateral streams—a Value Stream and an Advantage Stream. These are combined to produce the final Q-value, helping the network focus on precise instantaneous changes in the environment.
- Prioritized Experience Replay: To address the "sparse reward" problem, the system stores and more frequently samples experiences with high learning potential.
Experimental Results
The agent was trained and tested in a 3D car racing environment within OpenAI Gym across three tracks: Farm, Raceway, and Mountain. The model's performance was benchmarked against a supervised model and human players.
Key Findings
- Robustness: While supervised models achieved fast times on ideal lines, they frequently recorded "Did Not Finish" (DNF) results on complex tracks like "Mountain" because they couldn't navigate back to the road after a mistake. In contrast, the DRQN agent finished every race, demonstrating superior recovery capabilities.
- Competitive Performance: On the "Farm" track, the model achieved a mean time of 98.33s, very close to human performance (94.07s).
- Learning Stability: As shown in the training curves, the agent's rewards stabilize at a high level after approximately 1,400 episodes, following an initial exploration phase governed by an $\epsilon$-greedy strategy.
Conclusion
This research demonstrates that a brain-inspired RL approach, relying solely on raw visual input and temporal memory, can produce robust self-driving behavior that surpasses traditional supervised models in challenging scenarios. Future work will focus on incorporating full RGB input and continuous action spaces to achieve smoother driving and better environmental recognition.
Summary updated