Dynamic Vehicle Traffic Control Using Deep Reinforcement Learning in Automated Material Handling System

Younkook Kang, Sungwon Lyu, Jeeyung Kim, Bongjoon Park, Sungzoon ChoAAAI 2019 · OpenReview

Abstract: In automated material handling systems (AMHS), delivery time is an important issue directly associated with the production cost and the quality of the product. In this paper, we propose a dynamic routing strategy to shorten delivery time and delay. We set the target of control by analyzing traffic flows and selecting the region with the highest flow rate and congestion frequency. Then, we impose a routing cost in order to dynamically reflect the real-time changes of traffic states. Our deep reinforcement learning model consists of a Q-learning step and a recurrent neural network, through which traffic states and action values are predicted. Experiment results show that the proposed method decreases manufacturing costs while increasing productivity. Additionally, we find evidence the reinforcement learning structure proposed in this study can autonomously and dynamically adjust to the changes in traffic patterns.

Introduction

In high-tech manufacturing environments, such as semiconductor fabrication, Automated Material Handling Systems (AMHS) serve as the critical infrastructure for material transport. These systems involve hundreds of transfer robots moving materials between machines across expansive layouts. Even minor delays in these deliveries can lead to significant production bottlenecks and increased operational costs.

The primary challenge in AMHS management is erratic congestion. Traditional traffic control methods often rely on static routing or optimization techniques that struggle to adapt to the "bursty" and unpredictable nature of factory traffic. While deep learning has shown promise in predicting traffic patterns, applying it to real-time control in complex manufacturing environments remains a significant hurdle. This research aims to bridge this gap by developing a system that senses real-time traffic states and autonomously adjusts routing costs to steer vehicles away from emerging bottlenecks.

Methods

The proposed Deep Reinforcement Learning (DRL) framework is specifically tailored for the constraints of a factory floor through several key components:

  1. Targeted Traffic Flow Analysis: To maintain computational efficiency, the researchers identified the 40 most critical bottlenecks out of 2,000 possible paths, focusing control efforts where they are most impactful.
  2. State Representation: The model utilizes time-series data, including flow rates, vehicle velocity, velocity deviation, and congestion duration, to represent the current state of the system.
  3. Hybrid RL Architecture:
    • RNN Component: A Recurrent Neural Network (RNN) processes time-series state data to capture the temporal dependencies inherent in traffic build-up.
  4. Q-Learning: The model selects from three candidate routing cost values. By dynamically increasing the cost of a congested path, the system encourages vehicle controllers to seek alternative routes.
  5. Reward Function: The agent is optimized based on a linear combination of reduced delivery time and minimized delay counts.

Experimental Setup

The study utilized a simulation environment modeled after an actual semiconductor factory. The layout included 50 bays and 500 vehicles, supporting a transfer rate of 8,000 loads per hour. The model was trained on 24 hours of historical transfer event data and evaluated using a separate 8-hour block of event data.

Key Results

The dynamic DRL model was compared against a "baseline" (no specific traffic weights) and a "static" model (optimized but fixed weights). The results demonstrate that the dynamic approach consistently outperforms traditional methods.

Performance comparison of routing strategies.
  • Delivery Time: The dynamic model reduced average delivery time by approximately 0.7 seconds compared to the optimal static model. In a system performing 8,000 transfers per hour, these small individual gains compound into significant overall productivity improvements.
  • Delay Counts: The most significant improvement was a 7.7% reduction in delay counts (defined as 30-second delays in total sections) compared to the static model.
  • Autonomy: The system demonstrated a capacity to autonomously adjust to changing traffic patterns with minimal human interference, fulfilling a core requirement for modern "Smart Factory" operations.

Conclusion and Future Work

This study highlights the effectiveness of targeted control and temporal awareness in managing AMHS traffic. By focusing on key bottlenecks and integrating an RNN to recognize congestion patterns, the DRL agent successfully smoothed global traffic flow.

Future research may explore expanding the action space to continuous values for finer control and testing the model's generalizability across different factory architectures. Additionally, investigating the scalability of the RNN-RL loop for "Mega-fabs" operating thousands of vehicles remains an important area for further study.

Summary updated

Abstract

In automated material handling systems (AMHS), delivery time is an important issue directly associated with the production cost and the quality of the product. In this paper, we propose a dynamic routing strategy to shorten delivery time and delay. We set the target of control by analyzing traffic flows and selecting the region with the highest flow rate and congestion frequency. Then, we impose a routing cost in order to dynamically reflect the real-time changes of traffic states. Our deep reinforcement learning model consists of a Q-learning step and a recurrent neural network, through which traffic states and action values are predicted. Experiment results show that the proposed method decreases manufacturing costs while increasing productivity. Additionally, we find evidence the reinforcement learning structure proposed in this study can autonomously and dynamically adjust to the changes in traffic patterns.
BibTeX
@inproceedings{kang2019dynamic,
  author = {Younkook Kang and Sungwon Lyu and Jeeyung Kim and Bongjoon Park and Sungzoon Cho},
  title = {{Dynamic Vehicle Traffic Control Using Deep Reinforcement Learning in Automated Material Handling System}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=suYBpRfpa7e},
}
Figure