Abstract
The existing railway wagon flow forecast method cannot meet the demand of the actual railway transport organization. The old system is embodied in the slow computing speed and low accuracy. The railway wagon flow forecast system based on Hadoop-Hazelcast is designed to improve the accuracy of the forecast, shorten the calculation time and expand the scale of calculation. Using the Hadoop framework to analyze history wagon flow data to get the wagon flow data feature, using Hazelcast architecture to solve IO bottlenecks, the new forecast system integrates the scattered memory into memory computing cluster to compute time window in parallel. The results show that this method and system can greatly improve the forecast accuracy and speed.
Introduction: The Challenge of Scale in Railway Logistics
In the complex world of logistics, Wagon Flow Forecast of Freight Transport (WFFFT) serves as the critical heartbeat of railway operations. In China, this task involves tracking approximately 880,000 wagons across a network of more than 6,000 stations. The scale of data generated is immense: the system produces 10 million records daily (roughly 4GB), and historical data archives reach the Petabyte (PT) level.
Traditional forecasting methods have struggled to keep pace with this volume. Legacy systems suffer from severe I/O bottlenecks and slow computing speeds, often requiring more than 24 hours to complete a single forecast cycle. This significant lag prevents railway managers from responding effectively to real-time network changes. To transition toward an "intelligent railway," a new architecture was required—one capable of processing massive historical datasets while simultaneously handling high-frequency, real-time data streams.
Methodology: A Hybrid Hadoop-Hazelcast Architecture
The proposed system addresses these challenges through a two-tier architecture that leverages the complementary strengths of Hadoop and Hazelcast.
Historical Feature Extraction with Hadoop
The system utilizes the Hadoop framework to perform intensive batch processing on historical XML data. Through a Map-Reduce workflow, the system slices and sorts historical wagon flows to identify key features, such as typical travel cycles and station-specific delay patterns. This analysis provides the foundational "intelligence" required for accurate forecasting.
Real-Time In-Memory Processing with Hazelcast
To overcome the disk I/O bottlenecks inherent in traditional databases, the system employs Hazelcast, an in-memory data grid. Dynamic "on-net" wagon data is ingested via message queues at a rate of approximately 50 messages per second. This data is serialized into objects and stored within a distributed memory cluster.
The forecasting process follows a structured 7-step workflow, as illustrated in Figure 1, ranging from basic data initialization to the final forecast output, which includes wagon counts in stations and moment forecasts.
Parallel Time-Window Computing
Rather than treating the entire network as a single monolithic task, the system divides the forecast into "time windows" (hourly units). By integrating scattered memory into a unified computing cluster, the system calculates these windows in parallel. This approach allows for nearly instantaneous updates to wagon positions and Origin-Destination (OD) times.
As shown in Figure 6, the Hazelcast architecture distributes data across a RAM storage cluster, where entry processors handle the wagon flow data stream in parallel across multiple nodes.
Key Results and Performance Gains
The transition to a hybrid Hadoop-Hazelcast system has yielded substantial performance improvements:
- Computational Speed: The traditional calculation method required over 24 hours to process the wagon time window for the entire fleet. The new system completes this task in just 18 minutes (0.3 hours), representing a massive reduction in latency.
- Accuracy: Forecast results have shown significant improvement when compared against actual statistics from the China Railway Corporation. By moving from aggregate flow statistics to individual "Car No." tracking, the system achieves much higher precision in predicting arrival and departure moments.
- Scalability: The system successfully manages a high-frequency environment, with calculation frequencies reaching 100,000 beats per minute.
Conclusion and Future Work
The Hadoop-Hazelcast hybrid approach effectively eliminates the primary bottlenecks of previous railway forecasting systems. By bypassing disk I/O and utilizing parallel in-memory computing, the system enables real-time tracking and forecasting at a national scale.
Future research will focus on further optimizing the integration between the Hadoop HDFS namespace and the live Hazelcast grid to reduce the current "hysteresis" or lag in two-level unit storage. Additionally, there is potential to integrate machine learning models to better account for stochastic variables such as weather-related delays or equipment failures, further enhancing the system's predictive capabilities.
Summary updated