Abstract
Recurrent Neural Networks (RNNs) have long relied on complex gating mechanisms, such as those in LSTMs and GRUs, to manage information flow in sequence learning. This paper introduces the Fusion Recurrent Neural Network (Fusion RNN), a "gate-free" architecture that replaces intricate gates with a multi-round interactive fusion module. By focusing on the deep interaction between inputs and hidden states rather than blocking information flow, Fusion RNN achieves state-of-the-art performance with significantly lower parameter counts and faster inference. Evaluated on a massive real-world dataset of 500 million trajectories for Estimated Time of Arrival (ETA) prediction, Fusion RNN consistently outperforms traditional gated RNNs.
Introduction
For decades, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) have been the standard for sequence modeling. These architectures use input, forget, and output gates to prevent the vanishing gradient problem and control memory. However, these gates introduce significant computational overhead and increase the number of model parameters.
The authors of this research challenge the necessity of these complex gates. They argue that the fundamental purpose of a gate is to facilitate the interaction between the current input ($x_t$) and the previous hidden state ($h_{t-1}$). By simplifying this "fusion" process, it is possible to build more efficient models that maintain, or even exceed, the accuracy of their gated counterparts.
The Fusion RNN Architecture
The core innovation of Fusion RNN is the separation of the recurrent step into two distinct modules: the Fusion Module and the Transport Module.
1. Fusion Module
Instead of using gates to filter information, this module performs multi-round interactive fusion. Through element-wise multiplication and activation functions, the input vector learns to incorporate information from the hidden state, and vice versa. This iterative process is controlled by a hyperparameter $r$ (the number of fusion rounds), ensuring a deep "mixing" of features before they are processed further.
2. Transport Module
This module acts as a simplified recurrent unit, similar to an Elman network. It takes the fused vectors from the previous module and calculates the new hidden state ($h_t$) using a standard $tanh$ activation function.
When the input and hidden sizes are equal, Fusion RNN uses approximately half the parameters of an LSTM. This efficiency allows for faster training and deployment in high-throughput environments.
Case Study: Estimated Time of Arrival (ETA)
To demonstrate the model's efficacy, the authors developed FusionRNN-ETA, a specialized model for real-time traffic navigation. This model integrates non-sequential features (such as Driver ID and Time of Day) with sequential road-link data. These features are passed through an embedding layer and a Multi-Layer Perceptron (MLP) before being processed by the Fusion RNN.
Dataset: Beijing 2018
The model was rigorously tested on the Beijing 2018 dataset provided by DiDi Chuxing. This massive dataset contains over 500 million trajectories of taxi drivers in Beijing collected over four months. The data includes GPS coordinates, road link IDs, travel speeds, and timestamps. To ensure generalizability, the data was split chronologically: 16 weeks for training, and 2 weeks each for validation and testing.
Performance Evaluation
In head-to-head comparisons for ETA prediction, Fusion RNN outperformed all baselines, including LSTM and GRU, across multiple error metrics.
- Accuracy: Fusion RNN achieved a Mean Absolute Percentage Error (MAPE) of 19.579%, surpassing LSTM (19.598%) and GRU (19.673%).
- Error Reduction: It reduced the Mean Absolute Error (MAE) by approximately 0.38% and the Root Mean Square Error (RMSE) by 0.21% compared to LSTM.
- The Impact of Iteration ($r$): The authors found that the number of fusion rounds significantly impacts performance. Interestingly, even with just two rounds ($r=2$), the model already becomes more effective than its gated counterparts. As long as $r < 6$, Fusion RNN remains computationally more efficient than an LSTM.
Conclusion
Fusion RNN represents a shift toward simpler, more interactive recurrent architectures. By demonstrating that gates are not mandatory for high-performance sequence learning, the authors provide a lightweight alternative for engineering tasks that require both high accuracy and low latency. While currently validated on spatiotemporal data for ETA prediction, future work will explore the model's performance in other domains like Natural Language Processing (NLP) and Speech Recognition to determine its universal applicability.
Summary updated