Fusion Recurrent Neural Network

Yiwen Sun, Yulu Wang, Kun Fu, Zheng Wang, Changshui Zhang, Jieping Ye

Abstract

Recurrent Neural Networks (RNNs) have long relied on complex gating mechanisms, such as those in LSTMs and GRUs, to manage information flow in sequence learning. This paper introduces the Fusion Recurrent Neural Network (Fusion RNN), a "gate-free" architecture that replaces intricate gates with a multi-round interactive fusion module. By focusing on the deep interaction between inputs and hidden states rather than blocking information flow, Fusion RNN achieves state-of-the-art performance with significantly lower parameter counts and faster inference. Evaluated on a massive real-world dataset of 500 million trajectories for Estimated Time of Arrival (ETA) prediction, Fusion RNN consistently outperforms traditional gated RNNs.

Introduction

For decades, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) have been the standard for sequence modeling. These architectures use input, forget, and output gates to prevent the vanishing gradient problem and control memory. However, these gates introduce significant computational overhead and increase the number of model parameters.

The authors of this research challenge the necessity of these complex gates. They argue that the fundamental purpose of a gate is to facilitate the interaction between the current input ($x_t$) and the previous hidden state ($h_{t-1}$). By simplifying this "fusion" process, it is possible to build more efficient models that maintain, or even exceed, the accuracy of their gated counterparts.

The Fusion RNN Architecture

The core innovation of Fusion RNN is the separation of the recurrent step into two distinct modules: the Fusion Module and the Transport Module.

1. Fusion Module

Instead of using gates to filter information, this module performs multi-round interactive fusion. Through element-wise multiplication and activation functions, the input vector learns to incorporate information from the hidden state, and vice versa. This iterative process is controlled by a hyperparameter $r$ (the number of fusion rounds), ensuring a deep "mixing" of features before they are processed further.

2. Transport Module

This module acts as a simplified recurrent unit, similar to an Elman network. It takes the fused vectors from the previous module and calculates the new hidden state ($h_t$) using a standard $tanh$ activation function.

The Fusion RNN cell architecture, highlighting the multi-round Fusion Module and the Transport Module.

When the input and hidden sizes are equal, Fusion RNN uses approximately half the parameters of an LSTM. This efficiency allows for faster training and deployment in high-throughput environments.

Case Study: Estimated Time of Arrival (ETA)

To demonstrate the model's efficacy, the authors developed FusionRNN-ETA, a specialized model for real-time traffic navigation. This model integrates non-sequential features (such as Driver ID and Time of Day) with sequential road-link data. These features are passed through an embedding layer and a Multi-Layer Perceptron (MLP) before being processed by the Fusion RNN.

Dataset: Beijing 2018

The model was rigorously tested on the Beijing 2018 dataset provided by DiDi Chuxing. This massive dataset contains over 500 million trajectories of taxi drivers in Beijing collected over four months. The data includes GPS coordinates, road link IDs, travel speeds, and timestamps. To ensure generalizability, the data was split chronologically: 16 weeks for training, and 2 weeks each for validation and testing.

Performance Evaluation

In head-to-head comparisons for ETA prediction, Fusion RNN outperformed all baselines, including LSTM and GRU, across multiple error metrics.

  • Accuracy: Fusion RNN achieved a Mean Absolute Percentage Error (MAPE) of 19.579%, surpassing LSTM (19.598%) and GRU (19.673%).
  • Error Reduction: It reduced the Mean Absolute Error (MAE) by approximately 0.38% and the Root Mean Square Error (RMSE) by 0.21% compared to LSTM.
  • The Impact of Iteration ($r$): The authors found that the number of fusion rounds significantly impacts performance. Interestingly, even with just two rounds ($r=2$), the model already becomes more effective than its gated counterparts. As long as $r < 6$, Fusion RNN remains computationally more efficient than an LSTM.
Comparison of MAPE (%) across different fusion rounds (r) against LSTM and GRU baselines.

Conclusion

Fusion RNN represents a shift toward simpler, more interactive recurrent architectures. By demonstrating that gates are not mandatory for high-performance sequence learning, the authors provide a lightweight alternative for engineering tasks that require both high accuracy and low latency. While currently validated on spatiotemporal data for ETA prediction, future work will explore the model's performance in other domains like Natural Language Processing (NLP) and Speech Recognition to determine its universal applicability.

Summary updated

Abstract

Considering deep sequence learning for practical application, two representative RNNs - LSTM and GRU may come to mind first. Nevertheless, is there no chance for other RNNs? Will there be a better RNN in the future? In this work, we propose a novel, succinct and promising RNN - Fusion Recurrent Neural Network (Fusion RNN). Fusion RNN is composed of Fusion module and Transport module every time step. Fusion module realizes the multi-round fusion of the input and hidden state vector. Transport module which mainly refers to simple recurrent network calculate the hidden state and prepare to pass it to the next time step. Furthermore, in order to evaluate Fusion RNN's sequence feature extraction capability, we choose a representative data mining task for sequence data, estimated time of arrival (ETA) and present a novel model based on Fusion RNN. We contrast our method and other variants of RNN for ETA under massive vehicle travel data from DiDi Chuxing. The results demonstrate that for ETA, Fusion RNN is comparable to state-of-the-art LSTM and GRU which are more complicated than Fusion RNN.
BibTeX
@misc{sun2020fusion,
  author = {Yiwen Sun and Yulu Wang and Kun Fu and Zheng Wang and Changshui Zhang and Jieping Ye},
  title = {{Fusion Recurrent Neural Network}},
  year = {2020},
  eprint = {2006.04069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.04069},
}
Figure