Abstract
The Delayed Memory Unit (DMU) is a parameter-efficient alternative to traditional gated recurrent networks like LSTMs and GRUs. By incorporating a delay line and a specialized "delay gate," the DMU addresses the vanishing gradient problem through direct information projection into the future. Experimental results demonstrate that the DMU achieves state-of-the-art performance on complex temporal tasks, such as speech and gesture recognition, while utilizing up to 4x fewer parameters than standard models.
Introduction
Recurrent Neural Networks (RNNs) are fundamental for sequential data processing but face significant challenges. The vanishing and exploding gradient problems hinder the learning of long-range dependencies, as information is passed incrementally through time steps. While LSTMs and GRUs were developed to mitigate these issues, they are often over-parameterized, leading to high computational costs and potential overfitting on smaller datasets. The Delayed Memory Unit (DMU) addresses these limitations by providing a lean mechanism for long-term memory without the heavy parameter overhead.
Methods: The DMU Architecture
The core innovation of the DMU is its delay line structure, which replaces the standard incremental hidden state update with a direct projection mechanism.
The architecture consists of several key components:
- The Delay Gate ($d_t$): A learnable gate that determines how to distribute current information to specific future time instants.
- Sliding Window Memory ($m_t$): Instead of a single vector, the DMU maintains a window of future slots. The delay gate "projects" information into these slots. At each step, the network sums the current hidden state with information previously projected into the current time slot.
- Temporal Credit Assignment: This structure creates "shortcuts" for gradients, allowing them to flow back in time more effectively, similar to skip connections in spatial networks.
- Efficiency Strategies: To optimize memory, the authors utilize dilated delays (skipping steps) and thresholding to prune insignificant computations.
Experimental Evaluation
The DMU was evaluated across a diverse "decathlon" of temporal modeling tasks, including TIMIT phoneme recognition, Hey Snips wake-word detection, and the Spiking Heidelberg Dataset (SHD). It was also tested on the SoLi dataset for gesture recognition and the Permuted Sequential MNIST (PS-MNIST) benchmark for long-range logic.
Results and Key Insights
The DMU consistently outperformed traditional models with higher efficiency. On the PS-MNIST benchmark, the DMU achieved 96.39% accuracy with only 49K parameters, whereas an LSTM reached only 89.86% with 165K parameters. Similarly, on the SHD spiking speech task, the DMU (91.48% accuracy) significantly surpassed both Bi-LSTMs and standard LSTMs.
A notable insight is the "Recurrent Burden Relief." Analysis of weight magnitudes reveals that when a DMU is integrated, standard recurrent weights decrease, suggesting the delay line handles long-term memory while recurrent connections focus on immediate dynamics. Furthermore, the DMU exhibits high training stability, converging much faster than LSTMs without the need for advanced normalization techniques.
Conclusion
The Delayed Memory Unit offers a parameter-efficient solution for modeling long-term temporal dependencies. By projecting information directly to future time steps, it circumvents the gradient issues of standard RNNs. Future work will explore multiscale delay lines to better capture signals with varying temporal resolutions and hardware-specific optimizations for sliding window operations.