Abstract
Recurrent Neural Networks architectures excel at processing sequences by modelling dependencies over different timescales. The recently introduced Recurrent Weighted Average (RWA) unit captures long term dependencies far better than an LSTM on several challenging tasks. The RWA achieves this by applying attention to each input and computing a weighted average over the full history of its computations. Unfortunately, the RWA cannot change the attention it has assigned to previous timesteps, and so struggles with carrying out consecutive tasks or tasks with changing requirements. We present the Recurrent Discounted Attention (RDA) unit that builds on the RWA by additionally allowing the discounting of the past. We empirically compare our model to RWA, LSTM and GRU units on several challenging tasks. On tasks with a single output the RWA, RDA and GRU units learn much quicker than the LSTM and with better performance. On the multiple sequence copy task our RDA unit learns the task three times as quickly as the LSTM or GRU units while the RWA fails to learn at all. On the Wikipedia character prediction task the LSTM performs best but it followed closely by our RDA unit. Overall our RDA unit performs well and is sample efficient on a large variety of sequence tasks.
Introduction
Recurrent Neural Networks (RNNs), particularly LSTMs, are foundational for sequential data processing but face two significant challenges: recency bias and the quadratic computational cost ($O(N^2)$) of standard attention mechanisms. While the Recurrent Weighted Average (RWA) unit addressed efficiency by maintaining a running weighted average ($O(N)$), it suffers from a "stiffness" in its state. Because RWA accumulates information in a monotonically increasing denominator, it becomes increasingly difficult to update the model's state over time. This prevents the model from "forgetting" or clearing memory, causing it to fail on tasks with changing requirements or consecutive independent sub-tasks.
Methods
The Recurrent Discounted Attention (RDA) unit is proposed to resolve the rigidity of the RWA. Like its predecessor, the RDA computes a running numerator ($n_t$) and denominator ($d_t$) to create a weighted average of features. The key innovation is the introduction of a discount factor ($\\gamma_t$), a sigmoid-gated value that multiplies the previous state before incorporating new input.
This mechanism allows the model to dynamically down-weight the past. When $\\gamma_t$ is near 1, the unit maintains perfect memory (similar to RWA); when $\\gamma_t$ is low, it effectively resets its state to focus on new information. Crucially, RDA maintains linear $O(N)$ time complexity, as it only requires the summary from the previous timestep.
Results and Discussion
The authors evaluated RDA against RWA, LSTM, and GRU units across algorithmic, vision, and language tasks.
Algorithmic and Memory Tasks
A standout result occurred in the Multiple Sequence Copy task, where the model must output 50 consecutive sequences. The RDA learned this task three times faster than the LSTM or GRU. In contrast, the RWA failed to learn the task entirely, illustrating the necessity of the discounting mechanism for multi-stage memory tasks.
Language Modeling
On the Hutter Prize Wikipedia dataset (enwik8), the RDA significantly outperformed the RWA, which was unable to effectively model the language (5.067 BPC). While the LSTM remained the top performer at 1.492 BPC, the RDA followed closely at 1.529 BPC, demonstrating its versatility in dense prediction tasks.
Sample Efficiency and Vision
On sequential MNIST, RDA achieved 98.7% accuracy, outperforming RWA (97.9%) and an LSTM configuration that struggled significantly (11.4%). Across most benchmarks, RDA and GRU units demonstrated superior sample efficiency, reaching target performance levels in fewer training steps than LSTMs.
Conclusion
The Recurrent Discounted Attention (RDA) unit successfully bridges the gap between the global reach of attention and the flexible gating of LSTMs. By solving the "denominator problem" of the RWA through learnable discounting, RDA provides a sample-efficient, $O(N)$ architecture capable of handling complex, multi-stage sequential tasks where previous weighted-average models failed.
Summary updated