Abstract
Recurrent neural networks (RNNs) have shown state of the art results for speech recognition, natural language processing, image captioning and video summarizing applications. Many of these applications run on low-power platforms, so their energy efficiency is extremely important. We observed that cache-oblivious RNN scheduling during inference typically results in 30-50x more data transferred on and off the CPU than the application's working set size. This can potentially impact its energy efficiency. This paper presents a new metric called Data Reuse Efficiency to gauge the RNN scheduling efficiency of a platform and shows the factors that influence the DRE value. Additionally, this paper discusses an optimization to improve reuse in RNNs and highlights the positive impact of this optimization on the total amount of memory read from or written to the memory controller (and, hence, the DRE value) during the execution of an RNN application for a mobile SoC.
Introduction
In the realm of mobile and edge computing, energy efficiency is the primary constraint. While Recurrent Neural Networks (RNNs), such as LSTMs and GRUs, are workhorses for natural language processing (NLP), their execution on modern System-on-Chips (SoCs) is often surprisingly inefficient. Standard deep learning frameworks frequently execute RNNs in a "cache-oblivious" manner, moving 30-50x more data than necessary between main memory (DRAM) and the CPU.
The bottleneck is primarily memory bandwidth. Current scheduling algorithms treat RNN time steps sequentially without considering cache limits. If a model's weight matrices exceed the processor's Last-Level Cache (LLC), the system is forced to reload the same weights from DRAM for every single word in a sequence. For a 100-word input, this leads to massive energy waste as the system fetches the same weights 100 times.
Quantifying Inefficiency: Data Reuse Efficiency (DRE)
To address this, we introduce Data Reuse Efficiency (DRE), defined as the ratio of total data transferred to the memory controller versus the actual "working set" size (weights, embeddings, and intermediate values). An ideal system achieves a DRE of 1. However, our analysis of 320 synthetic benchmarks reveals that as input length increases, DRE values often skyrocket.
As shown in Figure 3, while 70.8% of benchmarks with an input length of 10 had a DRE > 1, for an input length of 100, 22% of benchmarks reached a staggering DRE > 50. This "cache cliff" occurs when the working set exceeds cache size even slightly, causing the system to begin "thrashing"—constantly swapping data in and out of memory.
Proposed Optimization: Schedule A+
We propose a "Schedule A+" optimization to mitigate this waste. Standard LSTMs typically concatenate input weights ($W$) and recurrent weights ($U$) into a single large matrix ($G$). We propose splitting $G$ back into $G1$ (input) and $G2$ (recurrent).
Because the entire input sequence is often available at once in NLP tasks, the system can multiply the full input matrix by $G1$ in a single pass. This ensures $G1$ is read from DRAM only once per layer, rather than once per time step. While $G2$ must still be read at every step due to sequential dependencies, the total bandwidth is roughly halved.
Evaluation and Results
We evaluated this approach using an analytical model of a 12MB LRU cache (typical for mobile SoCs) across four representative NLP applications: GNMT, DeepSpeech1, Language Modeling (LM), and byteNER.
The results in Figure 5 demonstrate that "Schedule A+" significantly reduces DRAM traffic for larger models. For the GNMT encoder, DRAM traffic decreased by 1.45x to 1.65x. In the Language Modeling benchmark, the first layer achieved a 2x improvement. Conversely, for tiny models like byteNER (9.4MB) that fit entirely within the 12MB cache, the optimization showed no benefit, confirming that the bottleneck is strictly related to cache-to-model size ratios.
Limitations and Future Work
The optimization is less effective for the "decoder" portion of translation models, where the input at time $t$ is the output of time $t-1$, preventing the pre-calculation of $G1$ multiplications. It also does not apply to attention layers with different data access patterns. Future work should focus on making deep learning compilers "cache-aware" to automatically perform these types of scheduling optimizations based on the target hardware's cache size.
Summary updated