SkipW: Resource Adaptable RNN with Strict Upper Computational Limit

Tsiry Mayet, Anne Lambert, Pascal Leguyadec, Françoise Le Bolzer, François SchnitzlerICLR 2021 · OpenReview

Introduction

SkipW (Skip-Window) is a novel recurrent neural network (RNN) architecture designed specifically for edge devices that must balance predictive accuracy with strict, real-time power and compute constraints. By grouping inputs into discrete windows and employing a "Top-K" selection mechanism, SkipW guarantees a hard upper limit on computation while maintaining—and often exceeding—the accuracy of existing flexible models.

The Challenge of Edge Computing

As deep learning increasingly moves from the cloud to "edge" devices like smartphones, wearables, and IoT sensors, researchers face a major hurdle: resource volatility. An edge device might suddenly need to throttle its CPU to save battery or reallocate memory to a new background process.

While existing "flexible" RNNs (such as SkipRNN) can skip unimportant data points to save time, they generally optimize for average computational cost. They lack a mechanism to enforce a strict upper limit on computation for a specific window of time. In a real-time system, an unexpected "burst" of computation in the middle of a sequence can lead to latency spikes or system failure. SkipW addresses this by allowing developers to set a hard "budget" that the model cannot exceed, even mid-sequence.

Methods: The SkipW Architecture

SkipW acts as a "wrapper" that can be placed around any standard RNN cell, such as an LSTM or GRU. Its core innovation lies in how it decides which inputs to process within a sequence.

  1. Windowing: Instead of making a skip/process decision at every single time step, SkipW evaluates a window of $L$ inputs at a time.
  2. Importance Scoring: At the start of each window, the model computes an "importance" vector for all $L$ inputs based on the current hidden state.
  3. The SelectK Mechanism: This provides the "hard limit" feature. The model uses a TopK function to identify the $K$ most important inputs in the window. Any input not in the top $K$ is automatically skipped.
  4. Thresholding: A secondary parameter, $thr$, allows the model to skip even more inputs if their importance scores fall below a certain level, providing a "soft" way to save additional energy.
  5. Dynamic Adaptation: Crucially, $K$ and $thr$ can be changed dynamically during inference. If a device's battery drops below 10%, the system can instantly lower $K$ to reduce the workload without needing to reload or retrain the model.
SkipW Cell Architecture

Key Insights and Performance

The authors evaluated SkipW across several diverse benchmarks, including Human Activity Recognition (HAR), Sequential MNIST, IMDB sentiment analysis, and a synthetic Adding Task.

Hard vs. Soft Constraints

The primary breakthrough is the realization that for real-world hardware, a guaranteed limit is better than an average target. By enforcing a $K/L$ ratio, SkipW ensures predictable performance. Furthermore, skipping an RNN update doesn't just save a few matrix multiplications; it allows the system to skip the entire input pipeline. For example, skipping a video frame means avoiding an expensive CNN-based feature extraction step entirely.

Efficiency and Baseline Comparisons

On the HAR dataset, SkipW demonstrated remarkable efficiency, processing as few as 15% of the inputs without any significant drop in accuracy.

Accuracy vs. Inputs Processed

SkipW consistently matched or outperformed baselines like SkipRNN and ThrRNN. In many cases, it achieved higher accuracy for a lower computational cost because its windowed approach allowed for more strategic skipping. It also significantly outperformed "naive" strategies like random subsampling or periodic skipping, proving that its learned importance mechanism is vital.

Hardware Validation

The authors validated the model on an Nvidia Jetson Nano. They found that reducing $K$ led to dramatic, non-linear savings in latency and energy—achieving up to a 50% energy gain in some configurations.

Limitations and Future Work

Despite its advantages, SkipW has a few limitations:

  • Non-Differentiability: The TopK and binarization functions are not naturally differentiable. The authors used a "straight-through estimator" to train the model, a common workaround that can sometimes introduce training instability.
  • Sensitivity on Dense Sequences: On the Sequential MNIST task, SkipW's accuracy dropped faster than SkipRNN's when the computational budget was extremely tight. This suggests that for very long, dense sequences, the windowed approach might be slightly less granular than step-by-step methods.
  • Hyperparameter Tuning: The window size $L$ is a critical hyperparameter. Choosing an $L$ that is too large might make the model slow to react to changes, while an $L$ that is too small limits the flexibility of the TopK selection. Future work could involve making $L$ adaptive during inference.

Summary updated

Abstract

We introduce Skip-Window, a method to allow recurrent neural networks (RNNs) to trade off accuracy for computational cost during the analysis of a sequence. Similarly to existing approaches, Skip-Window extends existing RNN cells by adding a mechanism to encourage the model to process fewer inputs. Unlike existing approaches, Skip-Window is able to respect a strict computational budget, making this model more suitable for limited hardware. We evaluate this approach on two datasets: a human activity recognition task and adding task. Our results show that Skip-Window is able to exceed the accuracy of existing approaches for a lower computational cost while strictly limiting said cost.
BibTeX
@inproceedings{mayet2021skipw,
  author = {Tsiry Mayet and Anne Lambert and Pascal Leguyadec and Françoise Le Bolzer and François Schnitzler},
  title = {{SkipW: Resource Adaptable RNN with Strict Upper Computational Limit}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=pAbnKurosPk},
}
Figure