Abstract
The increasing computational and memory complexities of deep neural networks have made it difficult to deploy them on low-resource electronic devices (e.g., mobile phones, tablets, wearables). Practitioners have developed numerous model compression methods to address these concerns, but few have condensed input representations themselves. In this work, we propose a fast, accurate, and lightweight convolutional representation that can be swapped into any neural model and compressed significantly (up to 32x) with a negligible reduction in performance. In addition, we show gains over recurrent representations when considering resource-centric metrics (e.g., model file size, latency, memory usage) on a Samsung Galaxy S9.
Introduction
Deploying state-of-the-art Natural Language Processing (NLP) models on "edge" devices like smartphones and wearables presents significant challenges. These devices operate under strict hardware constraints, including limited memory, tight power budgets, and the need for real-time responsiveness. While Recurrent Neural Networks (RNNs) have traditionally been the standard for text processing, their sequential nature makes them inherently slow. Convolutional Neural Networks (CNNs) offer better parallelism but are often too parameter-heavy for mobile deployment. This research introduces a fundamentally more efficient input representation designed to be lightweight from the outset, rather than relying solely on post-training compression.
Methodology
The proposed architecture builds upon the convolutional encoder but introduces three critical optimizations to reduce computational overhead:
- Efficient Non-Linearity: Gaussian Error Linear Units (GELU) replace Gated Linear Units (GLUs). Unlike GLUs, which double the number of output channels, GELUs maintain performance while significantly reducing internal capacity and parameter counts.
- Depthwise Separable Convolutions: The model decouples spatial feature learning from channel transitions. It utilizes Depthwise Convolutions (one filter per input channel) followed by Pointwise Convolutions ($1 \times 1$ kernels) to transition between channel spaces.
- Pointwise Bottlenecks: To optimize the $1 \times 1$ convolutions, which typically scale quadratically with channel count, a bottleneck dimension $b$ is introduced. This decomposes the operation into downsampling and upsampling steps, drastically lowering the total operation count.
Evaluation and Datasets
The approach was evaluated across three distinct NLP tasks:
- Next Word Prediction (NWP): 5.8M training sentences from social media, mimicking informal keyboard input.
- Joint Intent-Slot Modeling: 36.2K device control utterances (e.g., "turn up the volume") with 36 intent classes and 14 slot tags.
- Document Classification: A subset of the Amazon Product Reviews dataset (100K samples) for binary sentiment analysis, processed at the byte-level.
Key Results
Benchmarking on a Samsung Galaxy S9 demonstrated that the optimized convolutional block provides a superior balance of accuracy and efficiency compared to recurrent baselines.
Task-Specific Performance
- Intent-Slot Modeling: The optimized model achieved an 80% reduction in file size and an 86% reduction in latency compared to the recurrent baseline, with less than a 1% drop in F1 score. As shown in the resource metrics table below, the final model with bottlenecks reached a latency of just 3.7 ms.
- Next Word Prediction: Latency was reduced by 32% compared to the recurrent baseline while maintaining comparable keystroke savings.
- Document Classification: The model utilized 27x fewer parameters than the baseline convolutional model and 5x fewer than the recurrent model, while slightly improving accuracy over the standard CNN.
Discussion and Insights
This work highlights that rethinking input representations can yield massive efficiency gains before traditional pruning or quantization are even applied. While CNNs generally utilize slightly more peak memory than RNNs—a necessary engineering trade-off for speed—their ability to leverage multi-core mobile architectures makes them highly suitable for on-device deployment.
Future research should explore the viability of these lightweight representations in morphologically rich or low-resource languages, as well as their performance across diverse mobile System on a Chip (SoC) architectures beyond the Galaxy S9.
Summary updated