Abstract
Deploying sophisticated Natural Language Understanding (NLU) on power-constrained edge devices like smartwatches requires extreme model compression. This paper presents a novel "binarization as regularization" technique that constrains neural network weights to binary values (-1 or 1) during training. By employing a double-Gaussian regularization term, we achieve a 16x reduction in model size for intent classification with only a 1% decrease in accuracy. This approach enables high-performance NLU on hardware with minimal memory and computational overhead.
Introduction
The proliferation of voice-controlled edge devices—ranging from smartwatches to headphones—has increased the demand for local NLU processing. Local execution addresses critical concerns regarding latency, user privacy, and connectivity. However, standard deep learning models typically utilize 32-bit floating-point weights, resulting in memory footprints that exceed the capacity of many embedded systems. While 8-bit quantization offers some relief, this work explores the theoretical limit: weight binarization.
Methodology: Binarization as Regularization
The core innovation of this research is treating binarization not as a post-processing step, but as a probabilistic prior during training.
Double-Gaussian Regularization
Traditional $L_2$ regularization pulls weights toward zero to prevent overfitting. In contrast, we introduce a double-Gaussian loss function centered at -1 and 1. This creates two "gravity wells" that incentivize weights to converge toward binary states rather than zero.
The training process optimizes a hybrid loss function: $$L_{total} = L_{task} + \lambda L_{bin, \sigma}$$ where $L_{task}$ is the standard cross-entropy loss for the NLU task, and $L_{bin, \sigma}$ is the binarization term. This allows the network to find high-performance solutions within the binary constraint. As training progresses, the standard deviation $\sigma$ can be decreased to sharpen the peaks at -1 and 1.
Architectural Efficiency
In a fully binarized network, standard multiply-accumulate (MAC) operations are replaced by sign-toggles and additions. This shift significantly reduces power consumption and increases throughput on Digital Signal Processors (DSPs) and FPGAs, as the hardware no longer needs to perform complex floating-point multiplications.
Experimental Results
The proposed method was evaluated on an in-house corpus of 50,000 voice queries focused on home automation (e.g., "turn on the kitchen lights").
Intent Classification
For intent classification, the binarized model achieved a 16x compression ratio compared to the floating-point baseline while maintaining near-parity in accuracy.
As shown in the results, the model can be compressed to as little as 0.3kB while still providing 72% accuracy, which may be sufficient for simple "always-on" trigger applications or low-power wake-word detection.
Slot Tagging
In Recurrent Neural Network (RNN) architectures used for slot tagging, the method reduced model size from 1113kB to 180kB. The remaining memory footprint is largely attributed to floating-point word embeddings, which were not binarized in this study, highlighting a key area for future optimization.
Conclusion and Future Work
Binarization via double-Gaussian regularization provides a robust pathway for deploying NLU on ultra-low-power hardware. Future research will focus on binarizing word embeddings to further reduce memory usage and investigating the scalability of this approach to more complex, open-domain NLU tasks where the weight distribution may require more nuanced priors.
Summary updated