Ultra-Compact NLU: Neuronal Network Binarization as Regularization

Munir Georges, Krzysztof Czarnowski, Tobias BockletInterspeech 2019 · DOI

Abstract

Deploying sophisticated Natural Language Understanding (NLU) on power-constrained edge devices like smartwatches requires extreme model compression. This paper presents a novel "binarization as regularization" technique that constrains neural network weights to binary values (-1 or 1) during training. By employing a double-Gaussian regularization term, we achieve a 16x reduction in model size for intent classification with only a 1% decrease in accuracy. This approach enables high-performance NLU on hardware with minimal memory and computational overhead.

Introduction

The proliferation of voice-controlled edge devices—ranging from smartwatches to headphones—has increased the demand for local NLU processing. Local execution addresses critical concerns regarding latency, user privacy, and connectivity. However, standard deep learning models typically utilize 32-bit floating-point weights, resulting in memory footprints that exceed the capacity of many embedded systems. While 8-bit quantization offers some relief, this work explores the theoretical limit: weight binarization.

Methodology: Binarization as Regularization

The core innovation of this research is treating binarization not as a post-processing step, but as a probabilistic prior during training.

Double-Gaussian Regularization

Traditional $L_2$ regularization pulls weights toward zero to prevent overfitting. In contrast, we introduce a double-Gaussian loss function centered at -1 and 1. This creates two "gravity wells" that incentivize weights to converge toward binary states rather than zero.

The double-Gaussian loss function $L_{bin, \sigma}(w_i)$ for various values of $\sigma$, illustrating the potential wells at -1 and 1.

The training process optimizes a hybrid loss function: $$L_{total} = L_{task} + \lambda L_{bin, \sigma}$$ where $L_{task}$ is the standard cross-entropy loss for the NLU task, and $L_{bin, \sigma}$ is the binarization term. This allows the network to find high-performance solutions within the binary constraint. As training progresses, the standard deviation $\sigma$ can be decreased to sharpen the peaks at -1 and 1.

Architectural Efficiency

In a fully binarized network, standard multiply-accumulate (MAC) operations are replaced by sign-toggles and additions. This shift significantly reduces power consumption and increases throughput on Digital Signal Processors (DSPs) and FPGAs, as the hardware no longer needs to perform complex floating-point multiplications.

Experimental Results

The proposed method was evaluated on an in-house corpus of 50,000 voice queries focused on home automation (e.g., "turn on the kitchen lights").

Intent Classification

For intent classification, the binarized model achieved a 16x compression ratio compared to the floating-point baseline while maintaining near-parity in accuracy.

Comparison of memory size and accuracy for intent classification across different weight types and compression levels.

As shown in the results, the model can be compressed to as little as 0.3kB while still providing 72% accuracy, which may be sufficient for simple "always-on" trigger applications or low-power wake-word detection.

Slot Tagging

In Recurrent Neural Network (RNN) architectures used for slot tagging, the method reduced model size from 1113kB to 180kB. The remaining memory footprint is largely attributed to floating-point word embeddings, which were not binarized in this study, highlighting a key area for future optimization.

Conclusion and Future Work

Binarization via double-Gaussian regularization provides a robust pathway for deploying NLU on ultra-low-power hardware. Future research will focus on binarizing word embeddings to further reduce memory usage and investigating the scalability of this approach to more complex, open-domain NLU tasks where the weight distribution may require more nuanced priors.

Summary updated

BibTeX
@inproceedings{georges2019ultra,
  author = {Munir Georges and Krzysztof Czarnowski and Tobias Bocklet},
  title = {{Ultra-Compact NLU: Neuronal Network Binarization as Regularization}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2591},
}
Figure