A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

Hengshun Zhou, Jun Du, Chao-Han Huck Yang, Shifu Xiong, Chin-Hui Lee

This paper, titled "A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning," presents a robust solution for "always-on" voice assistants that need to work in noisy living rooms without draining the battery or memory of smart devices.

TL;DR

Researchers from USTC and Georgia Tech have developed a compact audio-visual wake word spotting (WWS) system that uses lip-reading to maintain accuracy in noisy environments. By applying a specialized pruning technique called LTH-IF (Lottery Ticket Hypothesis with Iterative Fine-tuning), they successfully slashed the model's size while actually improving its performance compared to the original, unpruned version.

What Problem Does This Paper Address?

Wake word spotting (WWS)—the technology behind "Hey Siri" or "Alexa"—faces two major hurdles:

  1. The "Cocktail Party" Problem: Audio-only systems struggle when there is background noise (like a TV or loud conversation). Adding visual data (lip movements) helps, but it usually makes the model much larger.
  2. The Resource Constraint: WWS must run continuously on "edge" devices (smart TVs, phones) with very limited memory and processing power.

The authors aim to bridge this gap by creating an audio-visual system that is both noise-robust and small enough for commercial product deployment.

Methods

The authors propose a two-pronged approach involving a multi-modal architecture and a sophisticated pruning strategy.

1. The Architecture

  • Audio Stream: Uses standard 40-dimensional Filter Bank (FBank) features.
  • Video Stream: Uses a MobileNetV2 backbone to encode detected lip movements into fixed-size vectors. To keep it lightweight, they replaced heavy bidirectional LSTMs with standard LSTMs.
  • Fusion: The audio and visual features are concatenated and fed into a "fusion stream" consisting of convolutional layers, an LSTM for temporal modeling, and fully connected (FC) layers for the final classification.

2. The Pruning Strategy: LTH-IF

Standard pruning often hurts accuracy. The authors instead use the Lottery Ticket Hypothesis with Iterative Fine-tuning (LTH-IF).

  • The Concept: Within a large neural network, there exists a smaller "winning ticket" subnetwork that, if trained correctly, can match or beat the performance of the full network.
  • The Process: They train the model, mask out the least important weights, reset the remaining weights to their original values, and re-train. They repeat this iteratively rather than all at once.
  • AV-Specific Tweak: They discovered that pruning the lip encoder separately before pruning the entire fusion network yields better results than pruning the whole system globally.

Datasets

The study uses an in-house audio-visual dataset recorded in a "smart home TV" scenario:

  • Participants: 250 speakers (1:1 male-to-female ratio).
  • Wake Word: "Xiao T Xiao T."
  • Training Set: 100 hours (50h positive, 50h negative) with various background noises added for augmentation.
  • Test Set: Includes three signal-to-noise ratios (-5dB, 0dB, and 5dB) to simulate different levels of environmental noise.

Key Insights and Ideas

  • Visuals are the Anchor: In high-noise environments (-5dB), the audio-only system's False Alarm Rate (FAR) jumps to over 8%. Adding visual data drops that rate to 3.25%, proving that lip-reading is a vital "fail-safe" for audio.
  • Pruning as a Regularizer: Surprisingly, the pruned models often performed better than the dense ones. For example, the pruned audio-visual model achieved a 2.29% FAR at -5dB, compared to 3.25% for the unpruned version. This suggests that pruning helps the model focus on the most relevant features and ignore noise.
  • The Lip Encoder Bottleneck: The researchers found that the lip encoder (MobileNetV2) accounts for the vast majority of the system's parameters and computations (FLOPs). This makes it the primary target for optimization.

Key Results

  • Noise Robustness: The audio-visual system significantly outperformed single-modal systems across all noise levels.
  • Extreme Compression: For the audio-only model, they were able to prune 80.21% of parameters while maintaining better performance than the baseline.
  • AV Efficiency: For the combined audio-visual model, they pruned 42.52% of the total parameters. This "winning ticket" subnetwork was more robust to noise and more computationally efficient than the original model.
  • Layer Consistency: Interestingly, the optimal pruning rate was remarkably consistent across different types of layers (Conv, LSTM, and FC), hovering around 65% for a specific iteration.

Limitations and Future Work

  • Dataset Access: The results are based on an in-house dataset, making it difficult for outside researchers to replicate the exact benchmarks on public datasets like Google Speech Commands or Lip Reading in the Wild (LRW).
  • Fixed Wake Word: The system is trained for a specific phrase. Future work could explore "user-defined" wake words or more generalized keyword spotting.
  • Architecture Search: The authors suggest that combining LTH-IF with Neural Architecture Search (NAS) could lead to even more efficient, hardware-specific designs.

Summary updated

Abstract

Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing a compact audio-visual WWS system by utilizing visual information to alleviate the degradation. Specifically, in order to use visual information, we first encode the detected lips to fixed-size vectors with MobileNet and concatenate them with acoustic features followed by the fusion network for WWS. However, the audio-visual model based on neural networks requires a large footprint and a high computational complexity. To meet the application requirements, we introduce a neural network pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning manner (LTH-IF), to the single-modal and multi-modal models, respectively. Tested on our in-house corpus for audio-visual WWS in a home TV scene, the proposed audio-visual system achieves significant performance improvements over the single-modality (audio-only or video-only) system under different noisy conditions. Moreover, LTH-IF pruning can largely reduce the network parameters and computations with no degradation of WWS performance, leading to a potential product solution for the TV wake-up scenario.
BibTeX
@inproceedings{zhou2022study,
  author = {Hengshun Zhou and Jun Du and Chao-Han Huck Yang and Shifu Xiong and Chin-Hui Lee},
  title = {{A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746360},
}

Loading related content...

Figure