Automatic Grammar Augmentation for Robust Voice Command Recognition

Yang Yang, Anusha Lalitha, Jinwon Lee, Chris Lott

TL;DR

Researchers have developed a way to boost the accuracy of voice command systems on low-power embedded devices without making the underlying AI models larger. By creating a "statistical pronunciation dictionary" that maps a small model’s consistent mistakes (e.g., hearing "pla music" instead of "play music"), they can augment the system's grammar to recognize these errors as valid commands, significantly reducing misclassification.

What Problem Does This Paper Address?

Voice User Interfaces (UIs) are everywhere, but running them on "edge" devices like Bluetooth speakers or hearables is a massive challenge. These devices lack the memory and power to run the giant, state-of-the-art models used by cloud services.

When you shrink an Acoustic Model (AM) down to a "small footprint" (in this case, just 211,000 parameters compared to the typical 18 million+), its accuracy plummets. A tiny model might have a Word Error Rate (WER) as high as 48%. However, the authors noticed a pattern: these small models aren't making random noise; they make consistent, phonetically plausible errors based on their training and the speaker's accent or environment. The challenge is: how do we improve command recognition using these "broken" but predictable outputs without increasing the model size?

Methods

The authors propose a pipeline for Automatic Grammar Augmentation. Instead of trying to fix the model, they adapt the "grammar" (the list of words the system is looking for) to match the model's quirks.

  1. AM-Specific Statistical Pronunciation Dictionary: They run a large general speech dataset through their small model and compare the outputs to the ground truth using the Levenshtein (edit distance) algorithm. This creates a map of how the model "hears" specific words. For example, the model might hear "two" as "to" 53% of the time and "too" 0.3% of the time.
  2. Candidate Set Generation: For every command (like "play music"), they generate a list of potential variations based on the top-k most likely misinterpretations from their dictionary.
  3. Augmentation Search Algorithms: Adding every possible variation is a bad idea because it increases the "False Alarm Rate" (FAR) and makes the system more likely to miss a command entirely. They tested four algorithms to find the optimal subset of variations:
  4. Naive Greedy Search: Iteratively adding the best-performing variation.
  5. Greedy with Refinement: Adding variations while pruning redundant ones.
  6. Beam Search: Keeping a list of the top-l best grammar sets.
  7. Cross-Entropy Method (CEM): A sophisticated combinatorial optimization approach that treats the search as a machine learning problem.

Datasets

  • Acoustic Model Training: Trained on Libri-speech and other general datasets using a Connectionist Temporal Classification (CTC) loss.
  • Dictionary Building: Used a 2000-hour general speech dataset to map the model's error patterns.
  • Evaluation: A custom dataset containing five specific commands: play music, pause music, stop music, next song, and previous song. This included diverse speakers, pitches, and noise conditions.
  • Out-of-Domain (OOD): A separate dataset used to ensure the system doesn't trigger on random background speech (False Alarm Rate).

Key Insights and Ideas

  • Errors are Features, Not Bugs: The core "aha moment" is realizing that a small model's errors are consistent. If a model consistently hears "recter" for "rector," "recter" effectively becomes the model's internal representation for that word.
  • The Grammar Bloat Penalty: You can't just add infinite variations. As the grammar set grows, the "confidence threshold" required to trigger a command must increase to keep the False Alarm Rate stable. If the threshold gets too high, the system starts ignoring legitimate commands (increasing the Mis-Detection Rate).
  • CEM Wins: The Cross-Entropy Method (CEM) significantly outperformed greedy algorithms. It was better at navigating the complex inter-dependencies between commands (e.g., adding a variation for "play" might accidentally make it look too much like "pause").

Key Results

  • Accuracy Boost: Using the CEM algorithm, the system improved the command success rate from 90.34% (baseline) to 94.44%.
  • Error Reduction: The principled augmentation significantly reduced mis-classification (hearing the wrong command) without a major spike in mis-detection (hearing nothing).
  • Efficiency: While adding all 150 candidate variations actually tanked performance to 80% accuracy, the optimized search found a "sweet spot" that maximized performance with minimal computational overhead during the search phase.

Limitations and Future Work

  • Fixed Front-End: The study assumes a fixed audio front-end and doesn't account for how different noise-suppression hardwares might change the error patterns.
  • Command Scale: The experiments were limited to a small set of five commands. It remains to be seen how well this scales to dozens or hundreds of commands where phonetic overlap becomes more likely.
  • Personalization: The authors suggest that the next step is personalization. By adapting the grammar to a specific user’s pronunciation patterns and the specific model's errors for that user, they could potentially push accuracy even higher.

Summary updated

Abstract

This paper proposes a novel pipeline for automatic grammar augmentation that provides a significant improvement in the voice command recognition accuracy for systems with small footprint acoustic model (AM). The improvement is achieved by augmenting the user-defined voice command set, also called grammar set, with alternate grammar expressions. For a given grammar set, a set of potential grammar expressions (candidate set) for augmentation is constructed from an AM-specific statistical pronunciation dictionary that captures the consistent patterns and errors in the decoding of AM induced by variations in pronunciation, pitch, tempo, accent, ambiguous spellings, and noise conditions. Using this candidate set, greedy optimization based and cross-entropy-method (CEM) based algorithms are considered to search for an augmented grammar set with improved recognition accuracy utilizing a command-specific dataset. Our experiments show that the proposed pipeline along with algorithms considered in this paper significantly reduce the mis-detection and mis-classification rate without increasing the false-alarm rate. Experiments also demonstrate the consistent superior performance of CEM method over greedy-based algorithms.
BibTeX
@inproceedings{yang2019automatic,
  author = {Yang Yang and Anusha Lalitha and Jinwon Lee and Chris Lott},
  title = {{Automatic Grammar Augmentation for Robust Voice Command Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=t3qNx0YgkR},
}
Figure