Bootstrapping a Music Voice Assistant with Weak Supervision

Sergio Oramas, Massimo Quadrana, Fabien GouyonNAACL 2021 · DOI

TL;DR

This paper introduces a novel weakly-supervised methodology for automatically labeling user queries for music voice assistants, significantly reducing the cost and time associated with traditional manual annotation. Their approach, combining heuristic labeling with a novel human filtering step, outperforms models trained on manually annotated or synthetic data, enabling the efficient bootstrapping of a large-scale music voice assistant.

What Problem Does This Paper Address?

Creating effective voice assistants, especially for domains with vast catalogs like music, hinges on accurately understanding user queries. A crucial step is "slot tagging," where entities (like artists, albums) and attributes (like genres, moods) are identified within a query. However, training robust slot tagging models requires massive amounts of accurately labeled user queries. Manually annotating these queries is prohibitively expensive and time-consuming, especially at the scale of millions of music entities. While synthetic data generation offers a way to create more labeled data, it often requires significant domain expertise and can still be costly. This paper addresses the challenge of efficiently generating high-quality training data for slot tagging in music voice assistants by leveraging abundant unlabeled query logs.

Methods

The core of the paper's contribution lies in its two-step methodology for weak supervision:

  1. Heuristic Labeling: This initial step automatically labels queries using a combination of corpus statistics and string matching rules.
  2. Entity Categorization: To handle the ambiguity of millions of music entities (e.g., a song title that's also a common phrase), entities are categorized into three sets: safe-set (high confidence the entity is intended), ignore-set (high confidence it's not the entity), and unsure-set (low confidence). This categorization uses a "frequency-popularity ratio" and "attribute overlap" with a predefined taxonomy of musical attributes.
  3. Query Labeling: After categorization, n-grams in queries are matched against entities in the safe-set and unsure-set. Queries with any unsure-set matches are discarded. Otherwise, matched n-grams are labeled with their entity type. Attributes are then labeled using simple string matching.
  4. Pattern Filtering via Human Curation: This step refines the heuristically labeled data.
  5. Pattern Extraction: Queries are converted into patterns by replacing entities and attributes with placeholders (e.g., "play the [theme] song [track]").
  6. Human Filtering: A human curator reviews the vocabulary of words appearing in these patterns. They identify and discard words that are likely noise (e.g., due to ASR errors, or words not truly part of the intended query pattern). Only patterns composed entirely of the curated vocabulary are kept, effectively filtering out noisy or incorrectly labeled queries.

Datasets

The paper uses real-world voice query logs from a major music streaming platform.

  • Training Data:
    • Manual Baseline (MAN): 5,000 manually annotated queries.
  • Synthetic Datasets (SYN): Four datasets (SYNS, SYNM, SYNL, SYNXL) generated by filling patterns extracted from the MAN dataset with catalog entities and attributes, varying in size.
  • Weakly-Supervised Datasets (WS): Four datasets (WSS, WSM, WSL, WSXL) generated by applying the heuristic labeling to progressively larger unlabeled query sets (100k, 1M, 10M, 100M).
  • Weakly-Supervised + Human Filtering (WS(F)): The WSXL dataset (100M unlabeled queries) further processed with the pattern filtering human curation step.
  • Evaluation: A held-out test set of 20% of the MAN dataset is used for evaluation.

Key Insights and Ideas

  • Leveraging Unlabeled Data is Key: The abundance of unlabeled voice queries is a valuable, often underutilized, resource for training robust NLP models.
  • Heuristics + Filtering = Powerful Weak Supervision: A simple heuristic labeling step, when combined with a targeted human filtering process (focused on pattern vocabulary rather than full queries), can yield surprisingly high-quality training data.
  • Cost-Benefit of Curation: The human filtering step, by focusing on a vocabulary of words within patterns, is significantly faster (hours vs. weeks) than full manual query annotation, yet achieves substantial improvements.
  • Scalability Matters: The performance of weakly-supervised models generally improves with the size of the training data, suggesting that the methodology scales well.
  • Noise Management is Crucial: The effectiveness of weak supervision is directly tied to how well noise (from ASR, ambiguous entities, etc.) is managed, which the pattern filtering step addresses.

Key Results

The paper demonstrates that models trained on weakly-supervised data significantly outperform the manually annotated baseline (MAN).

  • The WS(F) dataset, which includes human filtering, achieved the best test SER of 13.58, a substantial improvement over the MAN baseline (31.64).
  • Even without human filtering, the largest weakly-supervised dataset WSXL achieved an SER of 21.94, outperforming the MAN baseline and the largest synthetic dataset SYNXL (29.19).
  • The WS datasets generally showed incremental SER reduction with increasing dataset size, unlike the SYN datasets where SER sometimes increased with size.
  • The human filtering step for WS(F) required only about 4 hours of expert time to curate the pattern vocabulary, compared to the 140 hours it took for manual annotation of a similar amount of data.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the reduction in SER slows down as weakly-supervised dataset size increases, potentially due to accumulated noise.
  • The effectiveness of the SYN datasets was limited, with performance sometimes degrading as size increased, possibly due to bias amplification from a small set of initial patterns.
  • Future Work:
    • Investigate other manual curation tasks beyond vocabulary filtering, such as cleaning remaining patterns.
  • Explore how synthetic data generation can complement weakly-supervised datasets, especially for less frequent entities or attributes.
  • Experiment with different model architectures, including pre-trained language models.
  • Evaluate the methodology's applicability to other domains with large entity catalogs.

Summary updated

BibTeX
@inproceedings{oramas2021bootstrapping,
  author = {Sergio Oramas and Massimo Quadrana and Fabien Gouyon},
  title = {{Bootstrapping a Music Voice Assistant with Weak Supervision}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.7},
}
Figure