TopGuNN: Fast NLP Training Data Augmentation using Large Corpora

Rebecca Iglesias-Flores, Megha Mishra, Ajay Patel, Akanksha Malhotra, Reno Kriz, Martha Palmer, Chris Callison-BurchNAACL 2021

TL;DR

This paper introduces TopGuNN, a novel system for rapidly augmenting NLP training data by efficiently mining large text corpora. It leverages contextualized word embeddings and approximate k-NN search to find diverse, semantically similar examples, significantly reducing the manual effort and cost associated with data collection.

What Problem Does This Paper Address?

Acquiring high-quality training data for Natural Language Processing (NLP) models is a major bottleneck. Traditional methods rely on expensive and time-consuming manual annotation by domain experts or crowdsourcing. While large language models can help find similar text, exhaustively searching massive corpora for semantically relevant examples is computationally intractable. This paper addresses the challenge of efficiently and effectively scaling up training data generation from a few expert-crafted examples to thousands of diverse instances, crucial for improving model generalization.

Methods

TopGuNN employs a four-stage pipeline:

  1. Pre-processing: The system first processes a large corpus (Gigaword) by performing NLP analysis (e.g., POS tagging, dependency parsing) using spaCy. This step also filters out non-content words to reduce the size of the data to be indexed.
  2. Generating Embeddings: It uses BERT to generate contextualized word embeddings for each content word in the corpus. This is a key differentiator from older methods like word2vec, as BERT embeddings capture word meaning based on context.
  3. Indexing: The generated embeddings are then indexed using Annoy, an approximate k-NN (k-Nearest Neighbors) search library. Annoy is chosen for its ability to build on-disk indexes, making it memory-efficient for massive datasets. The corpus is partitioned into 960 Annoy indexes for parallel processing.
  4. Running Queries: When a query word or sentence is provided, TopGuNN searches across all Annoy indexes to retrieve the top-N most similar embeddings. It uses an iterative requery method to balance performance and the number of unique results.

Datasets

The primary dataset used is the English Gigaword Fifth Edition Corpus. This corpus contains approximately 183 million sentences and 4.3 billion tokens, spanning 16 years from 7 international news agencies. The paper does not introduce new benchmarks but demonstrates TopGuNN's utility on a well-established large-scale corpus.

Key Insights and Ideas

  • Contextual Embeddings are Key: Using BERT's contextualized word embeddings, rather than static ones, is crucial for capturing nuanced semantic similarity and retrieving diverse examples, especially for polysemous words.
  • Approximate k-NN for Scale: Employing approximate k-NN (Annoy) is essential for making similarity search over billions of embeddings computationally feasible and memory-efficient, enabling the use of large corpora.
  • Filtering Non-Content Words: Strategically filtering out non-content words (e.g., articles, prepositions) significantly reduces the index size (by a factor of 2.8x) without substantial loss in search speed or quality.
  • Human-in-the-Loop Efficiency: TopGuNN is designed to work effectively within a human-in-the-loop annotation process, where a few seed examples are used to generate thousands of candidates, which are then efficiently filtered by non-expert annotators.

Key Results

  • Scalability: TopGuNN successfully indexed and queried a corpus with approximately 1.5 billion embeddings (4.63TB) in less than a day.
  • Efficiency: The system can perform queries over the entire Gigaword corpus, with the first query in a batch taking around 21.4 hours and subsequent queries taking significantly less time due to caching.
  • Quality: The paper demonstrates with examples (e.g., "diagnosis," "disable," "destroy") that TopGuNN retrieves diverse and contextually relevant sentences, even for words with multiple senses.
  • Corpus Size Impact: Running TopGuNN on a smaller subset of Gigaword (400,000 sentences) still yields high-quality and diverse results, suggesting that for some applications, a smaller corpus might suffice for faster retrieval without a major sacrifice in quality.

Limitations and Future Work

  • BERT Biases: The paper acknowledges that TopGuNN inherits biases present in the BERT model, which can manifest in search results (e.g., associating "bombing" with "Muslim"). It suggests manual review and future work on debiasing techniques.
  • Querying Complexity: Future work could extend TopGuNN to query for multi-word expressions, hyphenated words, or natural language questions.
  • Antonymy: Identifying antonyms is suggested as a valuable extension for more fine-grained search results, as synonyms and antonyms can occupy similar embedding spaces.
  • Hardware Requirements: While Annoy allows for on-disk indexes, the initial embedding generation and indexing stages still require significant computational resources (GPUs, large RAM).

Summary updated

BibTeX
@inproceedings{iglesiasflores2021topgunn,
  author = {Rebecca Iglesias-Flores and Megha Mishra and Ajay Patel and Akanksha Malhotra and Reno Kriz and Martha Palmer and Chris Callison-Burch},
  title = {{TopGuNN: Fast NLP Training Data Augmentation using Large Corpora}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure