Iterative Data Programming for Expanding Text Classification Corpora

Neil Mallinar, Abhishek Shah, Tin Kam Ho, Rajendra Ugrani, Ayush Gupta

Building high-performing text classifiers requires large amounts of labeled data, which is often expensive and time-consuming to produce. This challenge is particularly acute in "high-skew" scenarios, where specific categories of interest (e.g., a niche customer intent) are buried like needles in a haystack of millions of unrelated documents.

In this paper, the authors propose Iterative Data Programming (I-DP), a framework that combines search-based retrieval with weak supervision to efficiently expand small, labeled text datasets. By automatically generating "labeling functions" from search results and denoising them through a generative model, I-DP identifies rare examples in massive unlabeled corpora more effectively than standard active learning or search-based methods.

The I-DP Framework

The core of the proposed approach is an iterative loop that alternates between a selection model and a human labeler. While a standard baseline might use simple search-based expansion (like Elasticsearch’s "More Like This" or I-MLT), I-DP introduces a more sophisticated mechanism:

  1. Automated Weak Models: For every batch of data a human labels, the system creates weak models based on lexical "neighborhoods" of positive and negative examples.
  2. Generative Denoising: A generative model (similar to the Snorkel framework) processes these overlapping or conflicting weak labels to produce a single probabilistic label for every unlabeled document.
  3. Ranking for Verification: The system uses these probabilities to rank the next batch of documents for human verification, ensuring the labeler focuses on the most promising candidates.
Comparison of relevant examples found per iteration for the 'graphics_tablet' subreddit.

A key advantage of this method is that it decouples the expansion process from the final "downstream" classifier. This avoids the massive overhead of retraining complex models (like BERT) during the data collection phase, making the process highly efficient.

Key Results and Insights

The authors evaluated I-DP on three distinct corpora: Reddit technology posts, and both English and Japanese customer service inquiries. The results consistently showed that I-DP outperforms both random sampling and simple search-based expansion (I-MLT).

  • Yield and Accuracy: On the Reddit dataset, a classifier trained on data from I-DP achieved 46.6% accuracy, compared to 40.8% for I-MLT and 30.7% for a random baseline. Furthermore, I-DP identified significantly more relevant examples (16,006 vs. 6,934 for I-MLT) before hitting termination criteria.
  • Sustaining the Expansion Path: As illustrated in Figure 1, simple search (I-MLT) tends to plateau quickly as it finds "low-hanging fruit." In contrast, I-DP uses its generative model to understand the relationships between weak models, allowing it to "push" into new areas of the data space and sustain expansion for many more iterations.
  • Convergence: Using KL-Divergence to measure how closely the expanded training set's vocabulary matches the optimal distribution, I-DP showed a much steeper and more consistent descent toward zero than the search-only method.
Performance metrics comparing I-MLT and I-DP across various Reddit intents, showing recall vs. examples and KL-divergence vs. iterations.

Limitations and Future Work

While effective, the method has limitations. If a class is highly fragmented—meaning users discuss the same topic using completely different, non-overlapping vocabularies—the algorithm might get stuck in one "island" of data. Future work will explore better ways to "scatter seeds" across the data space to ensure all fragments of a class are discovered, as well as further optimizing the computational cost of the generative model.

Summary updated

BibTeX
@inproceedings{mallinar2020iterative,
  author = {Neil Mallinar and Abhishek Shah and Tin Kam Ho and Rajendra Ugrani and Ayush Gupta},
  title = {{Iterative Data Programming for Expanding Text Classification Corpora}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=3LpS7JBwTs},
}
Figure