Iterative Data Programming for Expanding Text Classification Corpora

Neil Mallinar, Abhishek Shah, Tin Kam Ho, Rajendra Ugrani, Ayush Gupta

Abstract

Real-world text classification tasks often require many labeled training examples that are expensive to obtain. Recent advancements in machine teaching, specifically the data programming paradigm, facilitate the creation of training data sets quickly via a general framework for building weak models, also known as labeling functions, and denoising them through ensemble learning techniques. We present a fast, simple data programming method for augmenting text data sets by generating neighborhood-based weak models with minimal supervision. Furthermore, our method employs an iterative procedure to identify sparsely distributed examples from large volumes of unlabeled data. The iterative data programming techniques improve newer weak models as more labeled data is confirmed with human-in-loop. We show empirical results on sentence classification tasks, including those from a task of improving intent recognition in conversational agents.

Introduction

The "data bottleneck" remains the primary obstacle in developing effective text classifiers, such as those used in conversational agents. While machine learning algorithms have become increasingly sophisticated, they remain dependent on large volumes of labeled data, which is costly and slow to produce. This problem is particularly acute when dealing with highly skewed classes—rare intents buried within millions of unlabeled logs. Traditional active learning methods often struggle in these scenarios because they require a reasonably accurate starting model to be effective, creating a "chicken and egg" problem for new projects.

Methodology

The authors propose Iterative Data Programming (I-DP), a human-in-the-loop framework designed to rapidly expand small, labeled text datasets. I-DP bridges the gap between simple search-based retrieval and complex generative modeling.

The system operates through an iterative expansion procedure:

  1. Selection: A selection model ranks the unlabeled corpus and presents a batch to a human annotator.
  2. Labeling: The human labels the batch, and the system updates its model.
  3. I-MLT (Baseline): This uses "More Like This" search (via Elasticsearch), treating positive examples as queries. It is fast but ignores negative labels.
  4. I-DP (Innovation): This incorporates Data Programming by automatically generating "weak models" based on the neighborhoods of both positive and negative examples. A generative model (similar to the Snorkel framework) denoises these signals to create a refined ranking.

Crucially, I-DP is "downstream classifier agnostic," meaning the resulting data can be used to train any model, such as BERT or SVM.

Results and Analysis

The I-DP method consistently outperformed both random sampling and the I-MLT search baseline across multiple datasets, including Reddit titles and private customer service inquiry datasets.

Comparison of I-MLT and I-DP performance on the Reddit "graphics_tablet" category, showing the number of relevant examples found per iteration.

As illustrated in the figure above, I-DP is significantly more effective at maintaining a high yield of relevant examples over many iterations. While the I-MLT baseline's performance drops off as it "runs dry" of easily searchable examples, I-DP continues to find relevant data by leveraging both positive and negative feedback. On the Reddit dataset, I-DP extracted over 16,000 relevant examples, compared to just 6,900 for I-MLT.

Recall and KL-Divergence metrics for I-MLT and I-DP across various Reddit categories.

The overall quality of the expanded data is further validated by recall and vocabulary convergence metrics. A classifier trained on I-DP data achieved 46.6% accuracy on the Reddit task, significantly higher than the 40.8% achieved with I-MLT. Furthermore, the KL-Divergence plots show that I-DP's training set vocabulary converges toward the optimal distribution much faster than competing methods.

Conclusion

I-DP demonstrates that negative labels are highly valuable for steering data expansion away from confusing or overlapping clusters. However, the method faces challenges with "fragmented classes"—intents expressed with entirely different vocabularies. Future work will focus on improving computational efficiency and incorporating diversity-based sampling to better explore these isolated data "islands."

Summary updated

Abstract

Real-world text classification tasks often require many labeled training examples that are expensive to obtain. Recent advancements in machine teaching, specifically the data programming paradigm, facilitate the creation of training data sets quickly via a general framework for building weak models, also known as labeling functions, and denoising them through ensemble learning techniques. We present a fast, simple data programming method for augmenting text data sets by generating neighborhood-based weak models with minimal supervision. Furthermore, our method employs an iterative procedure to identify sparsely distributed examples from large volumes of unlabeled data. The iterative data programming techniques improve newer weak models as more labeled data is confirmed with human-in-loop. We show empirical results on sentence classification tasks, including those from a task of improving intent recognition in conversational agents.
BibTeX
@inproceedings{mallinar2020iterative,
  author = {Neil Mallinar and Abhishek Shah and Tin Kam Ho and Rajendra Ugrani and Ayush Gupta},
  title = {{Iterative Data Programming for Expanding Text Classification Corpora}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=3LpS7JBwTs},
}
Figure