Abstract
Real-world text classification tasks often require many labeled training examples that are expensive to obtain. Recent advancements in machine teaching, specifically the data programming paradigm, facilitate the creation of training data sets quickly via a general framework for building weak models, also known as labeling functions, and denoising them through ensemble learning techniques. We present a fast, simple data programming method for augmenting text data sets by generating neighborhood-based weak models with minimal supervision. Furthermore, our method employs an iterative procedure to identify sparsely distributed examples from large volumes of unlabeled data. The iterative data programming techniques improve newer weak models as more labeled data is confirmed with human-in-loop. We show empirical results on sentence classification tasks, including those from a task of improving intent recognition in conversational agents.
Introduction
The "data bottleneck" remains the primary obstacle in developing effective text classifiers, such as those used in conversational agents. While machine learning algorithms have become increasingly sophisticated, they remain dependent on large volumes of labeled data, which is costly and slow to produce. This problem is particularly acute when dealing with highly skewed classes—rare intents buried within millions of unlabeled logs. Traditional active learning methods often struggle in these scenarios because they require a reasonably accurate starting model to be effective, creating a "chicken and egg" problem for new projects.
Methodology
The authors propose Iterative Data Programming (I-DP), a human-in-the-loop framework designed to rapidly expand small, labeled text datasets. I-DP bridges the gap between simple search-based retrieval and complex generative modeling.
The system operates through an iterative expansion procedure:
- Selection: A selection model ranks the unlabeled corpus and presents a batch to a human annotator.
- Labeling: The human labels the batch, and the system updates its model.
- I-MLT (Baseline): This uses "More Like This" search (via Elasticsearch), treating positive examples as queries. It is fast but ignores negative labels.
- I-DP (Innovation): This incorporates Data Programming by automatically generating "weak models" based on the neighborhoods of both positive and negative examples. A generative model (similar to the Snorkel framework) denoises these signals to create a refined ranking.
Crucially, I-DP is "downstream classifier agnostic," meaning the resulting data can be used to train any model, such as BERT or SVM.
Results and Analysis
The I-DP method consistently outperformed both random sampling and the I-MLT search baseline across multiple datasets, including Reddit titles and private customer service inquiry datasets.
As illustrated in the figure above, I-DP is significantly more effective at maintaining a high yield of relevant examples over many iterations. While the I-MLT baseline's performance drops off as it "runs dry" of easily searchable examples, I-DP continues to find relevant data by leveraging both positive and negative feedback. On the Reddit dataset, I-DP extracted over 16,000 relevant examples, compared to just 6,900 for I-MLT.
The overall quality of the expanded data is further validated by recall and vocabulary convergence metrics. A classifier trained on I-DP data achieved 46.6% accuracy on the Reddit task, significantly higher than the 40.8% achieved with I-MLT. Furthermore, the KL-Divergence plots show that I-DP's training set vocabulary converges toward the optimal distribution much faster than competing methods.
Conclusion
I-DP demonstrates that negative labels are highly valuable for steering data expansion away from confusing or overlapping clusters. However, the method faces challenges with "fragmented classes"—intents expressed with entirely different vocabularies. Future work will focus on improving computational efficiency and incorporating diversity-based sampling to better explore these isolated data "islands."
Summary updated