Retrieval-based Text Selection for Addressing Class-Imbalanced Data in Classification

Sareh Ahmadi, Aditya Shah, Edward A. FoxarXiv 2023 · arXiv · OpenReview

TL;DR

This paper tackles the challenge of building accurate text classifiers when dealing with severely imbalanced datasets, particularly when annotation resources are limited. They propose a novel retrieval-based approach that uses SHAP values to generate targeted queries for Elasticsearch and semantic search, effectively identifying and prioritizing minority class examples for annotation, leading to improved classifier performance.

What Problem Does This Paper Address?

Training robust machine learning models, especially for classification tasks, relies heavily on high-quality labeled data. However, real-world datasets often suffer from class imbalance, where one class has significantly fewer examples than others. This imbalance can lead to models that are biased towards the majority class and perform poorly on the minority class, which is often the class of greatest interest (e.g., rare diseases, fraudulent transactions). Furthermore, manual annotation is expensive and time-consuming, making it impractical to label vast amounts of data. This paper addresses the dual challenge of class imbalance and limited annotation budgets by proposing a method to intelligently select which data points to annotate, focusing on those most likely to improve the model's understanding of the minority class.

Methods

The core of the proposed method involves a two-pronged retrieval strategy to identify potentially positive examples for minority classes. First, they leverage Elasticsearch, a powerful search engine, for keyword-based retrieval. Second, they employ semantic search using pre-trained sentence embedding models (specifically, all-mpnet-base-v2 from the Sentence-BERT library) to find texts semantically similar to queries.

The novelty lies in how these queries are generated. Instead of relying on manually crafted keywords, the authors use SHAP (SHapley Additive exPlanations) values. After training an initial classifier on a small batch of labeled data, SHAP values are used to understand which input features (words/phrases) were most influential in predicting a positive class. These influential keywords are then aggregated and refined into effective search queries for both Elasticsearch and semantic search.

To ensure the retrieved examples are truly representative of the minority class, a majority voting mechanism is used: a text is selected only if it is retrieved by both Elasticsearch and semantic search. This combined set of retrieved texts, along with the initially labeled data, forms a more balanced dataset for training improved classifiers. The paper explores several classifiers, including BERT, DistilBERT, XLNet, and SVMs.

Datasets

The study utilizes a dataset of Episodic Future Thinking (EFT) texts. These texts are generated by individuals experiencing conditions like diabetes or obesity, describing future events. The dataset comprises approximately 11,000 cues, each potentially belonging to one of 11 categories. For this research, they focus on three specific binary categories: "Alone," "Friends," and "Health," all of which exhibit severe class imbalance in their positive instances. The paper does not introduce a new benchmark dataset but rather uses this existing one to demonstrate their method's effectiveness.

Key Insights and Ideas

  • SHAP for Query Generation: Using SHAP values to derive search queries is a clever way to bridge the gap between model interpretability and active learning. It allows the model itself to guide the search for more informative data.
  • Synergy of Retrieval Methods: Combining keyword-based search (Elasticsearch) with semantic search is more effective than using either alone. This captures both explicit keyword matches and deeper contextual similarities.
  • Majority Vote for Robustness: Requiring a text to be retrieved by both methods acts as a strong filter, increasing the confidence that the selected text is indeed a positive example of the minority class.
  • Iterative Improvement: The approach is designed to be iterative. As more data is annotated, the classifiers improve, leading to better SHAP-derived queries for subsequent rounds of retrieval, creating a virtuous cycle.
  • Significant Gains for SVMs: The paper highlights that the SVM classifier, a non-neural model, shows particularly significant improvements, suggesting that this targeted sampling strategy is highly effective even when less data is available.

Key Results

The paper demonstrates that their proposed retrieval-based text selection method significantly improves the F1-score for minority classes.

  • In the initial phase with 1600 labeled cues, minority class F1-scores were generally below 90%.
  • After identifying and annotating a small batch of positively selected examples (e.g., 25 for "Alone" and "Friends," 35 for "Health"), F1-scores increased by 1-2% for transformer models.
  • With a second round of identification and annotation, F1-scores saw further substantial improvements. For instance, BERT achieved a macro F1-score of 96% for the "Alone" category.
  • The SVM classifier showed particularly strong gains, with improvements ranging from 2% to 8% across categories for the minority class F1-score.
  • In terms of precision and recall for the retrieval methods, Elasticsearch generally showed higher precision for the "Alone" category, while semantic search excelled in precision for the "Health" category. The combined approach aims to leverage the strengths of both.

Limitations and Future Work

  • Limited Data for Initial Training: The effectiveness of SHAP-derived queries depends on the initial classifier trained on a small batch of data. If this initial batch is not representative, the queries might not be optimal.
  • Manual Verification of Queries: While SHAP helps identify keywords, the paper mentions manual verification to form the final queries, which adds a human-in-the-loop step.
  • Future Work: The authors suggest exploring intelligent sampling strategies that consider other dataset constraints. They also propose fine-tuning the neural retrieval models if a larger quantity of labeled data becomes available to further enhance semantic search performance.

Summary updated

Abstract

This paper addresses the problem of selecting of a set of texts for annotation in text classification using retrieval methods when there are limits on the number of annotations due to constraints on human resources. An additional challenge addressed is dealing with binary categories that have a small number of positive instances, reflecting severe class imbalance. In our situation, where annotation occurs over a long time period, the selection of texts to be annotated can be made in batches, with previous annotations guiding the choice of the next set. To address these challenges, the paper proposes leveraging SHAP to construct a quality set of queries for Elasticsearch and semantic search, to try to identify optimal sets of texts for annotation that will help with class imbalance. The approach is tested on sets of cue texts describing possible future events, constructed by participants involved in studies aimed to help with the management of obesity and diabetes. We introduce an effective method for selecting a small set of texts for annotation and building high-quality classifiers. We integrate vector search, semantic search, and machine learning classifiers to yield a good solution. Our experiments demonstrate improved F1 scores for the minority classes in binary classification.
BibTeX
@misc{ahmadi2023retrieval,
  author = {Sareh Ahmadi and Aditya Shah and Edward A. Fox},
  title = {{Retrieval-based Text Selection for Addressing Class-Imbalanced Data in Classification}},
  year = {2023},
  eprint = {2307.14899},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.14899},
}
Figure