TL;DR
This paper investigates the prevalence of label errors in the widely-used BANKING77 intent classification dataset. By applying two automated methods, the authors found that a significant portion (14%) of the training data may be mislabeled, negatively impacting intent classifier performance. Removing these potentially erroneous labels led to notable improvements in both supervised and unsupervised classification metrics.
What Problem Does This Paper Address?
In natural language processing (NLP), benchmark datasets are crucial for developing, evaluating, and comparing different models. These datasets allow researchers to ensure fair comparisons and understand how methods perform under various conditions. However, many of these datasets are hand-labeled by humans, which can introduce errors due to subjectivity or simple mistakes. The BANKING77 dataset, a popular benchmark for intent classification with over 10,000 utterances and 77 intents, is one such dataset. The authors' own experience building an intent classifier for this dataset revealed unexpectedly poor performance, suggesting potential issues with the data itself. This paper aims to systematically identify and quantify these label errors in BANKING77 and demonstrate their impact on downstream NLP tasks.
Methods
The authors employed two primary automated approaches to identify potential label errors:
- Confident Learning Framework: This method, based on the work of Northcutt et al., estimates label noise by analyzing the joint distribution of noisy (given) labels and latent (unknown) labels. They trained a LightGBM classifier on SBERT sentence embeddings and used 10-fold cross-validation to generate out-of-sample predictions for error identification.
- Cosine Similarity Approach: This method was designed to catch semantic similarities between utterances that were assigned different labels. They calculated pairwise cosine similarity using SBERT MPNet embeddings and identified utterance pairs with a similarity score above 0.85 that had distinct labels.
Datasets
- BANKING77: This is the primary dataset investigated in the paper. It contains over 10,000 annotated customer service queries across 77 banking-related intents. The authors used the original, unmodified version and a "trimmed" version where potentially mislabeled utterances were removed.
Key Insights and Ideas
- Label Noise is Pervasive: The study highlights that even popular, widely-used datasets like BANKING77 can contain a substantial amount of label noise. This challenges the assumption of data quality in benchmark datasets.
- Impact on Performance: The presence of noisy labels can significantly degrade the performance of intent classifiers, leading to inaccurate evaluations and potentially misleading conclusions about model effectiveness.
- Conflicting Labels for Similar Semantics: The cosine similarity approach revealed instances where utterances with very similar meanings were assigned different labels, indicating a lack of clear semantic boundaries or inconsistent labeling.
- Specific Problematic Intents: The analysis pointed to certain intents, particularly those related to "transfers" and "top-up," as being more prone to label errors.
Key Results
- The authors identified over 1,400 (approximately 14%) of the 10,003 training utterances in BANKING77 as potentially mislabeled.
- The Confident Learning framework flagged 965 utterances, while the Cosine Similarity approach identified 590 utterances. A total of 127 utterances were flagged by both methods.
- Removing these potentially erroneous utterances led to significant performance improvements:
- Supervised Classifier (LightGBM): F1-Score increased by 4.5%.
- Unsupervised Classifier (Agglomerative Clustering): Adjusted Rand Index (ARI) increased by 8%.
- Table 4 shows that trimming the dataset reduced the number of unique labels from 77 to 77 (meaning no labels were removed entirely, but utterances within them were), reduced the total number of utterances, and slightly altered token counts.
Limitations and Future Work
- Manual Verification Needed: While automated methods are effective, the authors acknowledge that manual verification is still needed to confirm the accuracy of their findings and establish a ground truth.
- Automated Correction Tool: A promising future direction is to develop an automated tool that can not only identify but also correct label errors.
- Broader Dataset Application: The methodology should be applied to other benchmark datasets, such as CLINC150 and HWU64, to assess the generalizability of label error issues.
- Understanding Error Causes: Further investigation into why these errors occur (e.g., ambiguity in intent definitions, annotator fatigue) would be beneficial.
Summary updated