TL;DR
This paper introduces and evaluates token-level metrics for identifying incorrect labels in Named Entity Recognition (NER) datasets. They propose a novel "token category" approach and find that training dynamics metrics are most effective, significantly improving noise reduction and leading to modest gains in downstream performance after retraining.
What Problem Does This Paper Address?
Supervised learning for NLP tasks like Named Entity Recognition (NER) relies heavily on labeled data. However, these datasets often contain "label noise" – incorrect annotations. This noise can significantly degrade model performance. While various methods exist to filter out noisy samples, it's challenging to compare their effectiveness, especially when integrated into complex pipelines. Existing approaches often use computationally expensive techniques. This paper aims to systematically evaluate and compare different token-level metrics for detecting incorrect annotations in NER, focusing on efficiency and effectiveness.
Methods
The authors propose a disaggregated approach by dividing tokens into four categories: "Learned-O," "Misclassified-O," "Learned-entity," and "Misclassified-entity." This allows for more targeted metric selection. They evaluate three types of token-level metrics:
- Static Metrics: Calculated from a single training epoch (e.g., cross-entropy, maximum softmax probability (MSP), prediction margin, predictive entropy).
- Training Dynamics Metrics: Aggregate information over multiple epochs (e.g., confidence, variability, correctness, iteration learned, MILD, entropy-history).
- Layer Agreement Metrics: Consider predictions from each transformer block, probing prediction consistency and model complexity.
They then use these metrics to flag potentially noisy tokens and explore two strategies for handling them: masking (excluding from training) and relabeling (assigning the model's prediction). The effectiveness of these methods is evaluated by measuring noise reduction and the impact on downstream model performance after retraining.
Datasets
- NoiseBench: A dataset specifically designed for benchmarking the impact of real label noise on NER. It includes multiple noisy-labeled variants based on English CoNLL-03, with six types of noise (expert, crowdsourcing, distant supervision, weak supervision, LLM labels).
- German NoiseBench: A German variant of NoiseBench, based on German CoNLL-03, with expert and LLM label noise types.
- NoisyNER: An Estonian dataset covering various levels of distant supervision noise.
Key Insights and Ideas
- Token Categories Matter: Separating tokens into categories (Learned vs. Misclassified, O vs. Entity) significantly improves the ability to discriminate between clean and noisy samples compared to treating all tokens uniformly.
- Training Dynamics Lead the Pack: Metrics that consider the model's learning trajectory over multiple epochs (training dynamics metrics) generally outperform static metrics and layer agreement metrics in detecting label errors.
- Memorization is the Enemy: Errors that the model has already memorized (Learned categories) are much harder to detect and correct than those the model is still learning (Misclassified categories).
- Relabeling is Often Better Than Masking: For many error types, relabeling detected noisy tokens with the model's predictions is more effective than simply masking them, suggesting the model's predictions are often accurate for these errors.
- Naive Approaches Can Be Surprisingly Effective: For certain categories (like Misclassified-O), simple strategies like relabeling based on a low threshold can achieve near-optimal detection performance.
Key Results
- Noise Reduction: Relabeling successfully reduced noise share by an average of 3.3 percentage points across datasets and variants.
- Performance Gains: Retraining models on datasets with modified labels led to modest but statistically significant improvements in test F1 scores for most datasets and variants, with the largest gains observed on datasets with distant supervision noise.
- Metric Performance: Training dynamics metrics like confidence and variability showed the highest F0.5 scores for detecting errors, particularly in the "Learned-O" category. Static metrics performed well for "Misclassified-O" and "Misclassified-entity." Layer agreement metrics generally performed poorly.
- Category-Specific Performance: "Misclassified-O" and "Misclassified-entity" errors were generally easier to detect and correct than "Learned-entity" errors.
Limitations and Future Work
- Clean Data Dependency: The metric selection process requires access to clean training data to determine ground truth label quality, which is not always available.
- Metric Scope: The study focused on token-level metrics for NER. Further research could explore these metrics for other NLP tasks or even across different domains.
- Combined Strategies: Combining modifications across multiple categories did not always yield the best results, suggesting that category-specific strategies might be more beneficial. Future work could explore more sophisticated ways to combine these strategies.
- Generalizability: While the study spanned three languages, exploring a wider range of languages and tasks could further validate the generalizability of the findings.
Summary updated