TL;DR
This paper introduces segment influence, a method to trace specific token-level predictions in sequence tagging models (like Named Entity Recognition) back to the most influential snippets in the training data. By moving beyond whole-sentence attribution, the authors provide a surgical tool for debugging datasets and identifying systematic annotation errors that standard methods overlook.
What Problem Does This Paper Address?
Deep learning models for sequence tagging—tasks like identifying entities in text or labeling parts of speech—are often "black boxes." When a model incorrectly labels "Manchester United" as a Location instead of an Organization, practitioners want to know why.
Existing interpretability tools, specifically influence functions, typically operate at the "instance level." They can tell you which training sentences were most influential to a prediction, but in a long sentence with multiple labels, this is often too coarse to be useful. If a model makes a mistake on one specific word, instance-level influence might point to a training sentence that is relevant for entirely the wrong reasons. The challenge is to zoom in: how do we calculate the influence of a specific training segment on a specific test segment?
Methods
The authors define Exact Segment Influence as the change in loss for a specific test segment if a specific training segment's labels were removed during training. Since retraining a model thousands of times is computationally impossible, they derive an efficient approximation.
Key Innovations:
- Conditional Loss Upweighting: Instead of upweighting an entire training instance, they apply $\epsilon$-upweighting to the conditional loss of a specific segment. This isolates the influence of those specific labels.
- Mathematical Approximation: They use a first-order Taylor expansion (involving the Hessian of the loss and gradients) to estimate influence without retraining.
- Efficiency Simplifications: To make this practical for large models like BERT or BigBird, the authors:
- Only consider gradients with respect to the final output layer (the CRF parameters).
- Approximate the Hessian matrix as the identity (ignoring it to save massive amounts of memory).
- Focus on single-token segments to reduce the search space.
Datasets
The researchers evaluated their method on two primary Named Entity Recognition (NER) benchmarks:
- CoNLL-2003: A classic news-based NER dataset. It is known to contain "systematic" noise, such as sports teams (Organizations) frequently being mislabeled as Locations.
- EBM-NLP: A dataset of medical abstracts where "entities" are PICO elements (Population, Intervention, Comparison, Outcome). This dataset contains "lay-annotator" noise, particularly regarding whether drug dosages should be included in the "Intervention" label.
Key Insights and Ideas
- Granularity is King: Instance-level influence is often too "noisy" for structured prediction. Segment influence allows for a "surgical" look at data artifacts.
- Systematic vs. Random Noise: The authors found that while simple baselines (like ranking by loss) are good at finding random labeling mistakes, segment influence is significantly better at surfacing systematic errors—patterns of mislabeling that the model has actually "learned" and is now replicating.
- The "Aha!" Moment in EBM-NLP: The authors discovered a major inconsistency in the EBM-NLP dataset using their tool: some annotators included dosages (e.g., "20 mg") in the Intervention tag, while others did not. Segment influence clearly surfaced these conflicting training examples.
Key Results
- Approximation Validity: The authors compared their fast approximation to the "exact" (but slow) influence calculated by brute-force retraining on a smaller model. They found a high Pearson correlation of 0.89, proving the approximation is reliable.
- Synthetic Debugging: In a controlled experiment where they intentionally "poisoned" 10% of the data with a specific artifact, segment influence identified the correct culprit 100% of the time, whereas standard instance-level influence only succeeded 26.3% of the time.
- Real-World Utility: On the EBM-NLP dataset, segment influence identified conflicting dosage labels in 97.9% of cases, compared to just 35.4% for instance-level influence.
Limitations and Future Work
- Memory Overhead: Even with simplifications, storing gradients for every token in a large dataset is demanding. For CoNLL with a BERT-base model, it requires roughly 9GB of storage; for longer sequences or larger models, this could balloon to hundreds of gigabytes.
- Model Architecture: The current derivation is optimized for Linear-Chain CRFs. Extending this to more complex autoregressive models (like Seq2Seq transformers used in translation) is mathematically more difficult because the normalization term depends on the entire sequence.
- Non-Convexity: The "exact influence" baseline assumes a global minimum, but retraining with partial labels is non-convex, meaning the "true" influence might be slightly different than calculated.
Summary updated