Influence Functions for Sequence Tagging Models

Sarthak Jain, Varun Manjunatha, Byron C. Wallace, Ani Nenkova

TL;DR

This paper introduces segment influence, a method to trace specific token-level predictions in sequence tagging models (like Named Entity Recognition) back to the most influential snippets in the training data. By moving beyond whole-sentence attribution, the authors provide a surgical tool for debugging datasets and identifying systematic annotation errors that standard methods overlook.

What Problem Does This Paper Address?

Deep learning models for sequence tagging—tasks like identifying entities in text or labeling parts of speech—are often "black boxes." When a model incorrectly labels "Manchester United" as a Location instead of an Organization, practitioners want to know why.

Existing interpretability tools, specifically influence functions, typically operate at the "instance level." They can tell you which training sentences were most influential to a prediction, but in a long sentence with multiple labels, this is often too coarse to be useful. If a model makes a mistake on one specific word, instance-level influence might point to a training sentence that is relevant for entirely the wrong reasons. The challenge is to zoom in: how do we calculate the influence of a specific training segment on a specific test segment?

Figure 1: We propose and evaluate influence functions for sequence tagging tasks, which retrieve snippets (from token a to b) in train samples that most influenced predictions for test tokens c through d. Here this reveals a training example in which an ORG is problematically marked as a LOC, leading to the observed error.

Methods

The authors define Exact Segment Influence as the change in loss for a specific test segment if a specific training segment's labels were removed during training. Since retraining a model thousands of times is computationally impossible, they derive an efficient approximation.

Key Innovations:

  • Conditional Loss Upweighting: Instead of upweighting an entire training instance, they apply $\epsilon$-upweighting to the conditional loss of a specific segment. This isolates the influence of those specific labels.
  • Mathematical Approximation: They use a first-order Taylor expansion (involving the Hessian of the loss and gradients) to estimate influence without retraining.
  • Efficiency Simplifications: To make this practical for large models like BERT or BigBird, the authors:
  • Only consider gradients with respect to the final output layer (the CRF parameters).
  • Approximate the Hessian matrix as the identity (ignoring it to save massive amounts of memory).
  • Focus on single-token segments to reduce the search space.

Datasets

The researchers evaluated their method on two primary Named Entity Recognition (NER) benchmarks:

  1. CoNLL-2003: A classic news-based NER dataset. It is known to contain "systematic" noise, such as sports teams (Organizations) frequently being mislabeled as Locations.
  2. EBM-NLP: A dataset of medical abstracts where "entities" are PICO elements (Population, Intervention, Comparison, Outcome). This dataset contains "lay-annotator" noise, particularly regarding whether drug dosages should be included in the "Intervention" label.

Key Insights and Ideas

  • Granularity is King: Instance-level influence is often too "noisy" for structured prediction. Segment influence allows for a "surgical" look at data artifacts.
  • Systematic vs. Random Noise: The authors found that while simple baselines (like ranking by loss) are good at finding random labeling mistakes, segment influence is significantly better at surfacing systematic errors—patterns of mislabeling that the model has actually "learned" and is now replicating.
  • The "Aha!" Moment in EBM-NLP: The authors discovered a major inconsistency in the EBM-NLP dataset using their tool: some annotators included dosages (e.g., "20 mg") in the Intervention tag, while others did not. Segment influence clearly surfaced these conflicting training examples.

Key Results

  • Approximation Validity: The authors compared their fast approximation to the "exact" (but slow) influence calculated by brute-force retraining on a smaller model. They found a high Pearson correlation of 0.89, proving the approximation is reliable.
Figure 2: Predicted versus actual change in conditional loss over a sample of mispredicted validation tokens from CoNLL. The dotted line shows the line of best fit to the observations (solid line is perfect correlation).
  • Synthetic Debugging: In a controlled experiment where they intentionally "poisoned" 10% of the data with a specific artifact, segment influence identified the correct culprit 100% of the time, whereas standard instance-level influence only succeeded 26.3% of the time.
  • Real-World Utility: On the EBM-NLP dataset, segment influence identified conflicting dosage labels in 97.9% of cases, compared to just 35.4% for instance-level influence.
Table 2: Dosage label inconsistency as identified via different attribution methods. For each method, we are showing an example of the most typical result achieved. Segment Influence can identify the inconsistent labels for dosage entities in the training dataset. Segment NN retrieves similar examples when explicitly checking for inconsistency.

Limitations and Future Work

  • Memory Overhead: Even with simplifications, storing gradients for every token in a large dataset is demanding. For CoNLL with a BERT-base model, it requires roughly 9GB of storage; for longer sequences or larger models, this could balloon to hundreds of gigabytes.
  • Model Architecture: The current derivation is optimized for Linear-Chain CRFs. Extending this to more complex autoregressive models (like Seq2Seq transformers used in translation) is mathematically more difficult because the normalization term depends on the entire sequence.
  • Non-Convexity: The "exact influence" baseline assumes a global minimum, but retraining with partial labels is non-convex, meaning the "true" influence might be slightly different than calculated.

Summary updated

Abstract

Many language tasks (e.g., Named Entity Recognition, Part-of-Speech tagging, and Semantic Role Labeling) are naturally framed as sequence tagging problems. However, there has been comparatively little work on interpretability methods for sequence tagging models. In this paper, we extend influence functions - which aim to trace predictions back to the training points that informed them - to sequence tagging tasks. We define the influence of a training instance segment as the effect that perturbing the labels within this segment has on a test segment level prediction. We provide an efficient approximation to compute this, and show that it tracks with the true segment influence, measured empirically. We show the practical utility of segment influence by using the method to identify systematic annotation errors in two named entity recognition corpora. Code to reproduce our results is available at https://github.com/successar/Segment_Influence_Functions.
BibTeX
@inproceedings{jain2022influence,
  author = {Sarthak Jain and Varun Manjunatha and Byron C. Wallace and Ani Nenkova},
  title = {{Influence Functions for Sequence Tagging Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.58},
}
Figure