Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss

Michael Collins, Thomas Effland, Michael Collins 0001Trans. Assoc. Comput. Linguistics 2021 · DOI

TL;DR

This paper introduces a novel loss function, the Expected Entity Ratio (EER), for Named Entity Recognition (NER) when training data has missing entity annotations (low recall). Their method, EER-BERT, achieves state-of-the-art performance across multiple languages and scenarios, outperforming existing methods by a significant margin, especially in challenging low-resource settings.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) for identifying and classifying entities like people, organizations, and locations in text. While high-capacity models have achieved excellent performance on fully annotated datasets, creating such datasets is expensive and time-consuming, especially for low-resource languages or specialized domains. This paper tackles the challenge of training effective NER models with partially annotated data. This type of data is characterized by high precision (when an entity is annotated, it's likely correct) but low recall (many entities are missed). A common assumption in traditional NER is that unannotated tokens are not entities (the "O" tag). However, with low-recall data, this assumption is problematic, as unannotated tokens are ambiguous. The authors propose a method that treats these unannotated tags as latent variables, allowing for more robust learning from incomplete annotations.

Methods

The core of the paper's contribution lies in its novel loss function, the Expected Entity Ratio (EER), designed to work alongside a standard supervised marginal tag loss.

  • Tagging Model: They employ a BERT-based Transformer encoder followed by a linear-chain Conditional Random Field (CRF) layer. This architecture is standard practice for modern NER systems, leveraging powerful pre-trained language models and the sequence modeling capabilities of CRFs.
  • Supervised Marginal Tag Loss ($L_p$): This is the standard loss used for training on partially observed data. It maximizes the likelihood of the observed tags in the dataset. However, this loss alone can lead to models with high recall but low precision (i.e., predicting many entities, most of which are incorrect).
  • Expected Entity Ratio (EER) Loss ($L_u$): This is the novel component. It aims to control the precision of the model by encouraging the predicted marginal distribution of entity tags to match a pre-defined "expected entity ratio" (EER) for the dataset. This ratio represents the expected proportion of non-"O" tags. The loss is formulated as a margin-based objective, allowing for some uncertainty ($\gamma$) around the target EER. This loss is crucial for preventing the model from simply predicting "O" for everything to achieve high recall.
  • Combined Objective: The final loss function is a weighted sum of the supervised marginal tag loss and the EER loss ($L = L_p + \lambda_u L_u$). This principled combination aims to balance recall and precision.

The authors also provide theoretical justification, showing that under certain assumptions, their combined objective recovers the true tagging distribution in the limit of infinite data.

Datasets

The paper evaluates their approach on a diverse set of datasets:

  • Corpora: They use data from the CoNLL 2003 shared tasks (English, Spanish, German, Dutch) and the Ontonotes 5 corpus (English, Mandarin Chinese, Arabic).
  • Simulated Annotation Scenarios: To mimic real-world low-recall situations, they introduce two scenarios:
  • Non-Native Speaker (NNS): A scenario from prior work (Mayhew et al., 2019) that simulates annotators who miss entities but are generally accurate when they do annotate. This scenario results in approximately 50% recall and 90% precision.
  • Exploratory Expert (EE): A novel scenario introduced by the authors, simulating a domain expert with limited time who annotates entities inexhaustively. This scenario is significantly more challenging, with only 1,000 annotations (less than 10% of the original data) for most datasets.

Key Insights and Ideas

  • Treating unannotated tags as latent variables is key: Instead of assuming unannotated tokens are "O" tags (which fails in low-recall settings), the authors model them as latent variables, allowing for more flexible learning.
  • Balancing recall and precision is crucial: The supervised loss ($L_p$) drives recall, while the novel EER loss ($L_u$) controls precision. This dual objective is essential for effective learning from partially annotated data.
  • EER loss is robust to preprocessing: Unlike many baselines that rely heavily on data cleaning or specific preprocessing steps to handle false negatives, the EER approach is less sensitive to these issues because it doesn't make strong assumptions about unannotated tokens.
  • Sparse annotation can be competitive: The "Exploratory Expert" scenario demonstrates that with their proposed method, sparse annotation schemes can be competitive with, and even outperform, exhaustive annotation at modest budget sizes.

Key Results

  • State-of-the-art performance: EER-BERT significantly outperforms previous state-of-the-art methods on the challenging EE scenario with only 1,000 annotations, achieving an average F1 score of 71.7, outperforming baselines like SNS-BERT by 2.3 F1 points.
  • Efficiency: The EER approach is significantly faster than some baselines (e.g., CBL), as it avoids costly outer self-training loops.
  • Robustness: EER-BERT is less sensitive to preprocessing choices compared to baselines that struggle with the inherent noise of low-recall data.
  • Competitive with exhaustive annotation: In the EE vs. Exhaustive experiments, EER-BERT with sparse annotations achieved competitive results, even outperforming exhaustive annotation at smaller annotation counts.

Limitations and Future Work

  • Theoretical guarantees: While the paper provides theoretical justification for the loss function, the formal proof of convergence to the true distribution under specific distance metrics is left for future work.
  • Hyperparameter sensitivity: The paper investigates the sensitivity of the EER loss to its hyperparameters ($p$ and $\gamma$), suggesting that setting the upper bound of the acceptable EER range ($p + \gamma$) conservatively is important.
  • Hybrid annotation strategies: The authors suggest that a hybrid approach, starting with sparse annotation and potentially switching to exhaustive annotation as the process progresses, could be a promising direction.
  • Broader applicability: The authors believe their EER approach is compatible with various other incomplete annotation scenarios, such as incidental annotations or those derived from distant supervision, which they intend to explore in future work.

Summary updated

BibTeX
@article{collins2021partially,
  author = {Michael Collins and Thomas Effland and Michael Collins},
  title = {{Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00429},
}
Figure