Adaptive Active Learning for Coreference Resolution

Michelle Yuan, Patrick Xia, Benjamin Van Durme, Jordan L. Boyd-Graber

Abstract

Adapting coreference resolution (CR) models to new domains is notoriously expensive because annotators must read entire documents to understand entity relationships. This paper demonstrates that active learning—specifically targeting uncertainty in mention detection rather than just entity linking—can significantly cut costs. Furthermore, the authors find that labeling multiple spans within a single document is far more efficient than sampling across many documents, as it minimizes the "reading tax" paid by human annotators.

Introduction

Coreference Resolution (CR) is the task of determining which words or phrases in a text refer to the same real-world entity (e.g., recognizing that "the president," "he," and "Joe Biden" all point to the same person). While state-of-the-art neural models perform well on standard datasets like OntoNotes, they struggle to generalize to new domains like legal documents or biomedical texts.

The primary hurdle is data: CR requires "copiously labeled" in-domain data. However, labeling CR is uniquely difficult compared to simple text classification. An annotator can't just look at a single word; they must read and hold the context of the entire document in their head to identify links. This makes traditional active learning (AL)—which usually assumes every "label" has a fixed, independent cost—inefficient for CR.

The Challenge of Domain Transfer

Domain shifts often introduce new types of entities or change the usage of common words. For instance, the word "bonds" might refer to financial instruments in a news context but to chemical forces in a science text. Identifying whether a span is even a mention is a critical first step that often fails during domain transfer.

Figure 1: Examples of domain shift where the word "bonds" changes meaning and entity status across financial, chemical, and biological contexts.

The authors tackle two main questions:

  1. Which types of model uncertainty (detecting mentions vs. linking them) are most useful for sampling?
  2. How do we balance the cost of reading a document versus the cost of labeling a specific span?

Methodology: Entropy-Based Sampling

The researchers implemented an active learning framework using ICOREF, an incremental variant of the standard C2F-COREF model. Their approach introduces several new entropy-based sampling strategies to identify "confusing" spans:

  • ment-ent (Mention Entropy): Targets spans where the model is unsure if the text is even an entity mention at all.
  • clust-ent (Clustering Entropy): Targets spans where the model knows it's a mention but isn't sure which entity cluster it belongs to.
  • cond-ent & joint-ent: Mathematical combinations of the two above, looking for spans that are difficult to both detect and link.

Crucially, they also modeled the Reading-Labeling Trade-off. They compared a "FewDocs" strategy (labeling many spans in one document) against a "ManyDocs" strategy (labeling one span across many documents).

Key Results

The study utilized three primary datasets: OntoNotes 5.0 (source), PreCo (grade-school reading), and QBcoref (trivia).

Performance Gains

On the PreCo dataset, the ment-ent strategy reached an AVG F1 score of ~0.71 with only 300 labeled spans, compared to the source model's starting point of ~0.40. Surprisingly, sampling for uncertainty in mention detection (ment-ent) was often more effective than sampling for linking uncertainty. This is because many domain shifts involve the model failing to recognize what constitutes an entity in the first place.

Figure 5: Comparison of active learning strategies. The `ment-ent` strategy (blue) consistently outperforms others, reaching higher F1 scores with fewer labeled spans.

The "Reading Tax" and Efficiency

In a human user study, participants labeled twice as many spans in the "FewDocs" setting compared to "ManyDocs." Restricting the number of documents an annotator has to read significantly increases throughput without sacrificing model performance. The "FewDocs" group achieved a higher F1 score in the same amount of time because they spent less time context-switching between different stories and more time actually providing labels.

Conclusion

The research highlights that the bottleneck in coreference resolution domain transfer is often mention detection rather than entity linking. By focusing active learning on mention uncertainty and minimizing the "reading tax" through document-constrained sampling, researchers can adapt models to new domains with significantly less human effort. Future work includes applying these blueprints to low-resource languages and integrating adaptive CR into open-domain question-answering systems.

Summary updated

Abstract

Neural coreference resolution models trained on one dataset may not transfer to new, low-resource domains. Active learning mitigates this problem by sampling a small subset of data for annotators to label. While active learning is well-defined for classification tasks, its application to coreference resolution is neither well-defined nor fully understood. This paper explores how to actively label coreference, examining sources of model uncertainty and document reading costs. We compare uncertainty sampling strategies and their advantages through thorough error analysis. In both synthetic and human experiments, labeling spans within the same document is more effective than annotating spans across documents. The findings contribute to a more realistic development of coreference resolution models.
BibTeX
@inproceedings{yuan2022adaptive,
  author = {Michelle Yuan and Patrick Xia and Benjamin Van Durme and Jordan L. Boyd-Graber},
  title = {{Adaptive Active Learning for Coreference Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.519},
}
Figure