An Active Learning Approach to Coreference Resolution

Mrinmaya Sachan, Eduard H. Hovy, Eric P. Xing

Abstract

Coreference resolution—the task of grouping mentions that refer to the same entity—traditionally requires extensive and complex human annotation. This paper proposes a lightweight alternative that treats coreference as a constrained clustering problem. By utilizing active learning to solicit simple pairwise "yes/no" judgments, the system identifies the most informative constraints to learn a distance metric. Evaluations across English and Hindi datasets demonstrate that this method achieves competitive performance with significantly less annotated data and manual effort than traditional supervised models.

Introduction

Coreference resolution is a fundamental challenge in natural language processing, essential for understanding which mentions (e.g., "Barack Obama," "he," "the president") refer to the same real-world entity. Conventional supervised approaches face two primary hurdles: the high cognitive load of mapping mentions to global chains and a heavy reliance on domain-specific feature engineering. These factors make traditional systems difficult to adapt to new languages or informal text like social media. This paper introduces an active learning framework designed to simplify the human-in-the-loop process while maintaining high accuracy.

Methodology

The authors frame coreference as a clustering problem governed by pairwise constraints. Mentions are represented as points in a feature space, and coreference chains are formed as clusters. The model learns a distance metric via a hard-Expectation-Maximization (EM) algorithm, guided by "must-link" and "cannot-link" constraints provided by human annotators.

To optimize annotation efficiency, the system employs active learning heuristics to select the most informative pairs for judgment. The authors tested several strategies, including Uncertainty Modeling (UM) and Expected Judgment Error (EJE).

Comparison of active learning heuristics showing the superiority of the ensemble approach.

As shown in the evaluation of heuristics, an ensemble method that aggregates scores from multiple strategies consistently outperforms individual heuristics and random selection, reaching higher F1 scores with smaller training sets.

Evaluation and Results

The system was tested on diverse datasets, including the ACE-2008 English corpus, the IC Event Coreference Corpus, and a new Hindi news dataset. Results show that the model can achieve over 50 CoNLL F1 points on ACE-2008 using only 40% of the available training data.

A critical advantage of this approach is human efficiency. Pairwise judgments are significantly easier for humans to process than traditional cluster-level annotations.

F1 score versus annotation time (man-hours) for the Hindi dataset.

In a timed study on the Hindi dataset, providing pairwise judgments took 44.2 man-hours, whereas traditional cluster-level annotation required 54.4 man-hours. Despite the lower time investment, the pairwise approach resulted in a more accurate model. The system also outperformed state-of-the-art models for event coreference while using significantly fewer features (34 vs. 139).

Conclusion

By simplifying the annotation task to binary pairwise decisions and using active learning to prioritize queries, this approach offers a scalable solution for coreference resolution. While it may lack some specialized linguistic rules found in top-tier supervised systems, its efficiency and adaptability make it ideal for low-resource languages and noisy real-world data. Future work will focus on incorporating richer semantic features to further close the performance gap with heavily engineered supervised models.

Summary updated

Abstract

In this paper, we define the problem of coreference resolution in text as one of clustering with pairwise constraints where human experts are asked to provide pairwise constraints (pairwise judgments of coreferentiality) to guide the clustering process. Positing that these pairwise judgments are easy to obtain from humans given the right context, we show that with significantly lower number of pairwise judgments and feature-engineering effort, we can achieve competitive coreference performance. Further, we describe an active learning strategy that minimizes the overall number of such pairwise judgments needed by asking the most informative questions to human experts at each step of coreference resolution. We evaluate this hypothesis and our algorithms on both entity and event coreference tasks and on two languages.
BibTeX
@inproceedings{sachan2015active,
  author = {Mrinmaya Sachan and Eduard H. Hovy and Eric P. Xing},
  title = {{An Active Learning Approach to Coreference Resolution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure