Learning Word Representations with Cross-Sentence Dependencyfor End-to-End Co-reference Resolution

Hongyin Luo, Jim Glass

Abstract

Modern neural co-reference resolution models often process documents sentence-by-sentence to maintain computational efficiency, leading to a "context gap" that hinders the resolution of long-range dependencies. This paper introduces Attentional Sentence Linking (ASL), a method to enrich word embeddings with cross-sentence context. By treating the previous sentence as external memory and employing an attention mechanism with a learnable context gate, the model can better resolve ambiguous pronouns. Our approach, integrated into an end-to-end architecture, achieves a state-of-the-art average F1 score of 67.8% on the CoNLL-2012 benchmark.

Introduction

Co-reference resolution—the task of clustering all mentions in a text that refer to the same real-world entity—is fundamental to natural language understanding. For instance, a system must recognize that "the Navy destroyer," "USS Cole," and "the ship" all refer to the same object.

While modern neural models are effective, they typically process documents in batches of isolated sentences to remain computationally efficient. This creates a significant challenge: if a pronoun like "it" appears in one sentence but its antecedent was in the previous one, a standard LSTM encoder lacks the necessary semantic information to link them. While processing the entire document as a single sequence is a theoretical solution, it suffers from vanishing gradients and prohibitive computational costs.

Methodology

The authors propose two primary methods to bridge the inter-sentence gap without sacrificing efficiency:

Linear Sentence Linking (LSL)

A straightforward approach where the initial state of the LSTM for the current sentence is initialized using the final hidden states of the previous sentence. This effectively passes a summary of the past context forward.

Attentional Sentence Linking (ASL)

A more sophisticated approach inspired by memory networks:

  • External Memory: The model treats all words from the previous sentence as an external memory block.
  • Attention Mechanism: For every word in the current sentence, the model calculates an attention score over the words in the previous sentence to identify relevant context.
  • Context Gate: A learnable gate ($g_t$) determines the balance between the current word’s features and the retrieved context. For vague pronouns, the gate allows more information from the previous sentence to flow into the embedding.

These context-aware embeddings are then integrated into the end-to-end co-reference resolution architecture established by Lee et al. (2017).

Experimental Setup

The models were trained and evaluated on the English corpus of the CoNLL-2012 shared task, the standard benchmark for large-scale co-reference resolution.

Results

The proposed models were compared against several strong baselines, including the previous state-of-the-art by Lee et al. (2017). The results, summarized in the table below, demonstrate that both LSL and ASL improve performance, with ASL providing the most significant gains.

Performance comparison of the proposed models against state-of-the-art baselines on the CoNLL-2012 shared task.

The E2E-CR + ASL model achieved an average F1 score of 67.8%, a 0.6% absolute improvement over the baseline. While seemingly modest, this represents a significant gain in the highly competitive CoNLL-2012 task, particularly as it specifically addresses the difficult problem of cross-sentence entity linking.

Discussion and Analysis

A key insight from this work is that word representations themselves are often the bottleneck in co-reference resolution. If a pronoun's vector representation does not capture the semantics of its antecedent, downstream linking logic will struggle. The Context Gate in ASL is crucial here, as it prevents the model from being overwhelmed by noise when the current word is already semantically rich (e.g., a proper noun).

Qualitative analysis confirms that ASL specifically aids in pronoun resolution. For example, where a baseline model might fail to link "it" to "an SMS," the ASL model succeeds by incorporating the semantics of "SMS" from the preceding sentence into the embedding for "it."

Limitations and Future Work

Currently, the ASL implementation only considers the immediately preceding sentence. In complex narratives, antecedents may be several sentences or paragraphs away. Future work could explore expanding the attention window or applying this cross-sentence dependency approach to other long-range context tasks, such as dialogue generation or reading comprehension.

Summary updated

BibTeX
@inproceedings{luo2018learning,
  author = {Hongyin Luo and Jim Glass},
  title = {{Learning Word Representations with Cross-Sentence Dependencyfor End-to-End Co-reference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure