Understanding Points of Correspondence between Sentences for Abstractive Summarization

Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Walter Chang, Fei Liu

Abstract

Fusing sentences containing disparate content is a remarkable human ability that helps create informative and succinct summaries. Such a simple task for humans has remained challenging for modern abstractive summarizers, substantially restricting their applicability in real-world scenarios. In this paper, we present an investigation into fusing sentences drawn from a document by introducing the notion of points of correspondence, which are cohesive devices that tie any two sentences together into a coherent text. The types of points of correspondence are delineated by text cohesion theory, covering pronominal and nominal referencing, repetition and beyond. We create a dataset containing the documents, source and fusion sentences, and human annotations of points of correspondence between sentences. Our dataset bridges the gap between coreference resolution and summarization. It is publicly shared to serve as a basis for future work to measure the success of sentence fusion systems. (https://github.com/ucfnlp/points-of-correspondence)

Introduction

In abstractive summarization, "sentence fusion" is considered a critical capability. It involves merging two disparate sentences into a single, informative summary. However, modern neural models often struggle with this task, frequently producing "unfaithful" summaries or "hallucinations" where entities are incorrectly linked. This research addresses the root of this problem: a lack of understanding regarding text cohesion. We introduce the concept of Points of Correspondence (PoC)—the linguistic "glue" such as pronouns or shared events that link sentences—and provide a new human-annotated dataset to help models learn how to fuse information coherently.

Defining Points of Correspondence

We formalize the study of sentence fusion by defining five distinct types of PoC based on text cohesion theory. These categories represent the shared information that justifies merging two sentences:

  1. Pronominal Referencing: Links between a noun and a pronoun (e.g., "The bodies" → "They").
  2. Nominal Referencing: Links between a specific name and a descriptive phrase (e.g., "Johnny Kemp" → "The singer").
  3. Common-Noun Referencing: Links between different descriptions of the same group or entity (e.g., "five women" → "the victims").
  4. Repetition: Exact word matches (e.g., "Felix" → "Felix").
  5. Event Triggers: Related actions or states (e.g., "arrested" → "convicted").
Examples of the five types of Points of Correspondence (PoC) across source and summary sentences.

The PoC Dataset

To facilitate research, we developed the PoC Dataset, derived from the CNN/DailyMail corpus. It contains 1,599 instances across 1,174 documents. Each instance includes a pair of source sentences, a human-written fusion sentence, and human annotations of the PoCs connecting them. Unlike previous datasets that focused on near-paraphrases, this dataset specifically targets disparate sentences containing different but related information, which is a more realistic and difficult task for summarization models.

Experimental Analysis and Results

We evaluated how well state-of-the-art coreference resolvers and abstractive summarizers handle these correspondences.

Coreference Resolver Limitations

Standard coreference resolution tools (SpaCy, AllenNLP, and Stanford CoreNLP) struggle significantly with identifying PoCs. While they maintain decent precision, their recall is remarkably low, ranging from 20% to 26%. These tools are particularly ineffective at identifying "Event Triggers" and "Common-Noun Referencing," which are crucial for complex sentence fusion.

Performance of state-of-the-art coreference resolvers on identifying different PoC types.

Summarizer Performance

Neural models like the Pointer-Generator and Transformer were also benchmarked. While the Transformer model achieved a superior ROUGE-1 score of 38.81, models remain "fusion-shy." Neural models attempt to merge sentences far less frequently than humans (approx. 6% vs 32% in human summaries). When they do attempt fusion, the lack of PoC awareness often leads to factual errors because they fail to recognize that different descriptions refer to the same entity.

Conclusion and Future Work

Our investigation reveals that identifying the linguistic "glue" between sentences is a major hurdle for current NLP systems. Future work should focus on "PoC-aware" summarizers that explicitly incorporate coreference and event resolution into their architectures. Moving toward "Deep Sentence Fusion" will require models to understand not just entities, but how disparate events and descriptions relate to form a coherent narrative.

Summary updated

Abstract

Fusing sentences containing disparate content is a remarkable human ability that helps create informative and succinct summaries. Such a simple task for humans has remained challenging for modern abstractive summarizers, substantially restricting their applicability in real-world scenarios. In this paper, we present an investigation into fusing sentences drawn from a document by introducing the notion of points of correspondence, which are cohesive devices that tie any two sentences together into a coherent text. The types of points of correspondence are delineated by text cohesion theory, covering pronominal and nominal referencing, repetition and beyond. We create a dataset containing the documents, source and fusion sentences, and human annotations of points of correspondence between sentences. Our dataset bridges the gap between coreference resolution and summarization. It is publicly shared to serve as a basis for future work to measure the success of sentence fusion systems. (https://github.com/ucfnlp/points-of-correspondence)
BibTeX
@inproceedings{lebanoff2020understanding,
  author = {Logan Lebanoff and John Muchovej and Franck Dernoncourt and Doo Soon Kim and Lidan Wang and Walter Chang and Fei Liu},
  title = {{Understanding Points of Correspondence between Sentences for Abstractive Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-srw.26},
}
Figure