CORWA: A Citation-Oriented Related Work Annotation Dataset

Xiangci Li, Biswadip Mandal, Jessica OuyangNAACL 2022 · DOI

Introduction

Generating "Related Work" sections is a cornerstone of academic writing, yet it remains a significant challenge for automated systems. Traditional approaches often treat this as a standard summarization task at the sentence level. However, academic sentences are complex; they frequently blend transitions, author reflections, and specific fragments of cited research. Treating the entire sentence as a single unit often leads to "hallucinations" or the inclusion of irrelevant context. To address this, the CORWA dataset shifts the focus from full sentences to citation spans—the precise text fragments derived from cited papers.

Methodology: A Multi-Layered Annotation Scheme

The researchers introduced the Citation-Oriented Related Work Annotation (CORWA) framework, which employs a three-pronged approach to capture the nuances of literature reviews:

  1. Discourse Tagging: Sentences are categorized into roles such as single_summ (summarizing one paper), narrative_cite (high-level observations), or transition (connecting topics).
  2. Citation Span Detection: The model identifies the exact tokens directly attributable to a specific cited work.
  3. Citation Type Recognition: Citations are classified as Dominant (detailed explanations) or Reference (brief, illustrative mentions).
An example of the CORWA annotation scheme, showing discourse tags, citation spans, and the distinction between dominant and reference citations.

To automate this process, the authors developed a Joint Related Work Tagger based on SciBERT. This model utilizes multi-task learning to simultaneously predict discourse labels, identify spans, and classify citation types.

Dataset and Analysis

The CORWA dataset comprises 927 manually annotated related work sections from NLP papers. To scale the training data, the authors created "Distant CORWA" by applying their tagger to over 11,000 additional sections from the S2ORC corpus.

Analysis of the dataset reveals a stark contrast between citation types. Dominant spans tend to be significantly longer, averaging 34.5 tokens, as they provide deep dives into specific methodologies or results. In contrast, Reference spans average only 8.2 tokens, serving as concise pointers to support broader claims.

Distribution of span lengths for Dominant and Reference citations, highlighting the significant difference in token counts between the two types.

Key Results and Evaluation

The SciBERT-based joint tagger demonstrated high efficacy, achieving a micro-F1 score exceeding 0.9 across all three tasks (discourse, type, and span detection). Beyond tagging performance, the researchers found that focusing on spans significantly improves downstream tasks:

  • Retrieval Accuracy: Using specific spans as search queries rather than full sentences improved ROUGE recall when retrieving source sentences from cited papers, particularly for short "reference" citations.
  • Human Preference: In blind evaluations, NLP graduate students preferred span-based generation over traditional sentence-based methods, citing higher relevance and quality.

Conclusion and Future Directions

The CORWA dataset demonstrates that citation spans are a superior unit of measurement for literature review generation compared to full sentences. While currently focused on the NLP domain, the proposed "bottom-up" iterative framework—generating spans, grouping them into discourse blocks, and weaving them with transitions—offers a path toward more human-like, controllable AI writing assistants. Future work will explore the generalizability of this approach across other scientific disciplines.

Summary updated

BibTeX
@inproceedings{li2022corwa,
  author = {Xiangci Li and Biswadip Mandal and Jessica Ouyang},
  title = {{CORWA: A Citation-Oriented Related Work Annotation Dataset}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.397},
}
Figure