Supporting content evaluation of student summaries by Idea Unit embedding

Marcello Gecchele, Hiroaki Yamada, Takenobu Tokunaga, Yasuyo SawakiACL 2019

TL;DR

This paper introduces a novel method for automatically evaluating student summaries by aligning "Idea Units" (IUs) from the summary to the source text using word embeddings. This approach proves more robust than traditional n-gram based methods like ROUGE, especially when dealing with rephrased content, and achieves higher recall, which is crucial when human judgment ultimately decides the alignment.

What Problem Does This Paper Address?

Evaluating student summaries is a critical but challenging task for educators. Summarization requires students to comprehend complex texts and then rephrase key information concisely, a process that often involves significant paraphrasing. This rephrasing makes it difficult for automated evaluation tools, which typically rely on exact word or phrase matching, to accurately assess whether the summary captures the essential ideas of the original source. Traditional metrics like ROUGE struggle when sentences are rephrased, leading to potentially inaccurate assessments of summary quality.

Methods

The core of this paper's contribution lies in its "Idea Unit" (IU) based alignment method.

  1. Segmentation into Idea Units (IUs): Texts (both source and summary) are first broken down into "Idea Units," defined as minimal fragments conveying a coherent idea. This is a refinement of Kroll's (1977) definition, aiming for more granular units than sentences.
  2. Word Embedding for Similarity: To handle rephrasing, the authors represent each IU as a vector by averaging the GloVe word embeddings of its constituent words. This allows for semantic similarity to be captured even when the exact words differ.
  3. Prediction Ranking: For each Summary IU, its cosine similarity to all Source IUs is calculated. This generates a "Prediction Ranking" of Source IUs, ordered by similarity.
  4. Evaluation Metrics: The performance of this Vector-based ranking is compared against ROUGE-1, ROUGE-2, and ROUGE-L baselines using precision and recall. The authors emphasize recall as being more important since the final decision on IU alignment is left to the end user.

Datasets

The authors created a dataset consisting of:

  • Two source texts on "the preservation of endangered languages" and "the impact of noise on cognitive abilities."
  • Ten student summaries for each source text, written by Ph.D. students from the University of Cambridge.
  • Manually created "gold" IU links between summary and source texts, serving as the ground truth for evaluation.

Key Insights and Ideas

  • Idea Units are Key for Rephrasing: Breaking down texts into granular "Idea Units" is more effective than sentence-level analysis for capturing the essence of summaries, especially when rephrasing is involved.
  • Word Embeddings Bridge the Rephrasing Gap: Using word embeddings (GloVe) to represent IUs allows the system to recognize semantic similarity between rephrased content, a significant improvement over purely lexical matching.
  • Recall is Paramount for Human-in-the-Loop Evaluation: The method's higher recall is particularly valuable because it ensures that most relevant source information is presented to the human evaluator, who then makes the final judgment.

Key Results

  • The proposed Vector-based IU alignment method demonstrated comparable precision to ROUGE-based baselines.
  • Crucially, the method outperformed ROUGE baselines in recall, with the difference becoming more pronounced as the rank threshold increased. For example, in one case, recall reached 0.82 at a threshold of 6, more than double that of the baselines.
  • The method showed robustness against rephrasing, successfully linking semantically similar IUs that shared very few words, a task where ROUGE baselines failed significantly.

Limitations and Future Work

  • Manual Segmentation: A significant limitation is the manual segmentation of texts into IUs, which is time-consuming.
  • Future Work: The authors plan to implement an automatic segmentation algorithm to address this limitation, which they consider a "mission-critical feature." They also intend to conduct real-world tests before releasing their tool.

Summary updated

BibTeX
@inproceedings{gecchele2019supporting,
  author = {Marcello Gecchele and Hiroaki Yamada and Takenobu Tokunaga and Yasuyo Sawaki},
  title = {{Supporting content evaluation of student summaries by Idea Unit embedding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure