Summarizing Student Responses to Reflection Prompts

Wencan Luo, Diane J. LitmanEMNLP 2015 · DOI

Abstract

In large-scale educational environments, instructors often utilize reflection prompts to monitor student comprehension. However, the volume of responses can be overwhelming. This paper presents a novel phrase-based summarization algorithm that extracts noun phrases, clusters them semantically using K-Medoids and Latent Semantic Analysis, and ranks them based on "student coverage." Evaluation on a dataset of undergraduate responses demonstrates that this approach significantly outperforms traditional sentence-based summarization methods like LexRank and MEAD in terms of ROUGE scores.

Introduction

Instructors in large STEM courses or MOOCs frequently use reflection prompts (e.g., "What was the most confusing part of today's lecture?") to identify student difficulties. Manually processing hundreds of these responses is an onerous task. Existing automated summarization tools often fail because student responses vary significantly in length and linguistic structure. Furthermore, standard algorithms prioritize centrality or diversity, whereas educational contexts require prioritizing "coverage"—the number of unique students mentioning a specific concept.

Methods

The proposed phrase summarization pipeline consists of three stages:

  1. Candidate Phrase Extraction: Using the Senna syntax parser, the system extracts Noun Phrases (NPs) to focus on core knowledge concepts rather than filler text.
  2. Phrase Clustering: To handle lexical variability (e.g., "bicycle parts" vs. "bike elements"), phrases are grouped using K-Medoids clustering. Semantic distance is calculated via Latent Semantic Analysis (LSA) trained on an educational corpus.
  3. Phrase Ranking: Importance is defined by "student coverage"—the count of unique students contributing to a cluster. Within each cluster, LexRank is used to select the most representative phrase for the final summary.

Experimental Setup

The system was evaluated using a dataset from 53 undergraduates in an introductory materials science course, covering 25 lectures and over 900 responses. Reference summaries were provided by Teaching Assistants (TAs) to establish a gold standard for both content and student coverage counts.

Table 1: Descriptive statistics of the student responses and TA summaries, including word counts (WC) and phrase counts (PC).

Results

The proposed method was compared against several baselines, including Keyphrase extraction (Maui), LexRank, and MEAD, using ROUGE metrics. The results indicate that the phrase-based approach achieves significantly higher F-measures across R-1, R-2, and R-SU4 metrics.

Table 2: ROUGE evaluation results comparing the proposed phrase-based method against traditional summarization baselines.

Statistical tests confirmed that the model outperformed LexRank in precision across all metrics. Furthermore, the use of LexRank for intra-cluster selection proved superior to simple medoid selection, effectively reducing redundancy while maintaining representativeness. These findings suggest that semantic clustering combined with coverage-based ranking aligns more closely with human pedagogical summarization than standard graph-based centrality alone.

Conclusion

Phrase-based summarization offers a more effective way to digest student feedback, particularly for mobile interfaces where screen space is limited. By focusing on semantic coverage rather than just lexical overlap, the system provides instructors with a clearer picture of classroom needs. Future work will explore including broader linguistic constituents and conducting extrinsic evaluations to measure the impact on actual teaching outcomes and student learning.

Summary updated

BibTeX
@inproceedings{luo2015summarizing,
  author = {Wencan Luo and Diane J. Litman},
  title = {{Summarizing Student Responses to Reflection Prompts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1227},
}
Figure