TL;DR
This paper introduces CLASSE, a large-scale, open-source corpus of student-written summaries with expert human annotations, designed to advance automatic summarization evaluation. They also present a baseline model and discuss winning approaches from a Kaggle competition, highlighting the potential for AI to provide valuable feedback to students on their writing.
What Problem Does This Paper Address?
Assessing students' reading comprehension through summarization is a powerful educational tool, but manually grading these summaries is time-consuming and difficult to scale. While automatic summarization evaluation (ASE) methods exist, they often rely on limited, proprietary datasets, hindering broader research and development. This fragmentation, coupled with varied NLP approaches and scoring metrics, makes it challenging to build robust and generalizable ASE models. The paper aims to address this by creating a publicly available, high-quality corpus of student summaries.
Methods
The core of this work is the creation and analysis of the CLASSE corpus. Student summaries were scored by expert raters on six analytic features: main point/gist, details, organization, voice, paraphrasing/wording, and language beyond the source text. These scores were aggregated into two component scores: content and wording.
To understand the underlying structure of these human ratings, the authors performed a Principal Component Analysis (PCA). This revealed two primary components: "content" (encompassing main ideas, details, organization, and objectivity) and "wording" (covering paraphrasing and the use of language beyond the source text).
A baseline prediction model was developed using a Longformer model, a Transformer-based architecture suitable for long documents. This model was fine-tuned to predict the content and wording scores based on the student's summary and the original source text.
Datasets
The primary dataset introduced is the CLASSE (Common Lit Augmented Student Summary Evaluation) corpus. It contains:
- 11,213 summaries written by students in grades 3-12.
- Summaries are based on six different prompts related to various topics.
- Each summary is accompanied by expert human scores on analytic features, aggregated into content and wording scores.
- Metadata includes student grade level and English Language Learner (ELL) status.
The dataset was split into training (n=7,165), validation (n=2,021), and test sets (n=2,027) for the Kaggle competition.
Key Insights and Ideas
- The "Generation Effect" is Powerful: The paper reiterates that writing about what one has read (generation effects) significantly improves learning and comprehension, making summarization a valuable educational practice.
- Human Scoring is Reliable, but Nuanced: While overall inter-rater reliability for human scoring was substantial, paraphrasing/wording showed moderate reliability. This highlights the subjective nature of assessing how well a summary rephrases source material.
- Two Core Dimensions of Summary Quality: PCA revealed that summary quality can be effectively captured by two main components: "content" (what is said) and "wording" (how it is said). This provides a more parsimonious way to think about summary evaluation.
- Open-Source Corpora are Crucial: The lack of large, open-source datasets has been a bottleneck for ASE research. CLASSE aims to democratize this field.
- AI Can Augment Human Expertise: The Kaggle competition demonstrated that advanced NLP models can achieve high performance in scoring summaries, suggesting AI can be a valuable tool for providing students with timely feedback.
Key Results
- The CLASSE corpus contains 11,213 student summaries with expert scores.
- The baseline Longformer model achieved a Mean Columnwise Root Mean Squared Error (MCRMSE) of 0.582 on the test set. While decent, this was outperformed by Kaggle competition winners.
- The Kaggle competition saw over 2,000 teams submit more than 40,000 summary scoring models.
- The top-performing models in the Kaggle competition achieved MCRMSE scores below 0.46, with the first-place team at 0.452.
- These winning models often employed ensemble approaches, particularly using DeBERTa Transformer encoders, and crucially, augmented their training data with AI-generated prompts and summaries.
Limitations and Future Work
- Limited Number of Prompts: The CLASSE corpus only includes summaries from six source texts and prompts. Future work should involve creating more prompts and source texts to improve model generalizability.
- Grade Distribution Skew: The corpus is over-represented by students in grades 6-11, with fewer samples from younger and older grades. Greater representation across all grade levels, including college, is warranted.
- Limited Demographic Information: While student grade and ELL status are available, information on gender, race/ethnicity, or socioeconomic status is missing, which could influence human ratings and model performance.
- Future Directions: The authors envision integrating CLASSE scoring models into educational applications to provide students with immediate feedback, facilitating iterative revision and enhanced learning. They also suggest that the winning models from the Kaggle competition, which addressed some of CLASSE's limitations (e.g., by augmenting data), offer promising avenues for future research.
Summary updated