Enhancing Instructor-Student and Student-Student Interactions with Mobile Interfaces and Summarization

Wencan Luo, Xiangmin Fan, Muhsin Menekse, Jingtao Wang, Diane LitmanNAACL 2015

TL;DR

This paper introduces CourseMIRROR, a mobile application designed to foster better instructor-student and student-student interactions in large classrooms. It automates the collection and summarization of student reflections, providing concise overviews that highlight common points of confusion and student engagement.

What Problem Does This Paper Address?

In large educational settings, especially introductory STEM courses and MOOCs, it's challenging and resource-intensive for instructors to effectively gauge student understanding and identify areas of confusion. Traditional methods of collecting student feedback, like paper-based reflections, are cumbersome to administer and analyze. While reflection prompts are known to improve learning, the manual process of collecting, organizing, and summarizing these reflections for both instructors and students is a significant bottleneck. This paper aims to bridge this gap by leveraging mobile technology and NLP to create a scalable and efficient feedback loop.

Methods

CourseMIRROR employs a multi-stage approach to summarize student reflections:

  • Candidate Phrase Extraction: It uses the Senna toolkit to extract noun phrases (NPs) from student reflections. NPs are chosen because they often represent the "what" of a concept, aligning with the typical nature of reflection prompts. Malformed phrases (e.g., single stop words, phrases starting with punctuation) are filtered out.
  • Phrase Clustering: To group semantically similar phrases, the system uses K-Medoids clustering. This method is chosen for its robustness to sparse data (common in student responses) and its ability to work with arbitrary distance matrices. Semantic similarity is leveraged using SEMILAR. The clustering algorithm is run multiple times to find the best configuration.
  • Phrase Ranking: Phrases are ranked based on "student coverage" – the estimated number of students who mentioned a particular concept. The algorithm assumes that any phrase within a cluster can represent the entire cluster's coverage. LexRank is used within clusters to score candidate phrases, and only the top-ranked phrase from each cluster is included in the summary. This process continues until a user-defined summary length is reached.

A key innovation is the mobile interface, which allows for easy submission of reflections and presents summaries that highlight phrases mentioned by the current user, encouraging engagement.

Datasets

The paper mentions using an "engineering course corpus consisting of handwritten student reflections generated in response to instructor prompts at the end of each lecture, along with associated summaries manually generated by the course TA" for evaluating the phrase summarization method. This corpus appears to be a benchmark established by Menekse et al. (2011). The pilot study described in the paper involved two graduate-level courses (CS2001 and CS2610) with 21 lectures each, collecting a total of 344 reflections.

Key Insights and Ideas

  • Mobile is Key for Scalability: The always-on, connected nature of mobile devices significantly simplifies the administration and collection of reflections compared to traditional methods, making it practical for large classes.
  • Summaries as Learning Tools: Automatically generated summaries, especially when highlighting individual contributions, can serve as valuable tools for both students (to see peer perspectives and their own impact) and instructors (to quickly grasp common difficulties).
  • Phrase-Level Summarization is More Suitable: For short, diverse student inputs (ranging from single words to sentences), summarizing at the phrase level is more effective and user-friendly, particularly on small mobile screens.
  • Student Coverage as a Salience Metric: Ranking phrases by how many students mention them provides a direct measure of concept importance and widespread confusion, offering actionable insights for instructors.

Key Results

  • Reflection Length: Students typed significantly more words per reflection using CourseMIRROR (around 11.6 words) compared to handwritten reflections in a previous study (around 9.7 words), which was counter-intuitive given typing on small screens.
  • User Feedback: A pilot deployment yielded positive quantitative and qualitative feedback. Participants reported benefiting from reading reflection summaries and enjoyed seeing their own contributions highlighted.
  • Summarization Performance: The phrase summarization method achieved "significantly better ROUGE scores than baselines" like MEAD, LexRank, and MMR, indicating its effectiveness.

Limitations and Future Work

  • Android-First: Currently, only an Android mobile client is provided; an iOS version is under development, and a web client is available as an alternative.
  • Further Verification Needed: The finding on increased reflection length via typing requires further controlled experiments to verify.
  • Future Directions: The authors plan to examine how student responses (rate, length, quality) relate to actual student learning performance. They also aim to explore how the system can be further improved.

Summary updated

BibTeX
@inproceedings{luo2015enhancing,
  author = {Wencan Luo and Xiangmin Fan and Muhsin Menekse and Jingtao Wang and Diane Litman},
  title = {{Enhancing Instructor-Student and Student-Student Interactions with Mobile Interfaces and Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure