Multiple Instance Learning for Content Feedback Localization without Annotation

Scott Hellman, William Murray, Adam Wiemerslage, Mark Rosenstein, Peter Foltz, Lee Becker, Marcia DerrACL 2020

TL;DR

This paper introduces a novel approach to automatically identify and localize specific content topics within student essays, crucial for providing targeted feedback. By framing Automated Essay Scoring (AES) as a Multiple Instance Learning (MIL) problem, they achieve this localization without requiring any manually annotated text spans, a significant reduction in data collection costs.

What Problem Does This Paper Address?

Providing effective feedback to students on their writing is a cornerstone of education, but it's a resource-intensive process. While Automated Essay Scoring (AES) systems can reliably score essays, they often provide only holistic, document-level feedback. This leaves students to interpret where in their essay a particular piece of feedback applies. To make feedback truly actionable, it needs to be localized to the specific sentences or text spans where a topic is discussed or an error occurs.

Traditionally, building systems that can localize content requires extensive manual annotation of essays, where human experts mark the exact text segments corresponding to specific rubric criteria. This annotation process is extremely time-consuming and expensive, hindering the development of such systems. The authors aim to bridge this gap by developing a method for content localization that bypasses the need for this costly annotation data.

Methods

The core of the paper's contribution lies in framing the problem as Multiple Instance Learning (MIL). In MIL, data is organized into "bags" (in this case, essays), and each bag has a label. The individual items within the bag ("instances," i.e., sentences) have unknown labels. The bag label is assumed to be an aggregation of the instance labels.

The authors adapt MIL for both essay scoring and content localization. For scoring, they use a k-Nearest Neighbors (kNN) model trained on sentence embeddings. They then aggregate sentence-level scores to predict a document-level score. This aggregation can be done using either the "standard" MIL assumption (a bag is positive if any instance is positive, here using the maximum sentence score) or the "collective" assumption (the bag label is an aggregation of all instance labels, here using the mean sentence score).

For content localization, they leverage the sentence-level scores generated by their MIL model. They propose that a sentence receiving a high score from the MIL model is a weak signal for the presence of an annotation for that topic. This sentence-level score is then rescaled to a [0, 1] range, acting as an annotation prediction score.

Crucially, the authors demonstrate that this MIL model can be trained without explicit annotation data. They propose three ways to obtain training data:

  1. Document-level scores: Using existing holistic scores for essays.
  2. Rubric references: Using sentences from the rubric itself as examples.
  3. Manually selected references: Having human experts pick exemplar sentences from high-scoring essays.

Datasets

The authors utilize a proprietary Pearson corpus consisting of student responses to four university-level psychology writing prompts. These essays were originally scored holistically, and a subsequent effort factored content into multiple topics. Annotators assigned essay-level ratings (Complete, Partial, Incorrect, Omitted) for each topic and also marked text spans where the topic was discussed.

The dataset contains approximately 258-448 essays per prompt, with 5-8 topics per prompt. The authors note that due to time and cost constraints, adjudicated labels from a third SME were not available for all prompts, leading them to run experiments on both annotators separately.

Key Insights and Ideas

  • Bypassing Annotation Cost: The most significant insight is that content localization can be achieved without the prohibitive cost of manually annotating text spans. This opens the door for more widespread adoption of localized feedback systems.
  • MIL as a Natural Fit: Framing AES and localization as MIL problems proves effective. The idea that a single sentence can sometimes be sufficient to determine a topic's score (standard MIL assumption) aligns well with how students might discuss specific points.
  • Sentence Scores as Weak Signals: The paper demonstrates that the sentence-level scores produced by a MIL model, even when trained on document-level data, can serve as surprisingly effective weak signals for annotation prediction.
  • Document-Level Scores Inform Localization: The ability of the MIL model to perform well on both document-level scoring and sentence-level localization suggests a strong connection between understanding the overall essay quality and identifying specific content contributions.

Key Results

  • Annotation Prediction Performance: The kNN-MIL models achieved AUCs above 0.5 for annotation prediction, indicating performance better than random chance. The "Base KNN-MIL" model (trained on document-level scores) generally outperformed the "Manual KNN-MIL" model (trained on manually selected references), especially when using the tf-idf vector space.
  • Document-Level Scoring Performance: The Base KNN-MIL model also performed reasonably well on the document-level scoring task, with Pearson correlations ranging from 0.30 to 0.67 across different prompts and aggregation methods. The standard MIL assumption (using the maximum sentence score) generally outperformed the collective assumption (using the mean sentence score) for scoring.
  • Comparison to OLR: The ordinal logistic regression (OLR) model, a previous state-of-the-art for similar tasks, performed relatively poorly on annotation prediction compared to the kNN-MIL approaches, suggesting that its focus on sentence-level features for feedback doesn't directly translate to precise localization.

Limitations and Future Work

  • Sentence-Level Granularity: The current approach operates at the sentence level. The authors acknowledge that it is unlikely to correctly locate annotations that span multiple sentences. Future work could incorporate information across sentences, perhaps through recurrent neural networks or other sequence modeling techniques.
  • Inter-Topic Correlation: The Base KNN-MIL approach uses topics as negative examples for each other. This means it might not perform as well in situations where topic scores are highly correlated. The Manual KNN-MIL approach is less sensitive to this. Future work could explore better ways to incorporate negative examples.
  • Domain Specificity: The current work is focused on psychology essays. The authors suggest that future research should investigate how well the approach generalizes to other domains, especially those requiring more complex reasoning or where misconceptions are more critical.
  • Feedback Generation: Localization is only one step. Future work needs to explore how to translate these localized insights into pedagogically sound and individualized formative feedback, potentially incorporating various intervention methods beyond just text.

Summary updated

BibTeX
@inproceedings{hellman2020multiple,
  author = {Scott Hellman and William Murray and Adam Wiemerslage and Mark Rosenstein and Peter Foltz and Lee Becker and Marcia Derr},
  title = {{Multiple Instance Learning for Content Feedback Localization without Annotation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure