Implicit Phenomena in Short-answer Scoring Data

Marie Bexte, Andrea Horbach, Torsten ZeschACL 2021

TL;DR

This paper investigates how implicit language in student answers affects automatic scoring systems. They found that while humans effortlessly resolve these implicit cues, current AI models can struggle, and making answers more explicit often improves scoring accuracy. This work highlights the need for NLP models to better handle nuanced language for more robust educational AI.

What Problem Does This Paper Address?

Automatic short-answer scoring (SAS) systems aim to grade student responses based on content correctness, mimicking how human teachers would evaluate them. These systems are crucial in educational settings, from science education to language learning. The core challenge is that while the meaning of an answer is paramount, not its exact wording, many state-of-the-art SAS models heavily rely on surface-level lexical features like word embeddings or n-grams. This reliance means that subtle linguistic nuances, particularly implicit phenomena, can significantly impact how these models perceive an answer's correctness.

Implicit phenomena are linguistic elements where meaning is not directly stated but inferred from context. Examples include:

  • Coreference: Using pronouns (e.g., "it") to refer to previously mentioned entities (e.g., "the panda").
  • Bridging Anaphora: Indirect relationships between words, like referring to "the fur" of "the panda" without explicitly stating "the panda's fur."
  • Ellipsis: Omitting words or phrases that are understood from context (e.g., "Koalas are highly specialized, too" implies "Koalas are highly specialized").
  • Numeric Terms: Leaving out measurement units (e.g., "they turn 30" instead of "they turn 30 years old").
  • Information Structure: How information is presented in relation to a question, particularly what is considered "focus."
  • Presupposition: An underlying assumption required for a statement to be true (e.g., "pandas no longer eat bamboo" presupposes they used to).
  • Implicit Discourse Relations: Unstated logical connections between sentences (e.g., an implicit "therefore").

The paper's central question is: To what extent do these implicit phenomena occur in real student answers, and how do they influence the performance of automatic scoring models?

Methods

The researchers employed a multi-step approach:

  1. Phenomenon Identification: They first identified several key implicit linguistic phenomena relevant to short answers.
  2. Dataset Selection and Annotation: They selected publicly available short-answer datasets (SRA and ASAP) and then manually annotated a subset of answers for four specific phenomena: coreference, bridging anaphora, ellipsis, and numeric terms. This annotation involved identifying the implicit spans and linking them to their explicit antecedents or resolutions. They also calculated inter-annotator agreement to ensure reliability.
  3. Experimental Setup: They used a Support Vector Machine (SVM) classifier (specifically Weka's SMO) trained on uni- to trigram features and POS tags. For each prompt, they trained a separate classifier.
  4. Controlling for Explicitness: To assess the impact of implicitness, they created two experimental conditions:
  5. Baseline: The model was trained and tested on the original, unaltered dataset.
  6. Explicit: They created an "explicit" version of the dataset by using their annotations to resolve implicit phenomena. For example, pronouns were replaced with their antecedents, and omitted words were added back.
  7. Analysis: They compared the scoring performance (using QWK and kappa metrics) of models trained on the baseline versus the explicit datasets. They also analyzed how the predicted scores changed when answers were made explicit, comparing this to human grading tendencies.

Datasets

The study utilized two primary datasets:

  • Student Response Analysis Corpus (SRA): This corpus, from the 2013 SemEval task, includes data from two sources:
  • Beetle subset: 3,000 student answers to 56 questions about electricity and electronics.
  • SciEntsBank subset: 10,000 student answers to 197 questions across various science domains. Both subsets provide reference answers and 5-way appropriateness labels.

  • 2012 Automated Student Assessment Prize (ASAP): This dataset contains approximately 2,200 student answers for each of ten science-related prompts. Answers were rated on a 3- or 4-point scale.

The researchers selected specific prompts from these datasets for their annotation study, aiming for those with a higher prevalence of the target implicit phenomena.

Key Insights and Ideas

  • Implicitness is Pervasive: The study found that implicit phenomena are common in learner answers, with coreference appearing in 97% of the annotated answers for a specific prompt.
  • Human vs. Machine Resolution: Humans effortlessly resolve these implicit cues, but automatic systems often struggle, leading to potential scoring inaccuracies.
  • Explicitness Can Help: Making answers more explicit by resolving implicit phenomena generally led to improved automatic scoring performance, suggesting that current models benefit from clearer, less ambiguous language.
  • Phenomenon-Specific Impact: The effect of making answers explicit is not uniform across all phenomena. While coreference and ellipsis resolution consistently improved scores, bridging anaphora showed mixed results, and numeric terms had varied impacts depending on the prompt.
  • Annotation Challenges: While inter-annotator agreement was good for coreference and bridging, it was notably lower for ellipsis, indicating that even human annotators can find certain implicit structures ambiguous.

Key Results

  • Improved Scoring: Making answers more explicit led to slight increases in QWK scores for coreference (.01) and ellipsis (.03). For bridging, kappa increased by .03 and .07. For numeric terms, one prompt saw a significant increase of .17 in kappa.
  • Mixed Human-like Behavior: When analyzing how predicted scores changed, they found that for coreference, explicit versions sometimes received lower scores, but more often received higher scores. For ellipsis, explicitness was consistently beneficial.
  • Phenomenon-Dependent Trends: The impact of explicitness varied. For numeric terms, making them explicit generally improved predictions. However, for bridging, explicitness sometimes led to worse predicted scores, suggesting that the context and specific prompt play a crucial role.
  • Annotation Reliability: Token-level agreement ranged from .45 (ellipsis) to .86 (bridging), with antecedent agreement generally high (.90+) for coreference, bridging, and ellipsis, but lower for numeric terms (.51-.70).

Limitations and Future Work

  • Limited Scope: The study focused on only four specific implicit phenomena and a limited number of prompts.
  • Manual Annotation: The annotation process was manual and time-consuming, limiting the scale of the analysis.
  • Model Specificity: The experiments used a specific SVM classifier; results might differ with other models.

The authors suggest several future directions:

  • Automatic Resolution: Developing automatic methods to resolve implicit language into explicit forms to scale up these experiments.
  • Broader Phenomenon Coverage: Investigating a wider range of implicit phenomena.
  • Combined Resolution: Exploring the effect of resolving multiple implicit phenomena simultaneously within the same answer.
  • Publishing Annotations: The paper mentions they will publish their annotated dataset, which will be a valuable resource for future research.

Summary updated

BibTeX
@inproceedings{bexte2021implicit,
  author = {Marie Bexte and Andrea Horbach and Torsten Zesch},
  title = {{Implicit Phenomena in Short-answer Scoring Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure