Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments

Maja Stahl, Nick Düsterhus, Mei-Hua Chen, Henning WachsmuthEMNLP 2023 · DOI

TL;DR

This paper tackles the challenge of improving argumentative writing in learners by automatically creating datasets for detecting and filling "gaps" in arguments (enthymemes). They developed a novel self-supervised method to generate these enthymematic arguments from existing learner essays, enabling the training of models for enthymeme detection and reconstruction.

What Problem Does This Paper Address?

Writing strong arguments is tough for learners. It's not just about presenting evidence, but also about strategically omitting certain points (enthymemes) to connect with the audience. However, when crucial implicit premises or conclusions are missing, arguments become unclear or unconvincing. While NLP has shown promise in providing feedback on argument quality, there's a lack of resources and methods specifically for identifying and fixing these "enthymematic gaps" in learner writing. This paper aims to bridge this gap by creating a specialized corpus and defining tasks for enthymeme detection and reconstruction.

Methods

The core of this work is an automated corpus creation process. It starts with high-quality argumentative learner essays and systematically removes argumentative discourse units (ADUs) to create enthymemes. This process involves several key steps:

  • Finding Candidate Arguments: Essays are filtered to ensure they are of decent quality and have a minimum length, aiming to avoid arguments that are already too short or inherently flawed.
  • Finding Candidate Enthymemes: ADUs are selected as potential enthymemes based on two criteria:
  • Naturalness ADU Filtering: The removal of the ADU should not make the remaining text sound unnatural. This is checked using BERT's next sentence prediction to ensure sentence coherence.
  • Argumentativeness ADU Filtering: The removed ADU must be argumentative itself, ensuring its absence genuinely creates a logical gap.
  • Ranking Candidate Enthymemes: A PageRank-based approach is used to rank candidate ADUs for removal. This ranking considers both the ADU's centrality within the essay (how important it is to the overall structure) and its quality contribution (how much it contributes to the argument's quality). The hypothesis is that removing central, high-quality ADUs will create the most impactful enthymemes.

The authors also developed essay scoring models for organization, thesis clarity, and argument strength, which are used in the corpus creation process.

Datasets

  • Base Corpora:
    • ICLEv2 (Granger et al., 2009) for training essay scoring models.
  • ICLEv3 (Granger et al., 2020) as the primary source for creating the new enthymematic corpus. ICLEv3 contains 8,965 argumentative essays from learners with diverse language backgrounds.
  • Newly Introduced Benchmark: The paper introduces a new corpus of automatically generated enthymematic arguments. This corpus consists of 40,089 instances for enthymeme detection and 17,762 for enthymeme reconstruction. This differs from existing datasets by focusing specifically on inadequate enthymemes in learner arguments and using an automated generation process.

Key Insights and Ideas

  • Automated Enthymeme Generation is Feasible: The paper demonstrates that it's possible to automatically create a corpus of enthymematic arguments that mimic learner writing, reducing the need for costly manual annotation.
  • Centrality and Quality Matter for Enthymemes: The ranking mechanism, prioritizing ADUs that are both central to the argument and contribute significantly to its quality, is key to creating impactful enthymemes that genuinely reduce argument sufficiency.
  • Naturalness is Preservable: The naturalness ADU filtering step is crucial. It ensures that while arguments become logically weaker, they don't become jarringly unnatural, making them more realistic for studying learner writing.
  • Learner Arguments Often Lack Sufficiency: The manual evaluation suggests that many learner arguments already have issues with sufficiency, and the automated enthymeme creation process effectively exacerbates this.

Key Results

  • Corpus Creation: The automated process successfully generated 40,089 enthymeme detection instances and 17,762 reconstruction instances.
  • Sufficiency Reduction: Manual evaluation showed that the full approach significantly reduced argument sufficiency, outperforming variations and a random baseline. The full approach was also least likely to increase sufficiency.
  • Naturalness Maintenance: Manual evaluation indicated that the modified arguments maintained a high level of naturalness, with scores only slightly lower than original learner arguments.
  • Enthymeme Detection Baseline: A DeBERTa classifier achieved an F1-score of 0.68, significantly outperforming random and majority baselines, demonstrating the feasibility of automated detection.
  • Enthymeme Reconstruction Baseline: The BART-augmented approach achieved competitive results (BERTScore F1 of 0.252, ROUGE-L of 17.28) compared to a prior method, suggesting that providing topic and ADU type information aids reconstruction.

Limitations and Future Work

  • Lack of Human-Created Enthymemes: The authors cannot directly compare their automatically generated enthymemes to human-created ones, and there's a slight uncertainty about whether the generated enthymemes perfectly align with traditional argumentation theory.
  • Potential for Existing Enthymemes: Despite quality filters, original essays might still contain unknown inadequate enthymemes, potentially affecting corpus annotation quality.
  • Generalizability: The corpus creation and evaluation were performed on a single corpus (ICLEv3). Further research is needed to assess generalizability across different argumentative text types and learner proficiency levels.
  • Baseline Model Performance: The authors acknowledge that the performance of their baseline models for detection and reconstruction can be further improved, suggesting the need for more advanced approaches.
  • Future Work: The paper suggests exploring more advanced approaches for enthymeme detection and reconstruction, and applying these methods to a wider range of argumentative texts.

Summary updated

BibTeX
@inproceedings{stahl2023mind,
  author = {Maja Stahl and Nick Düsterhus and Mei-Hua Chen and Henning Wachsmuth},
  title = {{Mind the Gap: Automated Corpus Creation for Enthymeme Detection and Reconstruction in Learner Arguments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.312},
}
Figure