Argument Component Classification for Classroom Discussions

Luca Lugini, Diane LitmanEMNLP 2018

TL;DR

This paper tackles the challenge of automatically identifying argument components (claims, evidence, warrants) within spoken classroom discussions, a domain previously underexplored in argument mining. By adapting and combining features from essay and online dialogue argument mining, and experimenting with neural network architectures and multi-task learning, the authors significantly improve classification performance over existing methods.

What Problem Does This Paper Address?

Argument mining, a subfield of Natural Language Processing (NLP), aims to automatically extract structured arguments from unstructured text. While significant progress has been made in analyzing arguments in student essays and online forums, the domain of spoken classroom discussions remains largely unaddressed. This is a critical gap because classroom discussions are a vital pedagogical tool for developing students' critical thinking, reasoning, and communication skills. Automatically analyzing these discussions could provide valuable insights for educators to understand student learning and improve teaching strategies. The authors highlight that existing argument mining systems, trained on essays or online dialogues, perform poorly on classroom discussions due to differences in linguistic structure and the collaborative, spoken nature of the data.

Methods

The researchers explored several approaches to argument component classification:

  • Baseline: They first evaluated an existing argument mining system (wLDA), originally developed for student essays, directly on their classroom discussion dataset. As expected, its performance was poor.
  • Feature Engineering: They then investigated the effectiveness of features from prior work on argument mining for student essays (wLDA features) and online dialogues. Combining these feature sets significantly improved performance.
  • Neural Networks: To automatically extract features, they experimented with two popular neural network architectures:
  • Convolutional Neural Networks (CNNs): These models were found to be more robust, particularly at the character level.
  • Recurrent Neural Networks (RNNs), specifically LSTMs: These models showed less consistent performance, especially without handcrafted features.
  • Both CNNs and LSTMs were evaluated using both character-level and word-level embeddings.
  • Hybrid Models: To address the limited dataset size, they combined handcrafted features with the outputs of neural networks.
  • Multi-task Learning: Recognizing a potential relationship between argument components and "specificity" (how focused an utterance is on specific characters or scenes), they implemented multi-task learning. This involved training models to simultaneously predict both argument component type and specificity, aiming to leverage shared representations and improve performance.

Datasets

The study utilized a newly collected dataset of 73 transcribed classroom discussions from high school English Language Arts (ELA) classes. These discussions were teacher-mediated in most cases and originated from urban, suburban, and small-town schools. The data was manually annotated for:

  • Argument Component: Claims, Evidence, and Warrants, based on the Toulmin model.
  • Specificity: A three-level scale (Low, Medium, High) indicating how specific an utterance is.

The unit of analysis was an "argument move," a segment of text containing a single argumentative discourse unit (ADU).

Key Insights and Ideas

  • Domain Specificity Matters: Argument mining models trained on essays do not directly transfer well to spoken classroom discussions, underscoring the need for domain-specific adaptation.
  • Feature Combination is Powerful: Combining features from different argument mining domains (essays and online dialogues) provided a strong performance boost, suggesting that these diverse sources capture complementary aspects of argumentation.
  • CNNs Shine in this Domain: Convolutional Neural Networks, especially when operating at the character level, demonstrated superior robustness and performance compared to LSTMs in this specific task, likely due to their ability to act as effective feature detectors.
  • Multi-task Learning for Specificity Boosts Performance: Jointly training models to predict argument components and specificity significantly improved performance, particularly for LSTMs, suggesting that specificity is a valuable auxiliary task.
  • Warrants are Challenging: Warrants, being the minority class, remained the most difficult component to classify accurately, highlighting an ongoing challenge in argument mining.

Key Results

  • The best performing model, a CNN incorporating both wLDA and online dialogue features, trained with multi-task learning, achieved a kappa score of 0.474 and an F-score of 0.468 for argument component classification.
  • This multi-task CNN model significantly outperformed the pre-trained wLDA system (kappa 0.077, F-score 0.269) and a logistic regression model trained on wLDA features alone (kappa 0.142, F-score 0.379).
  • Even when using neural networks alone, CNNs generally outperformed LSTMs.
  • The multi-task learning approach showed a notable improvement, with models trained on both argumentation and specificity consistently outperforming those trained only on argumentation.
  • For warrants specifically, the word-level CNN model using wLDA and online dialogue features, trained with multi-task learning, was the only model to achieve an F-score greater than 0.3.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Data Imbalance: The significant imbalance in class distribution (especially for warrants) remains a challenge, despite efforts like oversampling.
  • Performance Gap: While performance improved substantially, it still lags behind argument mining systems in other domains like legal texts or essays.
  • Contextual Information: The current models do not fully exploit the collaborative nature of classroom discussions. Future work could extend feature sets to better capture contextual information from previous argument moves.
  • Cross-Validation: The use of leave-one-transcript-out cross-validation introduces variability. Future work could explore other cross-validation strategies.
  • Additional Tasks: The success of multi-task learning suggests exploring additional auxiliary tasks to further boost performance.

Summary updated

BibTeX
@inproceedings{lugini2018argument,
  author = {Luca Lugini and Diane Litman},
  title = {{Argument Component Classification for Classroom Discussions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure