Response Construct Tagging: NLP-Aided Assessment for Engineering Education

Ananya Ganesh, Hugh Scribner, Jasdeep Singh, Katherine Goodman, Jean Hertzberg, Katharina KannNAACL 2022

TL;DR

This paper introduces "Response Construct Tagging" (RCT), a novel NLP task to automatically assess students' affective states towards specific engineering education constructs from open-ended survey responses. By fine-tuning a RoBERTa model and employing multi-task learning with sentiment analysis, they achieve a significant improvement in performance, paving the way for scalable, data-driven curriculum design in engineering education.

What Problem Does This Paper Address?

Engineering education research (EER) aims to improve pedagogical practices, often by understanding how students' experiences, particularly their "engineering thinking" and identity, are shaped by curricula. A common method is to use pre- and post-course surveys with open-ended questions to gauge student perceptions and affective gains. However, manually analyzing these qualitative responses at scale is time-consuming and labor-intensive. This paper addresses the challenge of automating this analysis by developing an NLP system that can automatically tag student responses for their affective state (positive, negative, neutral, or unavailable) across six predefined constructs related to transformative experiences and engineering identity.

Methods

The core of this work is the development of a Response Construct Tagging (RCT) task. Given a student's response and a specific construct, the goal is to predict the student's affect towards that construct.

The authors experimented with several approaches:

  • RoBERTa Classifier: They fine-tuned a pre-trained RoBERTa model, a state-of-the-art language representation model, for sequence classification.
  • Additional Input: They explored enriching the input by including the survey question corresponding to the response and the description of the construct itself.
  • Multi-Task Learning (MTL): To address potential data sparsity and leverage related tasks, they employed MTL by jointly training the RCT task with a sentiment classification task on a separate, larger dataset. This shared encoder approach aims to improve generalization.
  • Class Imbalance Strategies: Recognizing the imbalanced nature of their dataset (many "unavailable" labels), they experimented with class weighting and upsampling techniques to improve performance on rarer affect labels.

Datasets

  • Custom Survey Dataset: The primary dataset consists of 232 anonymized student responses from an industrial design course at a large public university. These responses were collected via pre- and post-course surveys and manually annotated by a trained annotator for affect towards six constructs. This dataset was split into training, development, and test sets, resulting in 708, 240, and 468 instances per construct, respectively.
  • Michigan EECS Targeted Sentiment Analysis Dataset: This dataset, containing 1144 student feedback responses from a Computer Engineering program, was used for the multi-task learning setup. It includes annotations for positive and negative sentiment.

Key Insights and Ideas

  • NLP for Affective State in EER: The paper demonstrates that NLP can effectively move beyond simple sentiment analysis to capture nuanced affective states related to specific educational constructs, offering a scalable solution for EER.
  • Multi-Task Learning is Crucial: Jointly training the RCT task with sentiment classification significantly boosted performance, especially for rarer affect labels (neutral and negative). This highlights the benefit of leveraging related tasks when dealing with limited data.
  • Construct Descriptions Matter: Including the description of the construct as additional input to the model improved performance, suggesting that explicit guidance on what each construct represents aids the model.
  • Affective States Differ from Sentiment: The analysis of lexicon overlap suggests that the affective states of interest in this EER context are distinct from general sentiment, requiring specialized models.
  • Label Imbalance is a Major Hurdle: The dataset is heavily skewed towards "unavailable" labels, making it challenging to accurately predict rarer affect categories like neutral and negative. While upsampling showed some promise for neutral labels, overall class imbalance remains a significant challenge.

Key Results

  • The best single-task RoBERTa model achieved a macro-averaged F1 score of 48.4.
  • Incorporating the survey question and construct description improved performance to 48.4 F1.
  • Multi-task learning with sentiment classification resulted in the highest performance, achieving a macro-averaged F1 score of 55.1. This represents a 6.7 F1 point improvement over the best single-task model.
  • MTL led to substantial improvements on rarer classes: negative affect improved by 15.9 F1 (from 33.3 to 49.2) and neutral affect by 14.2 F1 (from 5.8 to 20.0).
  • The model performs well on dominant labels (unavailable and positive) but struggles with rarer labels (neutral and negative) due to class imbalance.

Limitations and Future Work

  • Label Sparsity and Nuance: RCT is inherently challenging due to the nuanced differences between affect labels and constructs, and the sparsity of labels in the dataset. This makes learning rarer combinations difficult.
  • Unwillingness to Express Certain Affects: Students may be hesitant to express negative feelings about their engineering identity (e.g., negative "Identification"), leading to underrepresentation of such labels.
  • Future Work: The authors suggest exploring methods to mitigate label sparsity, such as generating artificial student responses for underrepresented constructs and affects, potentially through manual annotation or conditional generation. They also plan to investigate how to improve the detection of nuanced differences between affect labels and constructs.

Summary updated

BibTeX
@inproceedings{ganesh2022response,
  author = {Ananya Ganesh and Hugh Scribner and Jasdeep Singh and Katherine Goodman and Jean Hertzberg and Katharina Kann},
  title = {{Response Construct Tagging: NLP-Aided Assessment for Engineering Education}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure